Video Game Blurs (and how the best one works)

Blurs are the basic building block for many video game post-processing effects and essential for sleek and modern GUIs. Video game Depth of Field and Bloom or frosted panels in modern user interfaces - used subtly or obviously - they’re everywhere. Even your browser can do it, just tap this sentence!

Conceptually, “Make thing go blurry” is easy, boiling down to some form of “average colors in radius”. Doing so in realtime however, took many a graphics programmer through decades upon decades of research and experimentation, across computer science and maths. In this article, we’ll follow their footsteps.

A graphics programming time travel, if you will.

Using the GPU in the device you are reading this article on, and the WebGL capability of your browser, we’ll implement realtime blurring techniques and retrace the trade-offs graphics programmers had to make in order to marry two, sometimes opposing, worlds: Mathematical theory and Technological reality.

This is my submission to this year's Summer of Math Exposition

With many interactive visualizations to guide us, we’ll journey through a bunch of blurs, make a detour through frequency space manipulations, torture your graphics processor to measure performance, before finally arriving at an algorithm with years worth of cumulative graphics programmer sweat - The ✨ Dual Kawase Blur 🌟

Setup - No blur yet #

In the context of video game post-processing, a 3D scene is drawn, also called rendering, and saved to an intermediary image - a framebuffer. In turn, this framebuffer is processed to achieve various effects. Since this processing happens after a 3D scene is rendered, it’s called post-processing. All that, many times a second.

Depending on technique, framebuffers can hold non-image data and post-processing effects like Color-correction or Tone-mapping don't even require intermediate framebuffers: There's more than one way (@35:20)

This is where we jump in: with a framebuffer in hand, after the 3D scene was drawn. We’ll use a scene from a mod called NEOTOKYO°. Each time we’ll implement a blur, there will be a box, a canvas instructed with WebGL 1.0, rendering at native resolution of your device. Each box has controls and relevant parts of its code below.

No coding or graphics programming knowledge required to follow along. But also no curtains! You can always see how we talk with your GPU. Terms and meanings will be explained, once it's relevant.

Blur Fragment Shader noBlurYet.fs

/* This is the blur "fragment shader", a program that runs on the GPU.
   In *this* article, the blur fragment shader runs once per output pixel of the
   canvas */

/* Required in WebGL 1 Shaders and depending on platform may have no effect.
   For Later: Strong blurs may have a lot of minute color contributions, so we
   set it "highp" here, the maximum. */
precision highp float;

/* UV coordinates, passed in from the Vertex Shader "simpleQuad.vs".
   This tells our current output pixel where to read our texture from. */
varying vec2 uv;

/* lightBrightness input. The reason light brightness is in the fragment shader
   of the blur and not a value applied in a step before our blur shader before,
   is due to color precision limits. */
uniform float lightBrightness;

/* Out texture input */
uniform sampler2D texture;

/* The "main" function, where which is executed by our GPU */
void main() {
	/* gl_FragColor is the output of our shader. texture2D is the texture read,
	   performed with the current 'uv' coordinate. Then multiplied by
	   our lightBrightness value (a multiplier with eg. 1.0 at 100%, 0.5 at 50%)
	   In "scene" mode, this value is locked to 1.0 so it has no effect */
	gl_FragColor = texture2D(texture, uv) * lightBrightness;
}

WebGL Javascript simple.js

import * as util from '../utility.js'

export async function setupSimple() {
	/* Init */
	const WebGLBox = document.getElementById('WebGLBox-Simple');
	const canvas = WebGLBox.querySelector('canvas');

	/* Circle Rotation size */
	const radius = 0.12;

	/* Main WebGL 1.0 Context */
	const gl = canvas.getContext('webgl', {
		preserveDrawingBuffer: false,
		antialias: false,
		alpha: false,
	});

	/* State and Objects */
	const ctx = {
		/* State for of the Rendering */
		mode: "scene",
		flags: { isRendering: false, buffersInitialized: false, initComplete: false, benchMode: false },
		/* Textures */
		tex: { sdr: null, selfIllum: null, frame: null, frameFinal: null },
		/* Framebuffers */
		fb: { scene: null, final: null },
		/* Shaders and their respective Resource Locations */
		shd: {
			scene: { handle: null, uniforms: { offset: null, radius: null } },
			blur: { handle: null, uniforms: { frameSizeRCP: null, samplePosMult: null, lightBrightness: null } },
			bloom: { handle: null, uniforms: { offset: null, radius: null, texture: null, textureAdd: null } }
		}
	};

	/* UI Elements */
	const ui = {
		display: {
			spinner: canvas.parentElement.querySelector('svg', canvas.parentElement),
			contextLoss: canvas.parentElement.querySelector('div', canvas.parentElement),
			fps: WebGLBox.querySelector('#fps'),
			ms: WebGLBox.querySelector('#ms'),
			width: WebGLBox.querySelector('#width'),
			height: WebGLBox.querySelector('#height'),
		},
		rendering: {
			animate: WebGLBox.querySelector('#animateCheck'),
			modes: WebGLBox.querySelectorAll('input[type="radio"]'),
			lightBrightness: WebGLBox.querySelector('#lightBrightness'),
			lightBrightnessReset: WebGLBox.querySelector('#lightBrightnessReset'),
		}
	};

	/* Shaders */
	const circleAnimation = await util.fetchShader("shader/circleAnimation.vs");
	const simpleTexture = await util.fetchShader("shader/simpleTexture.fs");
	const bloomVert = await util.fetchShader("shader/bloom.vs");
	const bloomFrag = await util.fetchShader("shader/bloom.fs");
	const simpleQuad = await util.fetchShader("shader/simpleQuad.vs");
	const noBlurYetFrag = await util.fetchShader("shader/noBlurYet.fs");

	/* Elements that cause a redraw in the non-animation mode */
	ui.rendering.lightBrightness.addEventListener('input', () => { if (!ui.rendering.animate.checked) redraw() });

	/* Events */
	ui.rendering.animate.addEventListener("change", () => {
		if (ui.rendering.animate.checked)
			startRendering();
		else {
			ui.display.fps.value = "-";
			ui.display.ms.value = "-";
			ctx.flags.isRendering = false;
			redraw()
		}
	});

	canvas.addEventListener("webglcontextlost", () => {
		ui.display.contextLoss.style.display = "block";
	});

	/* Render Mode */
	ui.rendering.modes.forEach(radio => {
		/* Force set to scene to fix a reload bug in Firefox Android */
		if (radio.value === "scene")
			radio.checked = true;
		radio.addEventListener('change', (event) => {
			ctx.mode = event.target.value;
			ui.rendering.lightBrightness.disabled = ctx.mode === "scene";
			ui.rendering.lightBrightnessReset.disabled = ctx.mode === "scene";
			if (!ui.rendering.animate.checked) redraw();
		});
	});

	/* Draw Texture Shader */
	ctx.shd.scene = util.compileAndLinkShader(gl, circleAnimation, simpleTexture, ["offset", "radius"]);

	/* Draw bloom Shader */
	ctx.shd.bloom = util.compileAndLinkShader(gl, bloomVert, bloomFrag, ["texture", "textureAdd", "offset", "radius"]);

	/* Helper for recompilation */
	function reCompileBlurShader() {
		ctx.shd.blur = util.compileAndLinkShader(gl, simpleQuad, noBlurYetFrag, ["lightBrightness"]);
	}

	/* Blur Shader */
	reCompileBlurShader()

	/* Send Unit code verts to the GPU */
	util.bindUnitQuad(gl);

	async function setupTextureBuffers() {
		ui.display.spinner.style.display = "block";
		ctx.flags.buffersInitialized = true;
		ctx.flags.initComplete = false;

		gl.deleteFramebuffer(ctx.fb.scene);
		gl.deleteFramebuffer(ctx.fb.final);
		[ctx.fb.scene, ctx.tex.frame] = util.setupFramebuffer(gl, canvas.width, canvas.height);
		[ctx.fb.final, ctx.tex.frameFinal] = util.setupFramebuffer(gl, canvas.width, canvas.height);

		let [base, selfIllum] = await Promise.all([
			fetch("/dual-kawase/img/SDR_No_Sprite.png"),
			fetch("/dual-kawase/img/Selfillumination.png")
		]);
		let [baseBlob, selfIllumBlob] = await Promise.all([base.blob(), selfIllum.blob()]);
		let [baseBitmap, selfIllumBitmap] = await Promise.all([
			createImageBitmap(baseBlob, { colorSpaceConversion: 'none', resizeWidth: canvas.width * 1.12, resizeHeight: canvas.height * 1.12, resizeQuality: "high" }),
			createImageBitmap(selfIllumBlob, { colorSpaceConversion: 'none', resizeWidth: canvas.width * 1.12, resizeHeight: canvas.height * 1.12, resizeQuality: "high" })
		]);

		ctx.tex.sdr = util.setupTexture(gl, null, null, ctx.tex.sdr, gl.LINEAR, baseBitmap);
		ctx.tex.selfIllum = util.setupTexture(gl, null, null, ctx.tex.selfIllum, gl.LINEAR, selfIllumBitmap);

		baseBitmap.close();
		selfIllumBitmap.close();

		ctx.flags.initComplete = true;
		ui.display.spinner.style.display = "none";
	}

	let prevNow = performance.now();
	let lastStatsUpdate = prevNow;
	let fpsEMA = 60;
	let msEMA = 16;

	async function redraw() {
		if (!ctx.flags.buffersInitialized)
			await setupTextureBuffers();
		if (!ctx.flags.initComplete)
			return;

		/* UI Stats */
		ui.display.width.value = canvas.width;
		ui.display.height.value = canvas.height;

		/* Circle Motion */
		let radiusSwitch = ui.rendering.animate.checked ? radius : 0.0;
		let speed = (performance.now() / 10000) % Math.PI * 2;
		const offset = [radiusSwitch * Math.cos(speed), radiusSwitch * Math.sin(speed)];
		gl.useProgram(ctx.shd.scene.handle);
		const texture = ctx.mode == "scene" ? ctx.tex.sdr : ctx.tex.selfIllum;
		gl.activeTexture(gl.TEXTURE0);
		gl.bindTexture(gl.TEXTURE_2D, texture);
		gl.uniform2fv(ctx.shd.scene.uniforms.offset, offset);
		gl.uniform1f(ctx.shd.scene.uniforms.radius, radiusSwitch);

		/* Setup PostProcess Framebuffer */
		gl.bindFramebuffer(gl.FRAMEBUFFER, ctx.fb.scene);
		gl.viewport(0, 0, canvas.width, canvas.height);

		/* Draw Call */
		gl.drawArrays(gl.TRIANGLE_FAN, 0, 4);

		/* Box blur at native resolution */
		gl.useProgram(ctx.shd.blur.handle);
		const finalFB = ctx.mode == "bloom" ? ctx.fb.final : null;
		gl.bindFramebuffer(gl.FRAMEBUFFER, finalFB);
		gl.viewport(0, 0, canvas.width, canvas.height);
		gl.uniform1f(ctx.shd.blur.uniforms.lightBrightness, ctx.mode == "scene" ? 1.0 : ui.rendering.lightBrightness.value);
		gl.activeTexture(gl.TEXTURE0);
		gl.bindTexture(gl.TEXTURE_2D, ctx.tex.frame);
		gl.drawArrays(gl.TRIANGLE_FAN, 0, 4);

		if (ctx.mode == "bloom") {
			/* Now do the bloom composition to the screen */
			gl.bindFramebuffer(gl.FRAMEBUFFER, null);
			gl.useProgram(ctx.shd.bloom.handle);

			gl.uniform2fv(ctx.shd.bloom.uniforms.offset, offset);
			gl.uniform1f(ctx.shd.bloom.uniforms.radius, radiusSwitch);

			gl.activeTexture(gl.TEXTURE0);
			gl.bindTexture(gl.TEXTURE_2D, ctx.tex.sdr);
			gl.uniform1i(ctx.shd.bloom.uniforms.texture, 0);

			gl.activeTexture(gl.TEXTURE1);
			gl.bindTexture(gl.TEXTURE_2D, ctx.tex.frameFinal);
			gl.uniform1i(ctx.shd.bloom.uniforms.textureAdd, 1);

			gl.drawArrays(gl.TRIANGLE_FAN, 0, 4);
		}

		/* Ask for CPU-GPU Sync to prevent overloading the GPU during compositing.
		   In reality this is more likely to be flush, but still, it seems to
		   help on multiple devices with during low FPS */
		gl.finish();

		const now = performance.now();
		let dt = now - prevNow;

		if (dt > 0) {
			const instFPS = 1000 / dt;
			const ALPHA = 0.05;
			fpsEMA = ALPHA * instFPS + (1 - ALPHA) * fpsEMA;
			msEMA = ALPHA * dt + (1 - ALPHA) * msEMA;
		}
		prevNow = now;

		if (ui.rendering.animate.checked && now - lastStatsUpdate >= 1000) {
			ui.display.fps.value = fpsEMA.toFixed(0);
			ui.display.ms.value = msEMA.toFixed(2);
			lastStatsUpdate = now;
		}
	}

	let animationFrameId;

	/* Render at Native Resolution */
	function nativeResize() {
		const [width, height] = util.getNativeSize(canvas);

		if (width && canvas.width !== width || height && canvas.height !== height) {
			canvas.width = width;
			canvas.height = height;

			if (!ctx.flags.benchMode) {
				stopRendering();
				startRendering();
			}
			if (!ui.rendering.animate.checked)
				redraw();
		}
	}

	/* Resize Event */
	nativeResize();

	let resizePending = false;
	window.addEventListener('resize', () => {
		if (!resizePending) {
			resizePending = true;
			requestAnimationFrame(() => {
				resizePending = false;
				nativeResize();
			});
		}
	});

	function renderLoop() {
		if (ctx.flags.isRendering && ui.rendering.animate.checked) {
			redraw();
			animationFrameId = requestAnimationFrame(renderLoop);
		}
	}

	function startRendering() {
		/* Start rendering, when canvas visible */
		ctx.flags.isRendering = true;
		renderLoop();
	}

	function stopRendering() {
		/* Stop another redraw being called */
		ctx.flags.isRendering = false;
		cancelAnimationFrame(animationFrameId);
		/* Force the rendering pipeline to sync with CPU before we mess with it */
		gl.finish();

		/* Delete the buffers to free up memory */
		gl.deleteTexture(ctx.tex.sdr); ctx.tex.sdr = null;
		gl.deleteTexture(ctx.tex.selfIllum); ctx.tex.selfIllum = null;
		gl.deleteTexture(ctx.tex.frame); ctx.tex.frame = null;
		gl.deleteTexture(ctx.tex.frameFinal); ctx.tex.frameFinal = null;
		gl.deleteFramebuffer(ctx.fb.scene); ctx.fb.scene = null;
		gl.deleteFramebuffer(ctx.fb.final); ctx.fb.final = null;
		ctx.flags.buffersInitialized = false;
		ctx.flags.initComplete = false;
		ui.display.fps.value = "-";
		ui.display.ms.value = "-";
	}

	function handleIntersection(entries) {
		entries.forEach(entry => {
			if (entry.isIntersecting) {
				if (!ctx.flags.isRendering && !ctx.flags.benchMode) startRendering();
			} else {
				stopRendering();
			}
		});
	}

	/* Only render when the canvas is actually on screen */
	let observer = new IntersectionObserver(handleIntersection);
	observer.observe(canvas);
}

We don’t have a blur implemented yet, not much happening. Above the box you have an Animate button, which will move the scene around to tease out problems of upcoming algorithms. Movement happens before our blur will be applied, akin to the player character moving. To see our blur in different use-cases, there are 3 modes:

Different blur algorithms behave differently based on use-case. Some are very performance efficient, but break under movement. Some reveal their flaws with small, high contrast regions like far-away lights

• In Scene mode the blur will be applied across the whole image
• In Lights mode we see and blur just the Emission parts of the scene, sometimes called “Self-Illumination”
  • This also unlocks the lightBrightness slider, where you can boost the energy output of the lights
• In Bloom mode, we use the original scene and add the blurred lights from the previous mode on top to create a moody scene. This implements the effect of Bloom, an important use-case for blurs in real-time 3D graphics

Adding the blurred emission pass as we do in this article, or thresholding the scene and blurring that, is not actually how modern video games do bloom. We'll get into that a bit later.

Finally, you see Resolution of the canvas and Frames per Second / time taken per frame, aka “frametime”. A very important piece of the puzzle is performance, which will become more and more important as the article continues and the mother of invention behind our story.

Frame-rate will be capped at your screen's refresh rate, most likely 60 fps / 16.6 ms. We'll get into proper benchmarking as our hero descents this article into blurry madness

Technical breakdown #

Understanding the GPU code is not necessary to follow this article, but if you do choose to peek behind the curtain, here is what you need to know

We’ll implement our blurs as a fragment shader written in GLSL. In a nut-shell, a fragment shader is code that runs on the GPU for every output-pixel, in-parallel. Image inputs in shaders are called Textures. These textures have coordinates, often called UV coordinates - these are the numbers we care about.

Technically, fragment shaders run per fragment, which aren't necessarily pixel sized and there are other ways to read framebuffers, but none of that matters in the context of this article.

UV coordinates specify the position we read in the image, with bottom left being 0,0 and the top right being 1,1. Neither UV coordinates, nor shaders themselves have any concept of image resolution, screen resolution or aspect ratio. If we want to address individual pixels, it’s on us to express that in terms of UV coordinates.

Although there are ways to find out, we don't know which order output-pixels are processed in, and although the graphics pipeline can tell us, the shader doesn't even know which output-pixel it currently processes

The framebuffer is passed into the fragment shader in line uniform sampler2D texture as a texture. Using the blur shader, we draw a “Full Screen Quad”, a rectangle covering the entire canvas, with matching 0,0 in the bottom-left and 1,1 in the top-right varying vec2 uv UV coordinates to read from the texture.

Due to automatic interpolation

The texture’s aspect-ratio and resolution are the same as the output canvas’s aspect-ratio and resolution, thus there is a 1:1 pixel mapping between the texture we will process and our output canvas. The graphics pipeline steps and vertex shader responsible for this are not important for this article.

The blur fragment shader accesses the color of the texture with texture2D(texture, uv), at the matching output pixel’s position. In following examples, we’ll read from neighboring pixels, for which we’ll need to calculate a UV coordinate offset, a decimal fraction corresponding to one pixel step, calculated with with 1 / canvasResolution

One way to think of fragment shader code is "What are the instructions to construct this output pixel?"

Graphics programming is uniquely challenging in the beginning, because of how many rules and limitations the hardware, graphics APIs and the rendering pipeline impose. But it also unlocks incredible potential, as other limitations dissolve. Let’s find out how graphics programmers have leveraged that potential.

Box Blur #

From a programmer’s perspective, the most straight forward way is to average the neighbors of a pixel using a for-loop. What the fragment shader is expressing is: “look Y pixels up & down, X pixels left & right and average the colors”. The more we want to blur, the more we have to increase kernelSize, the bounds of our for-loop.

/* Read from the texture y amount of pixels above and below */
for (int y = -kernel_size; y <= kernel_size; ++y) {
/* Read from the texture x amount of pixels to the left and the right */
	for (int x = -kernel_size; x <= kernel_size; ++x) {
		/* Offset from current pixel, indicating which pixel to read */
		vec2 offset = vec2(x, y) * samplePosMult * frameSizeRCP;
		/* Read and sum up the color contribution of that pixel */
		sum += texture2D(texture, uv + offset);
	}
}

The bigger the for-loop, the more texture reads we perform, per output-pixel. Each texture read is often called a “texture tap” and the total amount of those “taps” per-frame will now also be displayed. New controls, new samplePosMultiplier, new terms - Play around with them, get a feel for them, with a constant eye on FPS.

Detailed Benchmark Results

・Resolution: 1600x1200

・Execution Time: ~

?

/ iteration

・Texture taps:

?

Million / iteration

・GPU info:

Blur Fragment Shader boxBlur.fs

/* Float precision to highp, if supported. Large Kernel Sizes result many color
   contributions and thus require the highest precision to avoid clipping.
   Required in WebGL 1 Shaders and depending on platform may have no effect */
precision highp float;
/* UV coordinates, passed in from the Vertex Shader */
varying vec2 uv;

/* Resolution Reciprocal. Getting to the next pixel requires us to calculate
   `UV Coordinate / frameSize`. On hardware, doing a division is slightly slower
   than doing a multiplication. Since the shader is run per pixel, we avoid the
   per-pixel division, by calculating the reciprocal 1 / frameSize and pass it
   into the shader. A very popular micro-optimization across graphics programming */
uniform vec2 frameSizeRCP;
uniform float samplePosMult; /* Multiply to push blur strength past the kernel size */

uniform float bloomStrength; /* bloom strength */

uniform sampler2D texture;
/* `KERNEL_SIZE` added during compilation */
const int kernel_size = KERNEL_SIZE;

void main() {
	/* Variable to hold our final color for the current pixel */
	vec4 sum = vec4(0.0);
	/* How big one side of the sampled square is */
	const int size = 2 * kernel_size + 1;
	/* Total number of samples we are going to read */
	const float totalSamples = float(size * size);

	/* Read from the texture y amount of pixels above and below */
	for (int y = -kernel_size; y <= kernel_size; ++y) {
	/* Read from the texture x amount of pixels to the left and the right */
		for (int x = -kernel_size; x <= kernel_size; ++x) {
			/* Offset from the current pixel, indicating which pixel to read */
			vec2 offset = vec2(x, y) * samplePosMult * frameSizeRCP;
			/* Read and sum up the contribution of that pixel */
			sum += texture2D(texture, uv + offset);
		}
	}

	/* Return the sum, divided by the number of samples (normalization) */
	gl_FragColor = (sum / totalSamples) * bloomStrength;
}

WebGL Javascript boxBlur.js

import * as util from '../utility.js'

export async function setupBoxBlur() {
	/* Init */
	const WebGLBox = document.getElementById('WebGLBox-BoxBlur');
	const WebGLBoxDetail = document.getElementById('WebGLBox-BoxBlurDetail');
	const canvas = WebGLBox.querySelector('canvas');

	/* Circle Rotation size */
	const radius = 0.12;

	/* Main WebGL 1.0 Context */
	const gl = canvas.getContext('webgl', {
		preserveDrawingBuffer: false,
		antialias: false,
		alpha: false,
	});

	/* State and Objects */
	const ctx = {
		/* State for of the Rendering */
		mode: "scene",
		flags: { isRendering: false, buffersInitialized: false, initComplete: false, benchMode: false },
		/* Textures */
		tex: { sdr: null, selfIllum: null, frame: null, frameFinal: null },
		/* Framebuffers */
		fb: { scene: null, final: null },
		/* Shaders and their respective Resource Locations */
		shd: {
			scene: { handle: null, uniforms: { offset: null, radius: null } },
			blur: { handle: null, uniforms: { frameSizeRCP: null, samplePosMult: null, bloomStrength: null } },
			bloom: { handle: null, uniforms: { offset: null, radius: null, texture: null, textureAdd: null } }
		}
	};

	/* UI Elements */
	const ui = {
		display: {
			spinner: canvas.parentElement.querySelector('svg', canvas.parentElement),
			contextLoss: canvas.parentElement.querySelector('div', canvas.parentElement),
			fps: WebGLBox.querySelector('#fps'),
			ms: WebGLBox.querySelector('#ms'),
			width: WebGLBox.querySelector('#width'),
			height: WebGLBox.querySelector('#height'),
			tapsCount: WebGLBox.querySelector('#taps'),
		},
		blur: {
			kernelSize: WebGLBox.querySelector('#sizeRange'),
			samplePos: WebGLBox.querySelector('#samplePosRange'),
			samplePosReset: WebGLBox.querySelector('#samplePosRangeReset'),
		},
		rendering: {
			animate: WebGLBox.querySelector('#animateCheck'),
			modes: WebGLBox.querySelectorAll('input[type="radio"]'),
			lightBrightness: WebGLBox.querySelector('#lightBrightness'),
			lightBrightnessReset: WebGLBox.querySelector('#lightBrightnessReset'),
		},
		benchmark: {
			button: WebGLBox.querySelector('#benchmark'),
			label: WebGLBox.querySelector('#benchmarkLabel'),
			iterOut: WebGLBox.querySelector('#iterOut'),
			renderer: WebGLBoxDetail.querySelector('#renderer'),
			iterTime: WebGLBoxDetail.querySelector('#iterTime'),
			tapsCount: WebGLBoxDetail.querySelector('#tapsCountBench'),
			iterations: WebGLBox.querySelector('#iterations')
		}
	};

	/* Shaders */
	const circleAnimation = await util.fetchShader("shader/circleAnimation.vs");
	const simpleTexture = await util.fetchShader("shader/simpleTexture.fs");
	const bloomVert = await util.fetchShader("shader/bloom.vs");
	const bloomFrag = await util.fetchShader("shader/bloom.fs");
	const simpleQuad = await util.fetchShader("shader/simpleQuad.vs");
	const boxBlurFrag = await util.fetchShader("shader/boxBlur.fs");

	/* Elements that cause a redraw in the non-animation mode */
	ui.blur.kernelSize.addEventListener('input', () => { if (!ui.rendering.animate.checked) redraw() });
	ui.blur.samplePos.addEventListener('input', () => { if (!ui.rendering.animate.checked) redraw() });
	ui.rendering.lightBrightness.addEventListener('input', () => { if (!ui.rendering.animate.checked) redraw() });

	/* Events */
	ui.rendering.animate.addEventListener("change", () => {
		if (ui.rendering.animate.checked)
			startRendering();
		else {
			ui.display.fps.value = "-";
			ui.display.ms.value = "-";
			ctx.flags.isRendering = false;
			redraw()
		}
	});

	canvas.addEventListener("webglcontextlost", () => {
		ui.display.contextLoss.style.display = "block";
	});

	ui.blur.kernelSize.addEventListener('input', () => {
		reCompileBlurShader(ui.blur.kernelSize.value);
		ui.blur.samplePos.disabled = ui.blur.kernelSize.value == 0;
		ui.blur.samplePosReset.disabled = ui.blur.kernelSize.value == 0;
	});

	/* Render Mode */
	ui.rendering.modes.forEach(radio => {
		/* Force set to scene to fix a reload bug in Firefox Android */
		if (radio.value === "scene")
			radio.checked = true;
		radio.addEventListener('change', (event) => {
			ctx.mode = event.target.value;
			ui.rendering.lightBrightness.disabled = ctx.mode === "scene";
			ui.rendering.lightBrightnessReset.disabled = ctx.mode === "scene";
			if (!ui.rendering.animate.checked) redraw();
		});
	});

	ui.benchmark.button.addEventListener("click", () => {
		ctx.flags.benchMode = true;
		stopRendering();
		ui.display.spinner.style.display = "block";
		ui.benchmark.button.disabled = true;

		/* spin up the Worker (ES-module) */
		const worker = new Worker("./js/benchmark/boxBlurBenchmark.js", { type: "module" });

		/* pass all data the worker needs */
		worker.postMessage({
			iterations: ui.benchmark.iterOut.value,
			blurShaderSrc: boxBlurFrag,
			kernelSize: ui.blur.kernelSize.value,
			samplePos: ui.blur.samplePos.value
		});

		/* Benchmark */
		worker.addEventListener("message", (event) => {
			if (event.data.type !== "done") return;

			ui.benchmark.label.textContent = event.data.benchText;
			ui.benchmark.tapsCount.textContent = event.data.tapsCount;
			ui.benchmark.iterTime.textContent = event.data.iterationText;
			ui.benchmark.renderer.textContent = event.data.renderer;

			worker.terminate();
			ui.benchmark.button.disabled = false;
			ctx.flags.benchMode = false;
			if (ui.rendering.animate.checked)
				startRendering();
			else
				redraw();
		});
	});

	ui.benchmark.iterations.addEventListener("change", (event) => {
		ui.benchmark.iterOut.value = event.target.value;
		ui.benchmark.label.textContent = "Benchmark";
	});

	/* Draw Texture Shader */
	ctx.shd.scene = util.compileAndLinkShader(gl, circleAnimation, simpleTexture, ["offset", "radius"]);

	/* Draw bloom Shader */
	ctx.shd.bloom = util.compileAndLinkShader(gl, bloomVert, bloomFrag, ["texture", "textureAdd", "offset", "radius"]);

	/* Helper for recompilation */
	function reCompileBlurShader(blurSize) {
		ctx.shd.blur = util.compileAndLinkShader(gl, simpleQuad, boxBlurFrag, ["frameSizeRCP", "samplePosMult", "bloomStrength"], "#define KERNEL_SIZE " + blurSize + '\n');
	}

	/* Blur Shader */
	reCompileBlurShader(ui.blur.kernelSize.value)

	/* Send Unit code verts to the GPU */
	util.bindUnitQuad(gl);

	async function setupTextureBuffers() {
		ui.display.spinner.style.display = "block";
		ctx.flags.buffersInitialized = true;
		ctx.flags.initComplete = false;

		gl.deleteFramebuffer(ctx.fb.scene);
		gl.deleteFramebuffer(ctx.fb.final);
		[ctx.fb.scene, ctx.tex.frame] = util.setupFramebuffer(gl, canvas.width, canvas.height);
		[ctx.fb.final, ctx.tex.frameFinal] = util.setupFramebuffer(gl, canvas.width, canvas.height);

		let [base, selfIllum] = await Promise.all([
			fetch("/dual-kawase/img/SDR_No_Sprite.png"),
			fetch("/dual-kawase/img/Selfillumination.png")
		]);
		let [baseBlob, selfIllumBlob] = await Promise.all([base.blob(), selfIllum.blob()]);
		let [baseBitmap, selfIllumBitmap] = await Promise.all([
			createImageBitmap(baseBlob, { colorSpaceConversion: 'none', resizeWidth: canvas.width * 1.12, resizeHeight: canvas.height * 1.12, resizeQuality: "high" }),
			createImageBitmap(selfIllumBlob, { colorSpaceConversion: 'none', resizeWidth: canvas.width * 1.12, resizeHeight: canvas.height * 1.12, resizeQuality: "high" })
		]);

		ctx.tex.sdr = util.setupTexture(gl, null, null, ctx.tex.sdr, gl.LINEAR, baseBitmap);
		ctx.tex.selfIllum = util.setupTexture(gl, null, null, ctx.tex.selfIllum, gl.LINEAR, selfIllumBitmap);

		baseBitmap.close();
		selfIllumBitmap.close();

		ctx.flags.initComplete = true;
		ui.display.spinner.style.display = "none";
	}

	let prevNow = performance.now();
	let lastStatsUpdate = prevNow;
	let fpsEMA = 60;
	let msEMA = 16;

	async function redraw() {
		if (!ctx.flags.buffersInitialized)
			await setupTextureBuffers();
		if (!ctx.flags.initComplete)
			return;

		/* UI Stats */
		const KernelSizeSide = ui.blur.kernelSize.value * 2 + 1;
		const tapsNewText = (canvas.width * canvas.height * KernelSizeSide * KernelSizeSide / 1000000).toFixed(1) + " Million";
		ui.display.tapsCount.value = tapsNewText;
		ui.display.width.value = canvas.width;
		ui.display.height.value = canvas.height;

		/* Circle Motion */
		let radiusSwitch = ui.rendering.animate.checked ? radius : 0.0;
		let speed = (performance.now() / 10000) % Math.PI * 2;
		const offset = [radiusSwitch * Math.cos(speed), radiusSwitch * Math.sin(speed)];
		gl.useProgram(ctx.shd.scene.handle);
		const texture = ctx.mode == "scene" ? ctx.tex.sdr : ctx.tex.selfIllum;
		gl.activeTexture(gl.TEXTURE0);
		gl.bindTexture(gl.TEXTURE_2D, texture);
		gl.uniform2fv(ctx.shd.scene.uniforms.offset, offset);
		gl.uniform1f(ctx.shd.scene.uniforms.radius, radiusSwitch);

		/* Setup PostProcess Framebuffer */
		gl.bindFramebuffer(gl.FRAMEBUFFER, ctx.fb.scene);
		gl.viewport(0, 0, canvas.width, canvas.height);

		/* Draw Call */
		gl.drawArrays(gl.TRIANGLE_FAN, 0, 4);

		/* Box blur at native resolution */
		gl.useProgram(ctx.shd.blur.handle);
		const finalFB = ctx.mode == "bloom" ? ctx.fb.final : null;
		gl.bindFramebuffer(gl.FRAMEBUFFER, finalFB);
		gl.viewport(0, 0, canvas.width, canvas.height);
		gl.uniform1f(ctx.shd.blur.uniforms.bloomStrength, ctx.mode == "scene" ? 1.0 : ui.rendering.lightBrightness.value);
		gl.activeTexture(gl.TEXTURE0);
		gl.bindTexture(gl.TEXTURE_2D, ctx.tex.frame);
		gl.uniform2f(ctx.shd.blur.uniforms.frameSizeRCP, 1.0 / canvas.width, 1.0 / canvas.height);
		gl.uniform1f(ctx.shd.blur.uniforms.samplePosMult, ui.blur.samplePos.value);
		gl.drawArrays(gl.TRIANGLE_FAN, 0, 4);

		if (ctx.mode == "bloom") {
			/* Now do the bloom composition to the screen */
			gl.bindFramebuffer(gl.FRAMEBUFFER, null);
			gl.useProgram(ctx.shd.bloom.handle);

			gl.uniform2fv(ctx.shd.bloom.uniforms.offset, offset);
			gl.uniform1f(ctx.shd.bloom.uniforms.radius, radiusSwitch);

			gl.activeTexture(gl.TEXTURE0);
			gl.bindTexture(gl.TEXTURE_2D, ctx.tex.sdr);
			gl.uniform1i(ctx.shd.bloom.uniforms.texture, 0);

			gl.activeTexture(gl.TEXTURE1);
			gl.bindTexture(gl.TEXTURE_2D, ctx.tex.frameFinal);
			gl.uniform1i(ctx.shd.bloom.uniforms.textureAdd, 1);

			gl.drawArrays(gl.TRIANGLE_FAN, 0, 4);
		}

		/* Ask for CPU-GPU Sync to prevent overloading the GPU during compositing.
		   In reality this is more likely to be flush, but still, it seems to
		   help on multiple devices with during low FPS */
		gl.finish();

		const now = performance.now();
		let dt = now - prevNow;

		if (dt > 0) {
			const instFPS = 1000 / dt;
			const ALPHA = 0.05;
			fpsEMA = ALPHA * instFPS + (1 - ALPHA) * fpsEMA;
			msEMA = ALPHA * dt + (1 - ALPHA) * msEMA;
		}
		prevNow = now;

		if (ui.rendering.animate.checked && now - lastStatsUpdate >= 1000) {
			ui.display.fps.value = fpsEMA.toFixed(0);
			ui.display.ms.value = msEMA.toFixed(2);
			lastStatsUpdate = now;
		}
	}

	let animationFrameId;

	/* Render at Native Resolution */
	function nativeResize() {
		const [width, height] = util.getNativeSize(canvas);

		if (width && canvas.width !== width || height && canvas.height !== height) {
			canvas.width = width;
			canvas.height = height;

			if (!ctx.flags.benchMode) {
				stopRendering();
				startRendering();
			}
			if (!ui.rendering.animate.checked)
				redraw();
		}
	}

	/* Resize Event */
	nativeResize();

	let resizePending = false;
	window.addEventListener('resize', () => {
		if (!resizePending) {
			resizePending = true;
			requestAnimationFrame(() => {
				resizePending = false;
				nativeResize();
			});
		}
	});

	function renderLoop() {
		if (ctx.flags.isRendering && ui.rendering.animate.checked) {
			redraw();
			animationFrameId = requestAnimationFrame(renderLoop);
		}
	}

	function startRendering() {
		/* Start rendering, when canvas visible */
		ctx.flags.isRendering = true;
		renderLoop();
	}

	function stopRendering() {
		/* Stop another redraw being called */
		ctx.flags.isRendering = false;
		cancelAnimationFrame(animationFrameId);
		/* Force the rendering pipeline to sync with CPU before we mess with it */
		gl.finish();

		/* Delete the buffers to free up memory */
		gl.deleteTexture(ctx.tex.sdr); ctx.tex.sdr = null;
		gl.deleteTexture(ctx.tex.selfIllum); ctx.tex.selfIllum = null;
		gl.deleteTexture(ctx.tex.frame); ctx.tex.frame = null;
		gl.deleteTexture(ctx.tex.frameFinal); ctx.tex.frameFinal = null;
		gl.deleteFramebuffer(ctx.fb.scene); ctx.fb.scene = null;
		gl.deleteFramebuffer(ctx.fb.final); ctx.fb.final = null;
		ctx.flags.buffersInitialized = false;
		ctx.flags.initComplete = false;
		ui.display.fps.value = "-";
		ui.display.ms.value = "-";
	}

	function handleIntersection(entries) {
		entries.forEach(entry => {
			if (entry.isIntersecting) {
				if (!ctx.flags.isRendering && !ctx.flags.benchMode) startRendering();
			} else {
				stopRendering();
			}
		});
	}

	/* Only render when the canvas is actually on screen */
	let observer = new IntersectionObserver(handleIntersection);
	observer.observe(canvas);
}

Visually, the result doesn’t look very pleasing. The stronger the blur, the more “boxy” features of the image become. This is due to us reading and averaging the texture in a square shape. Especially in bloom mode, with strong lightBrightness and big kernelSize, lights become literally squares.

Performance is also really bad. With bigger kernelSizes, our Texture Taps count skyrockets and performance drops. Mobile devices will come to a slog. Even the worlds fastest PC graphics cards will fall below screen refresh-rate by cranking kernelSize and zooming the article on PC, thus raising canvas resolution.

We kinda failed on all fronts. It looks bad and runs bad.

Then, there’s this samplePosMultiplier. It seems to also seemingly increase blur strength, without increasing textureTaps or lowering performance (or lowering performance just a little on certain devices). But if we crank that too much, we get artifacts in the form of repeating patterns. Let’s play with a schematic example:

• The white center square represents the output pixel
• Grey squares are the pixels we would read, with the current kernelSize, with samplePosMult untouched
• the black dots are our actual texture reads per-output-pixel, our “sample” positions

On can say, that an image is a “continous 2D signal”. When we texture tap at a specific coordinate, we are sampling the “image signal” at that coordinate. As previously mentioned, we use UV coordinates and are not bound by concepts like “pixels position”. Where we place our samples is completely up to us.

A fundamental blur algorithm option is increasing the sample distance away from the center, thus increasing the amount of image we cover with our samples - more bang for your sample buck. This works by multiplying the offset distance. That is what samplePosMult does and is something you will have access to going forward.

Doing it too much, brings ugly repeating patterns. This of course leaves some fundamental questions, like where these artifacts come from and what it even means to read between two pixels. And on top of that we have to address performance and the boxyness of our blur! But first…

What even is a kernel? #

What we have created with our for-loop, is a convolution. Very simplified, in the context of image processing, it’s usually a square of numbers constructing an output pixel, by gathering and weighting pixels, that the square covers. The square is called a kernel and was the thing we visualized previously.

For blurs, the kernel weights must sum up to 1. If that were not the case, we would either brighten or darken the image. Ensuring that is the normalization step. In the box blur above, this happens by dividing the summed pixel color by totalSamples, the total amount of samples taken. A basic “calculate the average” expression.

The same can be expressed as weights of a kernel, a number multiplied with each sample at that position. Since the box blur weighs all sample the same regardless of position, all weights are the same. This is visualized next. The bigger the kernel size, the smaller the weights.

Kernels applied at the edges of our image will read from areas “outside” the image, with UV coordinates smaller than 0,0 and bigger than 1,1. Luckily, the GPU handles this for us and we are free to decide what happens to those outside samples, by setting the Texture Wrapping mode.

Among others, we can define a solid color to be used, or to “clamp” to the nearest edge’s color. If we choose a solid color, then we will get color bleeding at the edges. Thus for almost all post-processing use-cases, edge color clamping is used, as it prevents weird things happening at the edges. This article does too.

You may have noticed a black "blob" streaking with stronger blur levels along the bottom. Specifically here, it happens because the lines between the floor tiles align with the bottom edge, extending black color to infinity

Convolution as a mathematical concept is surprisingly deep and 3blue1brown has an excellent video on it, that even covers the image processing topic. Theoretically, we won’t depart from convolutions. We can dissect our code and express it as weights and kernels. With the for-loop box blur, that was quite easy!

On a practical level though, understanding where the convolution is, how many there are and what kernels are at play will become more and more difficult, once we leave the realm of classical blurs and consider the wider implications of reading between pixel bounds. But for now, we stay with the classics:

Gaussian Blur #

The most famous of blur algorithms is the Gaussian Blur. It uses the normal distribution, also known as the bell Curve to weight the samples inside the kernel, with a new variable sigma σ to control the flatness of the curve. Other than generating the kernel weights, the algorithm is identical to the box blur algorithm.

To calculate the weights for point (x,y), the above formula is used. The gaussian formula has a weighting multiplier 1/√(2πσ²). In the code, there is no such thing though. The formula expresses the gaussian curve as a continuous function going to infinity. But our code and its for-loop are different - discrete and finite.

float gaussianWeight(float x, float y, float sigma)
{
	/* (x² + y²) / 2 σ² */
	return exp(-(x * x + y * y) / (2.0 * sigma * sigma));
}

For clarity, the kernel is generated in the fragment shader. Normally, that should be avoided. Fragment shaders run per-output-pixel, but the kernel weights stay the same, making this inefficient.

Just like with the box blur, weights are summed up and divided at the end, instead of the term 1/√(2πσ²) precalculating weights. sigma controls the sharpness of the curve and thus the blur strength, but wasn’t that the job of kernelSize? Play around with all the values below and get a feel for how the various values behave.

Detailed Benchmark Results

・Resolution: 1600x1200

・Execution Time: ~

?

/ iteration

・Texture taps:

?

Million / iteration

・GPU info:

Blur Fragment Shader gaussianBlur.fs

/* Float precision to highp, if supported. Large Kernel Sizes result many color
   contributions and thus require the highest precision to avoid clipping.
   Required in WebGL 1 Shaders and depending on platform may have no effect */
precision highp float;
/* UV coordinates, passed in from the Vertex Shader */
varying vec2 uv;

uniform vec2 frameSizeRCP; /* Resolution Reciprocal */
uniform float samplePosMult; /* Multiply to push blur strength past the kernel size */
uniform float sigma;

uniform float bloomStrength; /* bloom strength */

uniform sampler2D texture;
/* `KERNEL_SIZE` added during compilation */
const int kernel_size = KERNEL_SIZE;

float gaussianWeight(float x, float y, float sigma)
{
	/* (x² + y²) / 2 σ² */
	return exp(-(x * x + y * y) / (2.0 * sigma * sigma));
}

void main() {
	/* Variable to hold our final color for the current pixel */
	vec4 sum = vec4(0.0);
	/* Sum of all weights */

	float weightSum = 0.0;
	/* How big one side of the sampled square is */
	const int size = 2 * kernel_size + 1;
	/* Total number of samples we are going to read */
	const float totalSamples = float(size * size);

	/* Read from the texture y amount of pixels above and below */
	for (int y = -kernel_size; y <= kernel_size; ++y) {
	/* Read from the texture x amount of pixels to the left and the right */
		for (int x = -kernel_size; x <= kernel_size; ++x) {

			/* Calculate the required weight */
			float w = gaussianWeight(float(x), float(y), sigma);
			/* Offset from the current pixel, indicating which pixel to read */
			vec2 offset  = vec2(x, y) * samplePosMult * frameSizeRCP;

			/* Read and sum up the contribution of that pixel, weighted */
			sum += texture2D(texture, uv + offset) * w;
			weightSum += w;
		}
	}

	/* Return the sum, divided by the number of samples (normalization) */
	gl_FragColor = (sum / weightSum) * bloomStrength;
}

WebGL Javascript gaussianBlur.js

import * as util from '../utility.js'

export async function setupGaussianBlur() {
	/* Init */
	const WebGLBox = document.getElementById('WebGLBox-GaussianBlur');
	const WebGLBoxDetail = document.getElementById('WebGLBox-GaussianBlurDetail');
	const canvas = WebGLBox.querySelector('canvas');

	/* Circle Rotation size */
	const radius = 0.12;

	/* Main WebGL 1.0 Context */
	const gl = canvas.getContext('webgl', {
		preserveDrawingBuffer: false,
		antialias: false,
		alpha: false,
	});

	/* State and Objects */
	const ctx = {
		/* State for of the Rendering */
		mode: "scene",
		flags: { isRendering: false, buffersInitialized: false, initComplete: false, benchMode: false },
		/* Textures */
		tex: { sdr: null, selfIllum: null, frame: null, frameFinal: null },
		/* Framebuffers */
		fb: { scene: null, final: null },
		/* Shaders and their respective Resource Locations */
		shd: {
			scene: { handle: null, uniforms: { offset: null, radius: null } },
			blur: { handle: null, uniforms: { frameSizeRCP: null, samplePosMult: null, sigma: null, bloomStrength: null } },
			bloom: { handle: null, uniforms: { offset: null, radius: null, texture: null, textureAdd: null } }
		}
	};

	/* UI Elements */
	const ui = {
		display: {
			spinner: canvas.parentElement.querySelector('svg', canvas.parentElement),
			contextLoss: canvas.parentElement.querySelector('div', canvas.parentElement),
			fps: WebGLBox.querySelector('#fps'),
			ms: WebGLBox.querySelector('#ms'),
			width: WebGLBox.querySelector('#width'),
			height: WebGLBox.querySelector('#height'),
			tapsCount: WebGLBox.querySelector('#taps'),
		},
		blur: {
			kernelSize: WebGLBox.querySelector('#sizeRange'),
			sigma: WebGLBox.querySelector('#sigmaRange'),
			samplePos: WebGLBox.querySelector('#samplePosRange'),
			samplePosReset: WebGLBox.querySelector('#samplePosRangeReset'),
		},
		rendering: {
			animate: WebGLBox.querySelector('#animateCheck'),
			modes: WebGLBox.querySelectorAll('input[type="radio"]'),
			lightBrightness: WebGLBox.querySelector('#lightBrightness'),
			lightBrightnessReset: WebGLBox.querySelector('#lightBrightnessReset'),
		},
		benchmark: {
			button: WebGLBox.querySelector('#benchmark'),
			label: WebGLBox.querySelector('#benchmarkLabel'),
			iterOut: WebGLBox.querySelector('#iterOut'),
			renderer: WebGLBoxDetail.querySelector('#renderer'),
			iterTime: WebGLBoxDetail.querySelector('#iterTime'),
			tapsCount: WebGLBoxDetail.querySelector('#tapsCountBench'),
			iterations: WebGLBox.querySelector('#iterations')
		}
	};

	/* Shaders */
	const circleAnimation = await util.fetchShader("shader/circleAnimation.vs");
	const simpleTexture = await util.fetchShader("shader/simpleTexture.fs");
	const bloomVert = await util.fetchShader("shader/bloom.vs");
	const bloomFrag = await util.fetchShader("shader/bloom.fs");
	const simpleQuad = await util.fetchShader("shader/simpleQuad.vs");
	const gaussianBlurFrag = await util.fetchShader("shader/gaussianBlur.fs");

	/* Elements that cause a redraw in the non-animation mode */
	ui.blur.kernelSize.addEventListener('input', () => { if (!ui.rendering.animate.checked) redraw() });
	ui.blur.sigma.addEventListener('input', () => { if (!ui.rendering.animate.checked) redraw() });
	ui.blur.samplePos.addEventListener('input', () => { if (!ui.rendering.animate.checked) redraw() });
	ui.rendering.lightBrightness.addEventListener('input', () => { if (!ui.rendering.animate.checked) redraw() });

	/* Events */
	ui.rendering.animate.addEventListener("change", () => {
		if (ui.rendering.animate.checked)
			startRendering();
		else {
			ui.display.fps.value = "-";
			ui.display.ms.value = "-";
			ctx.flags.isRendering = false;
			redraw()
		}
	});

	canvas.addEventListener("webglcontextlost", () => {
		ui.display.contextLoss.style.display = "block";
	});

	ui.blur.kernelSize.addEventListener('input', () => {
		reCompileBlurShader(ui.blur.kernelSize.value);
		ui.blur.samplePos.disabled = ui.blur.kernelSize.value == 0;
		ui.blur.samplePosReset.disabled = ui.blur.kernelSize.value == 0;
	});

	/* Render Mode */
	ui.rendering.modes.forEach(radio => {
		/* Force set to scene to fix a reload bug in Firefox Android */
		if (radio.value === "scene")
			radio.checked = true;
		radio.addEventListener('change', (event) => {
			ctx.mode = event.target.value;
			ui.rendering.lightBrightness.disabled = ctx.mode === "scene";
			ui.rendering.lightBrightnessReset.disabled = ctx.mode === "scene";
			if (!ui.rendering.animate.checked) redraw();
		});
	});

	ui.benchmark.button.addEventListener("click", () => {
		ctx.flags.benchMode = true;
		stopRendering();
		ui.display.spinner.style.display = "block";
		ui.benchmark.button.disabled = true;

		/* spin up the Worker (ES-module) */
		const worker = new Worker("./js/benchmark/gaussianBlurBenchmark.js", { type: "module" });

		/* pass all data the worker needs */
		worker.postMessage({
			iterations: ui.benchmark.iterOut.value,
			blurShaderSrc: gaussianBlurFrag,
			kernelSize: ui.blur.kernelSize.value,
			samplePos: ui.blur.samplePos.value,
			sigma: ui.blur.sigma.value
		});

		/* Benchmark */
		worker.addEventListener("message", (event) => {
			if (event.data.type !== "done") return;

			ui.benchmark.label.textContent = event.data.benchText;
			ui.benchmark.tapsCount.textContent = event.data.tapsCount;
			ui.benchmark.iterTime.textContent = event.data.iterationText;
			ui.benchmark.renderer.textContent = event.data.renderer;

			worker.terminate();
			ui.benchmark.button.disabled = false;
			ctx.flags.benchMode = false;
			if (ui.rendering.animate.checked)
				startRendering();
			else
				redraw();
		});
	});

	ui.benchmark.iterations.addEventListener("change", (event) => {
		ui.benchmark.iterOut.value = event.target.value;
		ui.benchmark.label.textContent = "Benchmark";
	});

	/* Draw Texture Shader */
	ctx.shd.scene = util.compileAndLinkShader(gl, circleAnimation, simpleTexture, ["offset", "radius"]);

	/* Draw bloom Shader */
	ctx.shd.bloom = util.compileAndLinkShader(gl, bloomVert, bloomFrag, ["texture", "textureAdd", "offset", "radius"]);


	/* Helper for recompilation */
	function reCompileBlurShader(blurSize) {
		ctx.shd.blur = util.compileAndLinkShader(gl, simpleQuad, gaussianBlurFrag, ["frameSizeRCP", "samplePosMult", "bloomStrength", "sigma"], "#define KERNEL_SIZE " + blurSize + '\n');
	}

	/* Blur Shader */
	reCompileBlurShader(ui.blur.kernelSize.value)

	/* Send Unit code verts to the GPU */
	util.bindUnitQuad(gl);

	async function setupTextureBuffers() {
		ui.display.spinner.style.display = "block";
		ctx.flags.buffersInitialized = true;
		ctx.flags.initComplete = false;

		gl.deleteFramebuffer(ctx.fb.scene);
		gl.deleteFramebuffer(ctx.fb.final);
		[ctx.fb.scene, ctx.tex.frame] = util.setupFramebuffer(gl, canvas.width, canvas.height);
		[ctx.fb.final, ctx.tex.frameFinal] = util.setupFramebuffer(gl, canvas.width, canvas.height);


		let [base, selfIllum] = await Promise.all([
			fetch("/dual-kawase/img/SDR_No_Sprite.png"),
			fetch("/dual-kawase/img/Selfillumination.png")
		]);
		let [baseBlob, selfIllumBlob] = await Promise.all([base.blob(), selfIllum.blob()]);
		let [baseBitmap, selfIllumBitmap] = await Promise.all([
			createImageBitmap(baseBlob, { colorSpaceConversion: 'none', resizeWidth: canvas.width * 1.12, resizeHeight: canvas.height * 1.12, resizeQuality: "high" }),
			createImageBitmap(selfIllumBlob, { colorSpaceConversion: 'none', resizeWidth: canvas.width * 1.12, resizeHeight: canvas.height * 1.12, resizeQuality: "high" })
		]);

		ctx.tex.sdr = util.setupTexture(gl, null, null, ctx.tex.sdr, gl.LINEAR, baseBitmap);
		ctx.tex.selfIllum = util.setupTexture(gl, null, null, ctx.tex.selfIllum, gl.LINEAR, selfIllumBitmap);

		baseBitmap.close();
		selfIllumBitmap.close();

		ctx.flags.initComplete = true;
		ui.display.spinner.style.display = "none";
	}

	let prevNow = performance.now();
	let lastStatsUpdate = prevNow;
	let fpsEMA = 60;
	let msEMA = 16;

	async function redraw() {
		if (!ctx.flags.buffersInitialized)
			await setupTextureBuffers();
		if (!ctx.flags.initComplete)
			return;

		/* UI Stats */
		const KernelSizeSide = ui.blur.kernelSize.value * 2 + 1;
		const tapsNewText = (canvas.width * canvas.height * KernelSizeSide * KernelSizeSide / 1000000).toFixed(1) + " Million";
		ui.display.tapsCount.value = tapsNewText;
		ui.display.width.value = canvas.width;
		ui.display.height.value = canvas.height;

		/* Circle Motion */
		let radiusSwitch = ui.rendering.animate.checked ? radius : 0.0;
		let speed = (performance.now() / 10000) % Math.PI * 2;
		const offset = [radiusSwitch * Math.cos(speed), radiusSwitch * Math.sin(speed)];
		gl.useProgram(ctx.shd.scene.handle);
		const texture = ctx.mode == "scene" ? ctx.tex.sdr : ctx.tex.selfIllum;
		gl.activeTexture(gl.TEXTURE0);
		gl.bindTexture(gl.TEXTURE_2D, texture);
		gl.uniform2fv(ctx.shd.scene.uniforms.offset, offset);
		gl.uniform1f(ctx.shd.scene.uniforms.radius, radiusSwitch);

		/* Setup PostProcess Framebuffer */
		gl.bindFramebuffer(gl.FRAMEBUFFER, ctx.fb.scene);
		gl.viewport(0, 0, canvas.width, canvas.height);

		/* Draw Call */
		gl.drawArrays(gl.TRIANGLE_FAN, 0, 4);

		/* Gaussian blur at native resolution */
		gl.useProgram(ctx.shd.blur.handle);
		const finalFB = ctx.mode == "bloom" ? ctx.fb.final : null;
		gl.bindFramebuffer(gl.FRAMEBUFFER, finalFB);
		gl.viewport(0, 0, canvas.width, canvas.height);
		gl.uniform1f(ctx.shd.blur.uniforms.bloomStrength, ctx.mode == "scene" ? 1.0 : ui.rendering.lightBrightness.value);
		gl.activeTexture(gl.TEXTURE0);
		gl.bindTexture(gl.TEXTURE_2D, ctx.tex.frame);
		gl.uniform2f(ctx.shd.blur.uniforms.frameSizeRCP, 1.0 / canvas.width, 1.0 / canvas.height);
		gl.uniform1f(ctx.shd.blur.uniforms.samplePosMult, ui.blur.samplePos.value);
		gl.uniform1f(ctx.shd.blur.uniforms.sigma, Math.max(ui.blur.kernelSize.value / ui.blur.sigma.value, 0.001));
		gl.drawArrays(gl.TRIANGLE_FAN, 0, 4);

		if (ctx.mode == "bloom") {
			/* Now do the bloom composition to the screen */
			gl.bindFramebuffer(gl.FRAMEBUFFER, null);
			gl.useProgram(ctx.shd.bloom.handle);

			gl.uniform2fv(ctx.shd.bloom.uniforms.offset, offset);
			gl.uniform1f(ctx.shd.bloom.uniforms.radius, radiusSwitch);

			gl.activeTexture(gl.TEXTURE0);
			gl.bindTexture(gl.TEXTURE_2D, ctx.tex.sdr);
			gl.uniform1i(ctx.shd.bloom.uniforms.texture, 0);

			gl.activeTexture(gl.TEXTURE1);
			gl.bindTexture(gl.TEXTURE_2D, ctx.tex.frameFinal);
			gl.uniform1i(ctx.shd.bloom.uniforms.textureAdd, 1);

			gl.drawArrays(gl.TRIANGLE_FAN, 0, 4);
		}

		/* Ask for CPU-GPU Sync to prevent overloading the GPU during compositing.
		   In reality this is more likely to be flush, but still, it seems to
		   help on multiple devices with during low FPS */
		gl.finish();

		const now = performance.now();
		let dt = now - prevNow;

		if (dt > 0) {
			const instFPS = 1000 / dt;
			const ALPHA = 0.05;
			fpsEMA = ALPHA * instFPS + (1 - ALPHA) * fpsEMA;
			msEMA = ALPHA * dt + (1 - ALPHA) * msEMA;
		}
		prevNow = now;

		if (ui.rendering.animate.checked && now - lastStatsUpdate >= 1000) {
			ui.display.fps.value = fpsEMA.toFixed(0);
			ui.display.ms.value = msEMA.toFixed(2);
			lastStatsUpdate = now;
		}
	}

	let animationFrameId;

	/* Render at Native Resolution */
	function nativeResize() {
		const [width, height] = util.getNativeSize(canvas);

		if (width && canvas.width !== width || height && canvas.height !== height) {
			canvas.width = width;
			canvas.height = height;

			if (!ctx.flags.benchMode) {
				stopRendering();
				startRendering();
			}
			if (!ui.rendering.animate.checked)
				redraw();
		}
	}

	/* Resize Event */
	nativeResize();

	let resizePending = false;
	window.addEventListener('resize', () => {
		if (!resizePending) {
			resizePending = true;
			requestAnimationFrame(() => {
				resizePending = false;
				nativeResize();
			});
		}
	});

	function renderLoop() {
		if (ctx.flags.isRendering && ui.rendering.animate.checked) {
			redraw();
			animationFrameId = requestAnimationFrame(renderLoop);
		}
	}

	function startRendering() {
		/* Start rendering, when canvas visible */
		ctx.flags.isRendering = true;
		renderLoop();
	}

	function stopRendering() {
		/* Stop another redraw being called */
		ctx.flags.isRendering = false;
		cancelAnimationFrame(animationFrameId);
		/* Force the rendering pipeline to sync with CPU before we mess with it */
		gl.finish();

		/* Delete the buffers to free up memory */
		gl.deleteTexture(ctx.tex.sdr); ctx.tex.sdr = null;
		gl.deleteTexture(ctx.tex.selfIllum); ctx.tex.selfIllum = null;
		gl.deleteTexture(ctx.tex.frame); ctx.tex.frame = null;
		gl.deleteTexture(ctx.tex.frameFinal); ctx.tex.frameFinal = null;
		gl.deleteFramebuffer(ctx.fb.scene); ctx.fb.scene = null;
		gl.deleteFramebuffer(ctx.fb.final); ctx.fb.final = null;
		ctx.flags.buffersInitialized = false;
		ctx.flags.initComplete = false;
		ui.display.fps.value = "-";
		ui.display.ms.value = "-";
	}

	function handleIntersection(entries) {
		entries.forEach(entry => {
			if (entry.isIntersecting) {
				if (!ctx.flags.isRendering && !ctx.flags.benchMode) startRendering();
			} else {
				stopRendering();
			}
		});
	}

	/* Only render when the canvas is actually on screen */
	let observer = new IntersectionObserver(handleIntersection);
	observer.observe(canvas);
}

The blur looks way smoother than our previous box blur, with things generally taking on a “rounder” appearance, due to the bell curve’s smooth signal response. That is, unless you move the sigma slider down. If you move sigma too low, you will get our previous box blur like artifacts again.

Let’s clear up what the values actually represent and how they interact. The following visualization shows the kernel with its weights expressed as height in an Isometric Dimetric perspective projection. There are two different interaction modes with sigma when changing kernelSize and two ways to express sigma.

sigma describes the flatness of our mathematical curve, a curve going to infinity. But our algorithm has a limited kernelSize. Where the kernel stops, no more pixel contributions occur, leading to box-blur-like artifacts due to the cut-off. In the context of image processing, there are two ways to setup a gaussian blur…

A small sigma, thus a flat bell curve, paired with a small kernel size effectively is a box blur, with the weights making the kernel box-shaped.

… way 1: Absolute Sigma. sigma is an absolute value in pixels independent of kernelSize, with kernelSize acting as a “window into the curve” or way 2: sigma is expressed relative to the current kernelSize. For practical reasons (finicky sliders) the relative to kernelSize mode is used everywhere.

Eitherway, the infinite gaussian curve will have a cut-off somewhere. sigma too small? - We get box blur like artifacts. sigma too big? - We waste blur efficiency, as the same perceived blur strength requires bigger kernels, thus bigger for-loops with lower performance. An artistic trade-off every piece of software has to make.

An optimal kernel would be one, where the outer weights are almost zero. Thus, if we increased kernelSize in Absolute Sigma mode by one, it would make close to no more visual difference.

There are other ways of creating blur kernels, with other properties. One way is to follow Pascal’s triangle to get a set of predefined kernel sizes and weights. These are called Binomial Filters and lock us into specific “kernel presets”, but solve the infinity vs cut-off dilemma, by moving weights to zero within the sampling window.

Binomial Kernels are also Gaussian-like in their frequency response. We won’t expand on these further, just know that we can choose kernels by different mathematical criteria, chasing different signal response characteristics. But speaking of which, what even is Gaussian Like? Why do we care?

What is Gaussian-like? #

In Post-Processing Blur algorithms you generally find two categories. Bokeh Blurs and Gaussian-Like Blurs. The gaussian is chosen for its natural appearance, its ability to smooth colors without “standout features”. Gaussian Blurs are generally used as an ingredient in an overarching visual effect, be it frosted glass Interfaces or Bloom.

In contrast to that, when emulating lenses and or creating Depth of Field, is “Bokeh Blur” - also known as “Lens Blur” or “Cinematic Blur”. This type of blur is the target visual effect. The challenges and approaches are very much related, but algorithms used differ.

Algorithms get really creative in this space, all with different trade-offs and visuals. Some sample using a poission disk distribution and some have cool out of the box thinking: Computerphile covered a comlex numbers based approach to creating Bokeh Blurs, a fascinating number theory cross-over.

This article though doesn’t care about these stylistics approaches. We are here to chase a basic building block of graphics programming and realtime visual effects, a “Gaussian-Like” with good performance. Speaking of which!

Performance #

The main motivator of our journey here, is the chase of realtime performance. Everything we do must happen within few milliseconds. The expected performance of an algorithm and the practical cost once placed in the graphics pipeline, are sometimes surprisingly different numbers though. Gotta measure!

This chapter is about a very technical motivation. If you don't care about how fast a GPU does what it does, feel free to skip this section.

With performance being such a driving motivator, it would be a shame if we couldn’t measure it in this article. Each WebGL Box has a benchmark function, which blurs random noise at a fixed resolution of 1600x1200 with the respective blur settings you chose and a fixed iteration count workload, a feature hidden so far.

Realtime graphics programming is sometimes more about measuring than programming.

Benchmarking is best done by measuring shader execution time. This can be done in the browser reliably, but only on some platforms. No way exists to do so across all platforms. Luckily, there is the classic method of “stalling the graphics pipeline”, forcing a wait until all commands finish, a moment in time we can measure.

Across all platforms a stall is guaranteed to occur on command gl.readPixels(). Interestingly, the standards conform command for this: gl.finish() is simply ignored by mobile apple devices.

Below is a button, that unlocks this benchmarking feature, unhiding a benchmark button and Detailed Benchmark Results section under each blur. This allows you to start a benchmark with a preset workload, on a separate Browser Worker. There is only one issue: Browsers get very angry if you full-load the GPU this way.

If the graphics pipeline is doing work without reporting back (called “yielding”) to the browser for too long, browsers will simply kill all GPU access for the whole page, until tab reload. If we yield back, then the measured results are useless and from inside WebGL, we can’t stop the GPU, once its commands are issued.

⚠️ Especially on mobile: please increase kernelSize and iterations slowly. The previous algorithms have bad kernelSize performance scaling on purpose, be especially careful with them.

Stay below 2 seconds of execution time, or the browser will lock GPU access for the page, disabling all blur examples, until a browser restart is performed. On iOS Safari this requires a trip to the App Switcher, a page reload won't be enough.

iOS and iPad OS are especially strict, will keep GPU access disabled, even on Tab Reload. You will have go to the App Switcher (Double Tap Home Button), Swipe Safari Up to close it and relaunch it from scratch.

What are we optimizing for? #

With the above Box Blur and above Gaussian Blur, you will measure performance scaling very badly with kernelSize. Expressed in the Big O notation, it has a performance scaling of O(pixelCount * kernelSize²). Quadratic scaling of required texture taps in terms of kernelSize. We need to tackle this going forward.

Especially dedicated Laptop GPUs are slow to get out of their lower power states. Pressing the benchmark button multiple times in a row may result in the performance numbers getting better.

Despite the gaussian blur calculating the kernel completely from scratch on every single pixel in our implementation, the performance of the box blur and gaussian blur are very close to each other at higher iteration counts. In fact, by precalculating the those kernels we could performance match both.

But isn't gaussian blur is more complicated algorithm?

As opposed to chips from decades ago, modern graphics cards have very fast arithmetic, but comparatively slow memory access times. With workloads like these, the slowest thing becomes the memory access, in our case the texture taps. The more taps, the slower the algorithm.

Our blurs perform a dependant texture read, a graphics programming sin. This is when texture coordinates are determined during shader execution, which opts out of a many automated shader optimizations.

Especially on personal computers, you may also have noticed that increasing samplePosMultiplier will negatively impact performance (up to a point), even though the required texture taps stay the same.

This is due hardware texture caches accelerating texture reads which are spatially close together and not being able to do so effectively, if the texture reads are all too far apart. Platform dependant tools like Nvidia NSight can measure GPU cache utilization. The browser cannot.

These are key numbers graphics programmers chase when writing fragment shaders: Texture Taps and Cache Utilization. There is another one, we will get into in a moment. Clearly, our Blurs are slow. Time for a speed up!

Separable Gaussian Blur #

We have not yet left the classics of blur algorithms. One fundamental concept left on the table is “convolution separability”. Certain Convolutions like our Box Blur, our Gaussian Blur and the Binominal filtering mentioned in passing previously can all be performed in two separate passes, by two separate 1D Kernels.

Not all convolutions are separable. In the context of graphics programming: If you can express the kernel weights as a formula with axes X, Y and factor-out both X and Y into two separate formulas, then you have gained separability of a 2D kernel and can perform the convolution in two passes, massively saving on texture taps.

Some big budget video games have used effects with kernels that are not separable, but did it anyway in two passes + 1D Kernel for the performance gain, with the resulting artifacts being deemed not too bad.

Computerphile covered the concept of separability in the context of 2D image processing really well, if you are interested in a more formal explanation.

Here is our Gaussian Blur, but expressed as a separable Version. You can see just Pass 1 and Pass 2 in isolation or see the final result. Same visual quality as our Gaussian Blur, same dials, but massively faster, with no more quadratic scaling of required texture taps.

Detailed Benchmark Results

・Resolution: 1600x1200

・Pass Mode: ?

・Execution Time: ~

?

/ iteration

・Texture taps:

?

Million / iteration

・GPU info:

Blur Fragment Shader gaussianSeparableBlur.fs

/* Float precision to highp, if supported. Large Kernel Sizes result many color
   contributions and thus require the highest precision to avoid clipping.
   Required in WebGL 1 Shaders and depending on platform may have no effect */
precision highp float;
/* UV coordinates, passed in from the Vertex Shader */
varying vec2 uv;

uniform vec2 frameSizeRCP; /* Resolution Reciprocal */
uniform float samplePosMult; /* Multiply to push blur strength past the kernel size */
uniform float sigma;
uniform vec2 direction; /* Direction vector: (1,0) for horizontal, (0,1) for vertical */

uniform float bloomStrength; /* bloom strength */

uniform sampler2D texture;
/* `KERNEL_SIZE` added during compilation */
const int kernel_size = KERNEL_SIZE;

float gaussianWeight(float x, float sigma)
{
	/* x² / 2 σ² */
	return exp(-(x * x) / (2.0 * sigma * sigma));
}

void main() {
	/* Variable to hold our final color for the current pixel */
	vec4 sum = vec4(0.0);
	/* Sum of all weights */
	float weightSum = 0.0;
	
	/* How big one side of the sampled line is */
	const int size = 2 * kernel_size + 1;

	/* Sample along the direction vector (horizontal or vertical) */
	for (int i = -kernel_size; i <= kernel_size; ++i) {
		/* Calculate the required weight for this 1D sample */
		float w = gaussianWeight(float(i), sigma);
		
		/* Offset from the current pixel along the specified direction */
		vec2 offset = vec2(i) * direction * samplePosMult * frameSizeRCP;

		/* Read and sum up the contribution of that pixel, weighted */
		sum += texture2D(texture, uv + offset) * w;
		weightSum += w;
	}

	/* Return the sum, divided by the total weight (normalization) */
	gl_FragColor = (sum / weightSum) * bloomStrength;
}

WebGL Javascript gaussianSeparableBlur.js

import * as util from '../utility.js'

export async function setupGaussianSeparableBlur() {
	/* Init */
	const WebGLBox = document.getElementById('WebGLBox-GaussianSeparableBlur');
	const canvas = WebGLBox.querySelector('canvas');

	/* Circle Rotation size */
	const radius = 0.12;

	/* Main WebGL 1.0 Context */
	const gl = canvas.getContext('webgl', {
		preserveDrawingBuffer: false,
		antialias: false,
		alpha: false,
	});

	/* State and Objects */
	const ctx = {
		/* State for of the Rendering */
		mode: "scene",
		passMode: "pass1",
		flags: { isRendering: false, buffersInitialized: false, initComplete: false, benchMode: false },
		/* Textures */
		tex: { sdr: null, selfIllum: null, frame: null, frameIntermediate: null, frameFinal: null },
		/* Framebuffers */
		fb: { scene: null, intermediate: null, final: null },
		/* Shaders and their respective Resource Locations */
		shd: {
			scene: { handle: null, uniforms: { offset: null, radius: null } },
			blur: { handle: null, uniforms: { frameSizeRCP: null, samplePosMult: null, sigma: null, bloomStrength: null, direction: null } },
			bloom: { handle: null, uniforms: { offset: null, radius: null, texture: null, textureAdd: null } }
		}
	};

	/* UI Elements */
	const ui = {
		display: {
			spinner: canvas.parentElement.querySelector('svg', canvas.parentElement),
			contextLoss: canvas.parentElement.querySelector('div', canvas.parentElement),
			fps: WebGLBox.querySelector('#fps'),
			ms: WebGLBox.querySelector('#ms'),
			width: WebGLBox.querySelector('#width'),
			height: WebGLBox.querySelector('#height'),
			tapsCount: WebGLBox.querySelector('#taps'),
		},
		blur: {
			kernelSize: WebGLBox.querySelector('#sizeRange'),
			sigma: WebGLBox.querySelector('#sigmaRange'),
			samplePos: WebGLBox.querySelector('#samplePosRange'),
			samplePosReset: WebGLBox.querySelector('#samplePosRangeReset'),
		},
		rendering: {
			animate: WebGLBox.querySelector('#animateCheck'),
			modes: WebGLBox.querySelectorAll('input[name="modeGaussSep"]'),
			passModes: WebGLBox.querySelectorAll('input[name="passMode"]'),
			lightBrightness: WebGLBox.querySelector('#lightBrightness'),
			lightBrightnessReset: WebGLBox.querySelector('#lightBrightnessReset'),
		},
		benchmark: {
			button: WebGLBox.querySelector('#benchmark'),
			label: WebGLBox.querySelector('#benchmarkLabel'),
			iterOut: WebGLBox.querySelector('#iterOut'),
			renderer: document.getElementById('WebGLBox-GaussianSeparableBlurDetail').querySelector('#renderer'),
			passMode: document.getElementById('WebGLBox-GaussianSeparableBlurDetail').querySelector('#passMode'),
			iterTime: document.getElementById('WebGLBox-GaussianSeparableBlurDetail').querySelector('#iterTime'),
			tapsCount: document.getElementById('WebGLBox-GaussianSeparableBlurDetail').querySelector('#tapsCountBench'),
			iterations: WebGLBox.querySelector('#iterations')
		}
	};

	/* Shaders */
	const circleAnimation = await util.fetchShader("shader/circleAnimation.vs");
	const simpleTexture = await util.fetchShader("shader/simpleTexture.fs");
	const bloomVert = await util.fetchShader("shader/bloom.vs");
	const bloomFrag = await util.fetchShader("shader/bloom.fs");
	const simpleQuad = await util.fetchShader("shader/simpleQuad.vs");
	const gaussianBlurFrag = await util.fetchShader("shader/gaussianBlurSeparable.fs");

	/* Elements that cause a redraw in the non-animation mode */
	ui.blur.kernelSize.addEventListener('input', () => { if (!ui.rendering.animate.checked) redraw() });
	ui.blur.sigma.addEventListener('input', () => { if (!ui.rendering.animate.checked) redraw() });
	ui.blur.samplePos.addEventListener('input', () => { if (!ui.rendering.animate.checked) redraw() });
	ui.rendering.lightBrightness.addEventListener('input', () => { if (!ui.rendering.animate.checked) redraw() });

	/* Events */
	ui.rendering.animate.addEventListener("change", () => {
		if (ui.rendering.animate.checked)
			startRendering();
		else {
			ui.display.fps.value = "-";
			ui.display.ms.value = "-";
			ctx.flags.isRendering = false;
			redraw()
		}
	});

	canvas.addEventListener("webglcontextlost", () => {
		ui.display.contextLoss.style.display = "block";
	});

	ui.blur.kernelSize.addEventListener('input', () => {
		reCompileBlurShader(ui.blur.kernelSize.value);
		ui.blur.samplePos.disabled = ui.blur.kernelSize.value == 0;
		ui.blur.samplePosReset.disabled = ui.blur.kernelSize.value == 0;
	});

	/* Render Mode */
	ui.rendering.modes.forEach(radio => {
		/* Force set to scene to fix a reload bug in Firefox Android */
		if (radio.value === "scene")
			radio.checked = true;
		radio.addEventListener('change', (event) => {
			ctx.mode = event.target.value;
			ui.rendering.lightBrightness.disabled = ctx.mode === "scene";
			ui.rendering.lightBrightnessReset.disabled = ctx.mode === "scene";
			if (!ui.rendering.animate.checked) redraw();
		});
	});
	
	/* Pass Mode */
	ui.rendering.passModes.forEach(radio => {
		/* Force set to pass1 to fix a reload bug in Firefox Android */
		if (radio.value === "pass1")
			radio.checked = true;
		radio.addEventListener('change', (event) => {
			ctx.passMode = event.target.value;
			if (!ui.rendering.animate.checked) redraw();
		});
	});

	ui.benchmark.button.addEventListener("click", () => {
		ctx.flags.benchMode = true;
		stopRendering();
		ui.display.spinner.style.display = "block";
		ui.benchmark.button.disabled = true;

		/* spin up the Worker (ES-module) */
		const worker = new Worker("./js/benchmark/gaussianSeparableBlurBenchmark.js", { type: "module" });

		/* pass all data the worker needs */
		worker.postMessage({
			iterations: ui.benchmark.iterOut.value,
			blurShaderSrc: gaussianBlurFrag,
			kernelSize: ui.blur.kernelSize.value,
			samplePos: ui.blur.samplePos.value,
			sigma: ui.blur.sigma.value,
			passMode: ctx.passMode
		});

		/* Benchmark */
		worker.addEventListener("message", (event) => {
			if (event.data.type !== "done") return;

			ui.benchmark.label.textContent = event.data.benchText;
			ui.benchmark.tapsCount.textContent = event.data.tapsCount;
			ui.benchmark.iterTime.textContent = event.data.iterationText;
			ui.benchmark.renderer.textContent = event.data.renderer;
			ui.benchmark.passMode.textContent = event.data.passMode;

			worker.terminate();
			ui.benchmark.button.disabled = false;
			ctx.flags.benchMode = false;
			if (ui.rendering.animate.checked)
				startRendering();
			else
				redraw();
		});
	});

	ui.benchmark.iterations.addEventListener("change", (event) => {
		ui.benchmark.iterOut.value = event.target.value;
		ui.benchmark.label.textContent = "Benchmark";
	});

	/* Draw Texture Shader */
	ctx.shd.scene = util.compileAndLinkShader(gl, circleAnimation, simpleTexture, ["offset", "radius"]);

	/* Draw bloom Shader */
	ctx.shd.bloom = util.compileAndLinkShader(gl, bloomVert, bloomFrag, ["texture", "textureAdd", "offset", "radius"]);


	/* Helper for recompilation */
	function reCompileBlurShader(blurSize) {
		ctx.shd.blur = util.compileAndLinkShader(gl, simpleQuad, gaussianBlurFrag, ["frameSizeRCP", "samplePosMult", "bloomStrength", "sigma", "direction"], "#define KERNEL_SIZE " + blurSize + '\n');
	}

	/* Blur Shader */
	reCompileBlurShader(ui.blur.kernelSize.value)

	/* Send Unit code verts to the GPU */
	util.bindUnitQuad(gl);

	async function setupTextureBuffers() {
		ui.display.spinner.style.display = "block";
		ctx.flags.buffersInitialized = true;
		ctx.flags.initComplete = false;

		gl.deleteFramebuffer(ctx.fb.scene);
		gl.deleteFramebuffer(ctx.fb.intermediate);
		gl.deleteFramebuffer(ctx.fb.final);
		[ctx.fb.scene, ctx.tex.frame] = util.setupFramebuffer(gl, canvas.width, canvas.height);
		[ctx.fb.intermediate, ctx.tex.frameIntermediate] = util.setupFramebuffer(gl, canvas.width, canvas.height);
		[ctx.fb.final, ctx.tex.frameFinal] = util.setupFramebuffer(gl, canvas.width, canvas.height);

		// Clear intermediate texture to prevent lazy initialization warnings
		gl.bindFramebuffer(gl.FRAMEBUFFER, ctx.fb.intermediate);
		gl.clearColor(0.0, 0.0, 0.0, 1.0);
		gl.clear(gl.COLOR_BUFFER_BIT);


		let [base, selfIllum] = await Promise.all([
			fetch("/dual-kawase/img/SDR_No_Sprite.png"),
			fetch("/dual-kawase/img/Selfillumination.png")
		]);
		let [baseBlob, selfIllumBlob] = await Promise.all([base.blob(), selfIllum.blob()]);
		let [baseBitmap, selfIllumBitmap] = await Promise.all([
			createImageBitmap(baseBlob, { colorSpaceConversion: 'none', resizeWidth: canvas.width * 1.12, resizeHeight: canvas.height * 1.12, resizeQuality: "high" }),
			createImageBitmap(selfIllumBlob, { colorSpaceConversion: 'none', resizeWidth: canvas.width * 1.12, resizeHeight: canvas.height * 1.12, resizeQuality: "high" })
		]);

		ctx.tex.sdr = util.setupTexture(gl, null, null, ctx.tex.sdr, gl.LINEAR, baseBitmap);
		ctx.tex.selfIllum = util.setupTexture(gl, null, null, ctx.tex.selfIllum, gl.LINEAR, selfIllumBitmap);

		baseBitmap.close();
		selfIllumBitmap.close();

		ctx.flags.initComplete = true;
		ui.display.spinner.style.display = "none";
	}

	let prevNow = performance.now();
	let lastStatsUpdate = prevNow;
	let fpsEMA = 60;
	let msEMA = 16;

	async function redraw() {
		if (!ctx.flags.buffersInitialized)
			await setupTextureBuffers();
		if (!ctx.flags.initComplete)
			return;

		/* UI Stats */
		const KernelSizeSide = ui.blur.kernelSize.value * 2 + 1;
		/* Separable blur: pass1/pass2 = 1 pass, combined = 2 passes */
		const samplesPerPixel = ctx.passMode == "combined" ? KernelSizeSide * 2 : KernelSizeSide;
		const tapsNewText = (canvas.width * canvas.height * samplesPerPixel / 1000000).toFixed(1) + " Million";
		ui.display.tapsCount.value = tapsNewText;
		ui.display.width.value = canvas.width;
		ui.display.height.value = canvas.height;

		/* Circle Motion */
		let radiusSwitch = ui.rendering.animate.checked ? radius : 0.0;
		let speed = (performance.now() / 10000) % Math.PI * 2;
		const offset = [radiusSwitch * Math.cos(speed), radiusSwitch * Math.sin(speed)];
		gl.useProgram(ctx.shd.scene.handle);
		const texture = ctx.mode == "scene" ? ctx.tex.sdr : ctx.tex.selfIllum;
		gl.activeTexture(gl.TEXTURE0);
		gl.bindTexture(gl.TEXTURE_2D, texture);
		gl.uniform2fv(ctx.shd.scene.uniforms.offset, offset);
		gl.uniform1f(ctx.shd.scene.uniforms.radius, radiusSwitch);

		/* Setup PostProcess Framebuffer */
		gl.bindFramebuffer(gl.FRAMEBUFFER, ctx.fb.scene);
		gl.viewport(0, 0, canvas.width, canvas.height);

		/* Draw Call */
		gl.drawArrays(gl.TRIANGLE_FAN, 0, 4);

		/* Separable Gaussian blur implementation */
		gl.useProgram(ctx.shd.blur.handle);
		
		if (ctx.passMode == "pass1") {
			/* Pass 1 only: Horizontal blur directly to screen */
			const finalFB = ctx.mode == "bloom" ? ctx.fb.final : null;
			gl.bindFramebuffer(gl.FRAMEBUFFER, finalFB);
			gl.viewport(0, 0, canvas.width, canvas.height);
			gl.uniform1f(ctx.shd.blur.uniforms.bloomStrength, ctx.mode == "scene" ? 1.0 : ui.rendering.lightBrightness.value);
			gl.uniform2f(ctx.shd.blur.uniforms.direction, 1.0, 0.0); // Horizontal direction
			gl.activeTexture(gl.TEXTURE0);
			gl.bindTexture(gl.TEXTURE_2D, ctx.tex.frame);
			gl.uniform2f(ctx.shd.blur.uniforms.frameSizeRCP, 1.0 / canvas.width, 1.0 / canvas.height);
			gl.uniform1f(ctx.shd.blur.uniforms.samplePosMult, ui.blur.samplePos.value);
			gl.uniform1f(ctx.shd.blur.uniforms.sigma, Math.max(ui.blur.kernelSize.value / ui.blur.sigma.value, 0.001));
			gl.drawArrays(gl.TRIANGLE_FAN, 0, 4);
		} else if (ctx.passMode == "pass2") {
			/* Pass 2 only: Vertical blur directly to screen */
			const finalFB = ctx.mode == "bloom" ? ctx.fb.final : null;
			gl.bindFramebuffer(gl.FRAMEBUFFER, finalFB);
			gl.viewport(0, 0, canvas.width, canvas.height);
			gl.uniform1f(ctx.shd.blur.uniforms.bloomStrength, ctx.mode == "scene" ? 1.0 : ui.rendering.lightBrightness.value);
			gl.uniform2f(ctx.shd.blur.uniforms.direction, 0.0, 1.0); // Vertical direction
			gl.activeTexture(gl.TEXTURE0);
			gl.bindTexture(gl.TEXTURE_2D, ctx.tex.frame);
			gl.uniform2f(ctx.shd.blur.uniforms.frameSizeRCP, 1.0 / canvas.width, 1.0 / canvas.height);
			gl.uniform1f(ctx.shd.blur.uniforms.samplePosMult, ui.blur.samplePos.value);
			gl.uniform1f(ctx.shd.blur.uniforms.sigma, Math.max(ui.blur.kernelSize.value / ui.blur.sigma.value, 0.001));
			gl.drawArrays(gl.TRIANGLE_FAN, 0, 4);
		} else {
			/* Combined: Two-pass separable blur */
			/* Pass 1: Horizontal blur to intermediate buffer */
			gl.bindFramebuffer(gl.FRAMEBUFFER, ctx.fb.intermediate);
			gl.viewport(0, 0, canvas.width, canvas.height);
			gl.uniform1f(ctx.shd.blur.uniforms.bloomStrength, ctx.mode == "scene" ? 1.0 : ui.rendering.lightBrightness.value);
			gl.uniform2f(ctx.shd.blur.uniforms.direction, 1.0, 0.0); // Horizontal direction
			gl.activeTexture(gl.TEXTURE0);
			gl.bindTexture(gl.TEXTURE_2D, ctx.tex.frame);
			gl.uniform2f(ctx.shd.blur.uniforms.frameSizeRCP, 1.0 / canvas.width, 1.0 / canvas.height);
			gl.uniform1f(ctx.shd.blur.uniforms.samplePosMult, ui.blur.samplePos.value);
			gl.uniform1f(ctx.shd.blur.uniforms.sigma, Math.max(ui.blur.kernelSize.value / ui.blur.sigma.value, 0.001));
			gl.drawArrays(gl.TRIANGLE_FAN, 0, 4);
			
			/* Pass 2: Vertical blur to final destination */
			const finalFB = ctx.mode == "bloom" ? ctx.fb.final : null;
			gl.bindFramebuffer(gl.FRAMEBUFFER, finalFB);
			gl.viewport(0, 0, canvas.width, canvas.height);
			gl.uniform1f(ctx.shd.blur.uniforms.bloomStrength, ctx.mode == "scene" ? 1.0 : ui.rendering.lightBrightness.value);
			gl.uniform2f(ctx.shd.blur.uniforms.direction, 0.0, 1.0); // Vertical direction
			gl.activeTexture(gl.TEXTURE0);
			gl.bindTexture(gl.TEXTURE_2D, ctx.tex.frameIntermediate);
			gl.uniform2f(ctx.shd.blur.uniforms.frameSizeRCP, 1.0 / canvas.width, 1.0 / canvas.height);
			gl.uniform1f(ctx.shd.blur.uniforms.samplePosMult, ui.blur.samplePos.value);
			gl.uniform1f(ctx.shd.blur.uniforms.sigma, Math.max(ui.blur.kernelSize.value / ui.blur.sigma.value, 0.001));
			gl.drawArrays(gl.TRIANGLE_FAN, 0, 4);
		}

		if (ctx.mode == "bloom") {
			/* Now do the bloom composition to the screen */
			gl.bindFramebuffer(gl.FRAMEBUFFER, null);
			gl.useProgram(ctx.shd.bloom.handle);

			gl.uniform2fv(ctx.shd.bloom.uniforms.offset, offset);
			gl.uniform1f(ctx.shd.bloom.uniforms.radius, radiusSwitch);

			gl.activeTexture(gl.TEXTURE0);
			gl.bindTexture(gl.TEXTURE_2D, ctx.tex.sdr);
			gl.uniform1i(ctx.shd.bloom.uniforms.texture, 0);

			gl.activeTexture(gl.TEXTURE1);
			gl.bindTexture(gl.TEXTURE_2D, ctx.tex.frameFinal);
			gl.uniform1i(ctx.shd.bloom.uniforms.textureAdd, 1);

			gl.drawArrays(gl.TRIANGLE_FAN, 0, 4);
		}

		/* Ask for CPU-GPU Sync to prevent overloading the GPU during compositing.
		   In reality this is more likely to be flush, but still, it seems to
		   help on multiple devices with during low FPS */
		gl.finish();

		const now = performance.now();
		let dt = now - prevNow;

		if (dt > 0) {
			const instFPS = 1000 / dt;
			const ALPHA = 0.05;
			fpsEMA = ALPHA * instFPS + (1 - ALPHA) * fpsEMA;
			msEMA = ALPHA * dt + (1 - ALPHA) * msEMA;
		}
		prevNow = now;

		if (ui.rendering.animate.checked && now - lastStatsUpdate >= 1000) {
			ui.display.fps.value = fpsEMA.toFixed(0);
			ui.display.ms.value = msEMA.toFixed(2);
			lastStatsUpdate = now;
		}
	}

	let animationFrameId;

	/* Render at Native Resolution */
	function nativeResize() {
		const [width, height] = util.getNativeSize(canvas);

		if (width && canvas.width !== width || height && canvas.height !== height) {
			canvas.width = width;
			canvas.height = height;

			if (!ctx.flags.benchMode) {
				stopRendering();
				startRendering();
			}
			if (!ui.rendering.animate.checked)
				redraw();
		}
	}

	/* Resize Event */
	nativeResize();

	let resizePending = false;
	window.addEventListener('resize', () => {
		if (!resizePending) {
			resizePending = true;
			requestAnimationFrame(() => {
				resizePending = false;
				nativeResize();
			});
		}
	});

	function renderLoop() {
		if (ctx.flags.isRendering && ui.rendering.animate.checked) {
			redraw();
			animationFrameId = requestAnimationFrame(renderLoop);
		}
	}

	function startRendering() {
		/* Start rendering, when canvas visible */
		ctx.flags.isRendering = true;
		renderLoop();
	}

	function stopRendering() {
		/* Stop another redraw being called */
		ctx.flags.isRendering = false;
		cancelAnimationFrame(animationFrameId);
		/* Force the rendering pipeline to sync with CPU before we mess with it */
		gl.finish();

		/* Delete the buffers to free up memory */
		gl.deleteTexture(ctx.tex.sdr); ctx.tex.sdr = null;
		gl.deleteTexture(ctx.tex.selfIllum); ctx.tex.selfIllum = null;
		gl.deleteTexture(ctx.tex.frame); ctx.tex.frame = null;
		gl.deleteTexture(ctx.tex.frameIntermediate); ctx.tex.frameIntermediate = null;
		gl.deleteTexture(ctx.tex.frameFinal); ctx.tex.frameFinal = null;
		gl.deleteFramebuffer(ctx.fb.scene); ctx.fb.scene = null;
		gl.deleteFramebuffer(ctx.fb.intermediate); ctx.fb.intermediate = null;
		gl.deleteFramebuffer(ctx.fb.final); ctx.fb.final = null;
		ctx.flags.buffersInitialized = false;
		ctx.flags.initComplete = false;
		ui.display.fps.value = "-";
		ui.display.ms.value = "-";
	}

	function handleIntersection(entries) {
		entries.forEach(entry => {
			if (entry.isIntersecting) {
				if (!ctx.flags.isRendering && !ctx.flags.benchMode) startRendering();
			} else {
				stopRendering();
			}
		});
	}

	/* Only render when the canvas is actually on screen */
	let observer = new IntersectionObserver(handleIntersection);
	observer.observe(canvas);
}