A blueprint of our house as a Home Assistant dashboard, with html-in-canvas

A 3D blueprint of the house that controls our Home Assistant devices, with room names and the control panel drawn into the scene by Chrome's new html-in-canvas API, and the pitfalls we hit getting there.

A two-storey house drawn as a white-line blueprint on blue graph paper, its floors pulled apart. Room names such as Primary Bedroom, Kitchen and Office are lettered flat on each floor in perspective; a rail on the right names each floor, with leader lines to it.
The dashboard showing an invented sample house, floors pulled apart. The room names are HTML painted onto the floors with html-in-canvas.

Once House Scan had every floor of our house as data, I wanted to use it for something I’d look at every day: a 3D model of the house, drawn like a blueprint, that I can spin and zoom, tap a room in, and control that room’s Home Assistant devices from. What made it worth building was html-in-canvas, which shipped in Chrome 157. It lets real HTML be drawn inside a WebGL scene and still take clicks. The room names are lettered on the floors in perspective, and the control panel lies on the drawing beside the floor plan. Both are ordinary HTML with ordinary buttons.

The demo below is an invented house with pretend devices, running entirely in your browser. Drag to turn it, tap a floor, then tap a room. In Chrome 157 or later you get html-in-canvas; other browsers get a fallback, described under Other browsers.

Live demo, running in your browser: an invented house and pretend devices. Nothing here talks to a real home. Open full screen

The drawing

Each floor’s export lists its walls, doors, windows and openings with their size, position and rotation. The page draws each wall as a thin box with its openings cut out, in white lines on blueprint blue, with translucent faces lit from one side so the planes read as solid. Windows get glass, furniture is drawn dashed, and dimension strings with tick marks run along the outside walls. The lettering is set in Architects Daughter, a typeface based on an architect’s handwriting. On load, the lines draw themselves in, floor by floor.

ControlWhat it does
Spin and zoomDrag and scroll, or auto-rotate
ExpandPulls the floors apart to see inside
Floor railFloor names down the side, with leader lines to each floor. Tap one for its plan.
Room namesTap one for that room’s devices and controls

What html-in-canvas is

A canvas has always been a dead end for HTML. You can draw anything into it, but text, buttons and forms in a WebGL scene had to be redrawn by hand, or floated over the canvas as a separate layer. A floated layer can’t lie at an angle on a surface, go behind anything, or pick up the scene’s lighting. Libraries such as html2canvas copy a page into a canvas by re-implementing CSS, which only covers the properties they understand.

html-in-canvas lets the browser do it. A canvas marked as drawable lays out its child elements but doesn’t show them. Each frame in which one of them changes, the canvas gets a paint event listing the changed elements, and the page copies each element’s current picture into a texture: drawElementImage for a 2D canvas, texElementSubImage2D for WebGL, and drawElementImageToTexture for WebGPU. From there it’s a texture like any other, so it can go on any surface in a scene.

The other half is input. In a 2D canvas the browser knows where it drew the element. In 3D it can’t, so each frame the page reports where the element now appears with updateElementGeometry, as a transform from the element’s box to the canvas. Chrome hit-tests clicks through that transform, perspective included, and delivers them to the real element.

How the dashboard uses it

Room names are HTML elements painted flat on each floor. They tilt with the drawing like ink on the paper, and tapping one opens that room. A layer floated over the canvas can’t do this: it always faces the viewer.

The control panel is one HTML element laid on the sheet beside the floor plan, at about one screen pixel per CSS pixel so the text stays sharp. It lists the room’s devices with schematic symbols, the way an electrical drawing marks a lamp or a switch. Selecting one shows its controls: an on/off switch, a ruler of brightness ticks, a dial for the thermostat, lock and unlock, garage door, media and vacuum controls. When a device changes in Home Assistant, the panel’s HTML changes, Chrome fires a paint event, and the new picture is uploaded. There’s no drawing code for the panel at all; it’s styled with CSS like any page.

The main floor of the sample house in plan view, with room names lettered on the floor, and beside it on the drawing a control panel for the Hallway listing a front door lock, a thermostat, a robot vacuum and a porch camera, with the thermostat open: a dial at 68.5 degrees, set to 70, heating.
Plan view with the Hallway open. The panel is one HTML element drawn onto the sheet; its buttons take clicks. Sample house, pretend devices.

What Chrome got right

Privacy built into the painting. Any API that turns page content into pixels a script can read is a risk, because the pixels could reveal things the page shouldn’t know. Chrome’s answer is what the explainer calls read-back-allowed rendering: drawn elements never include cross-origin content, visited-link styling, spelling and grammar markers, pending autofill, system colours or IME pop-ups. Because nothing sensitive is ever painted, reading the pixels back is always safe, and there’s no tainted-canvas state to manage. An earlier option to turn this off was removed during the trial.

The elements stay real. A drawn button is still a button. It gets focus, a text field shows its caret, and the explainer says drawn content is exposed to accessibility tools the same way as regular DOM. Once an element has geometry, getBoundingClientRect even reports where it appears on screen, transform applied, which made our end-to-end tests simple to write.

One paint event per frame. The event fires once per frame, after Chrome paints, with only the elements that changed. The dashboard re-uploads a texture only when its element changed, and a DOM change made inside the handler shows up in the next frame instead of looping.

One model across 2D, WebGL, WebGPU and workers. The same element can go to any of the three, and captureElementImage makes a snapshot that can be sent to a worker and drawn into an OffscreenCanvas there.

Pitfalls we hit

The explainer describes the API but leaves out details you need from WebGL, and the API changed shape several times during its trial. These are the problems we hit or designed around, measured in Chrome 157:

ProblemCauseFix
Blank textures, and “offset overflows texture dimensions” in the consoleAn element’s painted picture can be bigger than its box, because of outlines and text shadows, so the copy overflows a texture sized to the boxPass the element’s box as the source rectangle in the upload options
Text upside downThe element’s top row lands in texture row 0Flip the plane’s texture coordinates
Clicks landing in the wrong placeThe transform maps the element’s CSS-pixel box into the canvas’s CSS pixels, not its drawing-buffer pixelsBuild the transform in CSS pixels
Nothing to upload intotexElementSubImage2D copies into an existing texture; it no longer allocates oneAllocate the texture at the element’s size first
Labels as wide as the canvasDrawable children are laid out as blocksSize them to their content
Keeping three.js in stepthree.js caches WebGL bindings, and the upload changes one without telling itReset three.js’s state after each upload
Taps reaching the camera controlsPresses on drawn elements bubble up to the canvas, where the camera controls listen, so a tap could also start turning the houseStop the press at the panel

One more, for testing: headless Chrome’s virtual clock barely runs animation frames in a WebGL scene, and an open event stream stops it from finishing at all. We tested in real time instead, driving Chrome with Playwright: click the centre of the box Chrome reports for a room name, check that the panel opened, tap a drawn switch, check that the device changed.

Talking to Home Assistant

The page never holds a Home Assistant token. A small server on the same machine holds it, connects to Home Assistant, and gives the page areas, devices and states, a stream of changes, and a fixed list of allowed actions, such as turning a light on or locking a door. It answers only pages from the same machine, checks where each request comes from, and strips attributes such as camera access tokens. The token belongs to a non-admin Home Assistant user made for the dashboard. Scanned rooms match Home Assistant areas by name where the match is unambiguous; for the rest, the panel lists the areas and you pick one.

Other browsers

For now this is a Chrome feature. WebKit has signalled support and has started implementing the API. Mozilla has no position, and its engineers have raised real concerns there: that the web should move away from building rich UIs in canvas, and that rendering HTML to readable pixels risks creating a “pixel oracle.” The read-back-allowed rendering above is Chrome’s answer to the second. The first is a fair worry for ordinary pages, which should stay HTML. A 3D drawing of a house is the case it fits: the drawing has to be WebGL, and html-in-canvas is what keeps the labels and controls in it real, accessible HTML instead of pixels.

I hope Safari and Firefox ship it. Most of my family’s phones are iPhones, so today they get the fallback: the same HTML floated over the canvas. Everything works, but the room names face you instead of lying on the floor, and the panel sits at the side of the screen. The dashboard detects support at runtime, and all the code that uses the API is in one module, because the names changed several times during the trial and may change again.

What’s next

Next come the outside: cameras, lights and the other devices around the house, with the driveway and landscaping drawn into the blueprint. After that the dashboard moves into Rocco, our family hub. The same scans also gave Pond House, the sticker dollhouse game our kids play on the hub with their friends… a version of our real house, scanned using the House Scan app, too.

The code is on GitHub, in the blueprint folder of the House Scan repository. The scanner itself is covered in the House Scan article.

← All writingRSS