scale

will this actually fit?

a chair can look perfect in a photo and still block your door. scale starts with a real object: something in a store, a secondhand listing, or furniture you already own. scan it, find similar products, and check them against your room before buying.

i worked on the quest runtime, physics, and layout solver, with the designer agent connecting those pieces, and on the expo app that captures rooms and objects on the phone. the interesting part was getting the scan, the math, and the room you stand in to agree.

a collection of furniture including chairs, beds, lamps, tables, and dressers

start with the room you have

the iphone app uses roomplan to capture the room in metres. a custom lidar path measures individual objects, while object capture makes textured models of things you already own. the phone converts those scans from usdz to glb before uploading them.

the app keeps those three paths separate on its capture tab: photograph the walls, walk around an object, or tap once to measure something. every room you capture lands in a library with its area, wall count, and openings, ready to push to the headset.

the capture tab with three modes: photograph the walls, capture an object in 3d, and measure an object — three ways in: walls, a full object, or a single measurement.

the rooms tab listing a 14.4 square metre room with four walls and two openings — a captured room with its area, walls, and openings.

two more captured rooms, 24.4 and 7.2 square metres, each with a use in headset button — rooms of very different sizes share the same card.

the same flow works for a chair, a small safe, or even a teammate. each object carries its dimensions and identity into the headset.

scanning a store chair with an iphone — a chair on the store floor.

the scanned chair placed in a room through the quest headset — the same chair, at full size in the room.

scanning a small safe in a store — the capture path works beyond furniture.

three copies of a scanned teammate in the headset — we also tried it on a teammate.

a scan is not done until it has a size. the scanned tab marks each object 3d ready only once its mesh and width, height, and depth are stored together, so the headset never has to guess.

the scanned tab listing two captured objects, both marked 3d ready with centimetre dimensions — two captured objects, both with a mesh and measurements.

a rotatable 3d model of a black mesh office chair, 66.8 by 60.6 by 56.2 centimetres — an office chair, measured at 66.8 × 60.6 × 56.2 cm.

a rotatable 3d model of a grey armchair with wooden legs, 78.4 by 83.2 by 86.5 centimetres — an armchair, 78.4 × 83.2 × 86.5 cm, ready for the headset.

find something like this, that fits here

siglip2 puts text and product images in the same 768-dimensional embedding space. vector search finds visual matches; measured dimensions filter what can fit.

shopify global catalog browserbase

the products are real. the shopify global catalog returns the merchant, the exact variant, price, availability, and a checkout link, so a match in the headset is a listing you can buy, not a stock render. the taro lift-top below is a poly & bark listing that made it into the demo catalog.

the taro lift-top coffee table product page on poly and bark's shopify storefront, with price, colour, and a view dimensions link — a shopify storefront, as the ingestion pipeline sees it. the price and variant come straight from the catalog api.

sizes are a different story. they are often buried in a tab, a diagram, or a spec table that no api exposes. ingestion checks structured data first, then hands the page to browserbase, which renders it in a real browser and lets the extraction step read what a shopper would read. on this page that is a dimensions tab: 47.25 × 24 × 15.5 inches, which becomes 120 × 61 × 39.4 cm with its source attached.

the dimensions tab of the taro lift-top listing, with overall dimensions of 47.25 by 24 by 15.5 inches and a measured line drawing — the dimensions tab browserbase reads. the same numbers show up on the phone as 120 × 39.4 × 61 cm.

in the headset, that same pipeline runs from a spoken request. a small panel floats by the desk while browserbase renders each merchant page, then fills in with the listings that came back, each one marked fits or placed at true scale.

inside the headset, a floating panel reads searching shopify via browserbase, listing poly and bark, instyle home, and sabai design with rendering the search page under each — searching shopify via browserbase, from inside the room. each merchant page is rendered as it goes.

a floating panel in the headset listing six results: nichols curio cabinet, hillhurst dresser, maro wardrobe, laso sideboard, amalfi cabinet, and cura wardrobe, each with merchant, centimetre dimensions, and a fits label — six listings for “hey, scale, this room looks a little empty”, with dimensions and a fit label on each.

missing evidence stays marked size unverified. in one check, a side table missed a 50 cm gap by 0.8 cm; another variant fit with 4.3 cm to spare.

on the phone, the result is a plain catalog. every listing shows its merchant, price, and recovered dimensions beside a 3d ready badge, so you can tell at a glance which products already have a model waiting in the headset.

the furniture tab with a search field and listings showing vendor, price, dimensions, and a 3d ready badge — real listings, with sizes pulled from merchant pages. the taro is fifth from the top.

the end of the furniture catalog, with a footer reading 44 listings across 8 merchants — 44 listings from 8 merchants during the demo.

a good-looking mesh still needs a ruler

when a product has photos but no model, stable fast 3d on baseten generates one. its geometry has no trustworthy physical size, so the binding step applies the measured dimensions once and moves the origin to the bottom centre.

services/gen/app/binding/core.py · excerpt

centre = (bounds[0] + bounds[1]) / 2
centre[1] = bounds[0, 1]
correction = np.eye(4)
correction[:3, :3] = np.diag(scale)
correction[:3, 3] = -centre * scale
transform = correction @ rotation

the binder preserves materials, exports the glb, reloads it, and checks the resulting bounds. the headset keeps its scale at one. a matching bounding box does not prove the generated shape is right, so distorted models can be flagged for review or shown as measured proxies.

make the room feel like a room

the quest app uses three.js and webxr for the scene, with rapier handling collisions. furniture drops onto the floor, stops at walls, and stays upright when dragged. you can grab, rotate, and lift objects, or pull another copy from the floating catalog.

room coordinates and scene coordinates differ by a recentering offset. keeping that conversion explicit prevents the solver from putting a desk somewhere different from where the headset draws it. elevenlabs handles push-to-talk and spoken replies.

getting a room into the headset is one scan of a qr code. the phone pushes the current version, and the headset opens that room with the same walls, openings, and objects. a diagnostics panel underneath shows the live sse link, which is how designer proposals arrive without a refresh.

the headset tab with a qr code, a push this version button, and diagnostics showing the api base, stub layer with 4 walls, and an open sse sync — scan the code, push the version, and watch the sync stay open.

the floating furniture catalog with dimensions in the quest — scans and catalog furniture share the same room and units.

everything you can do lives on one floating panel with four tabs: designer, my scans, furniture, and room. drag the bar to move it, push a window with the stick, and press a twice to delete. the object you are holding gets quick actions like turn, remove, and undo.

the floating panel in the headset on its designer tab, showing turn 90 degrees left, turn 90 degrees right, remove, and undo, with a red office wall behind it — the panel in a scanned office. the top bar explains the controls; the designer tab holds the quick actions.

ask for a change, then solve it

“turn this bedroom into a six-person workspace” becomes a plan with constraints. the language model interprets the request; or-tools cp-sat chooses positions and rotations on an integer centimetre grid. it never needs to load a 3d mesh.

services/fit/app/solver.py · excerpt

# the footprint stays inside the room
m.Add(o.x - o.hw >= bounds["minX"])
m.Add(o.x + o.hw <= bounds["maxX"])
m.Add(o.z - o.hd >= bounds["minZ"])
m.Add(o.z + o.hd <= bounds["maxZ"])

separate constraints keep pieces apart and protect door zones. a fit validator checks the result, and the agent can repair it or explain a compromise. in the apartment demo, it narrowed requested walkways from 90 cm to 75 cm and showed that tradeoff in the log.

changes arrive as proposals. you can keep one, put the room back, or ask again. each room has its own durable object for requests, preferences, and version history; accepting an old proposal is rejected if the room has changed underneath it.

the solver reporting a walkway tradeoff and a 36 millisecond layout — the solver explains what had to change.

the apartment after the furniture arrangement — the resulting layout, with an avatar for scale.

the designer panel showing undo, rearrange, and show listings above a standing desk and a newly placed wooden sideboard, with the listings panel beside it — after a rearrange: a sideboard from the listings has been placed behind the desk, and the panel offers undo.

those versions show up back on the phone. a room that has been through a few proposals carries its history with it, and any of them can be pushed to the headset again.

the rooms list showing a 41.3 square metre room and a 14.0 square metre room with six versions — the 14 m² room after six rounds with the designer.

the designer proposal with keep, put back, and ask again controls — a proposal you can inspect and undo.

how the pieces talk

cloudflare connects the phone, headset, and compute services. shared contracts keep an object's id, dimensions, and model consistent across storage, search, and placement.

scale architecture connecting iphone and quest clients, cloudflare state and workers, compute services, and external providers — the project's architecture diagram. click to view at full size.

piece	responsibility
captureexpo + swift	roomplan, lidar measurements, object capture, and on-device mesh export
room runtimewebxr + three.js + rapier	the room, furniture, physics, controllers, and floating panels
shared statecloudflare	workers route requests; d1 stores records, r2 stores assets, and vectorize indexes products
designerroom-scoped durable object	request history, planning, solver calls, proposals, and live events
computedocker + baseten	search, merchant ingestion, fit checks, and image-to-3d generation

slow generation runs through an outbox, queue, dispatcher, and workflow. admission control limits concurrent gpu jobs; completed assets land in r2. the headset refreshes the scan library, while designer events stream over sse. heavy services ran in docker behind cloudflare tunnels during the hackathon.

what we actually measured

0.8 cm / 4.3 cm: one table was too wide; another fit with clearance
1.19 s: a warm image-to-3d request; the first took 28.71 s
36 ms: an optimal layout for the five-piece apartment demo
262 products: dimensions recovered from rendered merchant pages
100 products: in the curated catalog, across five categories
< 3 × 10⁻⁸ m: exported mesh bounds versus supplied dimensions

these are individual measurements recorded in the project readme. warm generation timing excludes a cold start; mesh-bound error measures the export math, not the accuracy of a real-world scan.