Mixed Reality

An XR project runs as virtual reality or mixed reality. The difference is one field on the XR rig — passthrough — and everything else follows from it.

With passthrough on, the experience composites over the user's real room instead of a virtual background. Turn it on in the inspector (XR Rig → Mixed Reality), or from a script by authoring the rig.

Mixed reality passthrough is a Quest and Android XR capability.

Apple Vision Pro

Safari on visionOS — checked through visionOS 27 — supports WebXR immersive-vr only. It grants immersive-ar to no web page, and exposes no plane detection, mesh detection or depth sensing. This is a platform policy, not a missing feature on our side; every web XR tool works within it (the industry-standard answer is a VR fallback mode).

A mixed-reality project still works there, three ways:

1. Automatic VR fallback. The session probe falls back to immersive-vr, the authored background applies, and hands/pinch interaction works. The entry page says so honestly: "Passthrough isn't available in this browser — this experience will run in VR."
2. A synthetic floor, no scan needed. With environmentPhysics on and no scene understanding granted at all, the engine synthesizes one floor collider whose top face sits at y = 0 — which local-floor makes the user's REAL floor. Dropped objects land at floor height instead of falling forever. (This only happens when the detection features are entirely absent; a Quest that merely has not scanned yet keeps its room-capture prompt instead.)
3. "View in your room" via AR Quick Look. On Apple devices the entry page offers a native placement preview: the scene's content is exported to .usdz on the fly and handed to AR Quick Look, which places it in the real room with passthrough — Apple's sanctioned web-AR path (it also works on iPhone and iPad). This is a static snapshot: no scripts, no physics.

What the rig controls

Field	What it does
passthrough	Show the real room. The VR/MR switch.
depthOcclusion	Real objects hide virtual ones.
environmentPhysics	Give the real room physics colliders.
environmentDebug	Draw the detected room as a wireframe, on-device.
hitTest	Let scripts raycast against the real room.
anchors	Pin content to a real-world spot.

Everything except passthrough is off by default, and that is deliberate: plane detection, mesh detection and depth sensing each make the headset ask the user for permission to share a scan of their room. Turn on only what you use.

Physics against the real room

Set environmentPhysics and the room becomes solid:

• "planes" — the flat surfaces the headset detects (floor, walls, desks). Cheaper, and available on more devices.
• "mesh" — the full room scan, including furniture. Far more faithful, but only where the runtime provides one.
• "auto" — the mesh where the device has one, falling back to planes.
• "none" (default) — objects pass straight through the room.

Each detected surface becomes an ordinary entity, so it goes through the same physics as everything else: a dynamic rigidbody dropped in a mixed-reality scene lands on the real floor with no extra code.

The room has to have been scanned first

This is the single most common reason a mixed-reality scene comes up empty, and it is not a bug in the experience.

Plane and mesh detection do not scan anything. They replay a scan the user captured earlier through the headset's own room setup — on Quest, Settings > Guardian > Mixed Reality (Space Setup). If the user has never run it, detectedPlanes and detectedMeshes simply stay empty, with no prompt and no error. If they ran it in a different room, you get that room's furniture overlaid on this one, which looks like the tracking is broken.

XREnvironmentSystem handles this: when environmentPhysics is on and nothing at all has been detected three seconds into the session, it calls XRSession.initiateRoomCapture(), which launches room setup from inside the session. It is a Meta extension, so it is feature-detected and silently skipped elsewhere, and — per Meta's contract — it is only ever called once per session.

If a user reports "it never asked to scan my room" and the geometry does not match, they are almost certainly running against a stale scan from another space; re-running Space Setup for the current room fixes it.

Reading the environmentDebug overlay

The overlay colours surfaces by what the runtime says they are, so a glance (or a screenshot) tells you which surface is wrong rather than just "wireframe somewhere":

• green — the floor plane
• blue — walls
• purple — the ceiling
• orange — labelled things: tables, couches, shelves, doors, windows
• faint gray — the raw room scan (the "global mesh")

Two things about the global mesh are easy to misread. It is deliberately COARSE — a couch becomes a melted lump, a wall gets bumps — because it is a collision mesh, not a picture; those lumps are not phantom objects. And under environmentPhysics: "auto" it is what the physics uses, with the flat planes skipped entirely, so orange furniture boxes and gray scan lumps can legitimately overlap the same real object.

The overlay is depth-tested like ordinary geometry — the near wall hides the far one — but it never writes depth, so it cannot occlude actual content. (It was briefly rendered X-ray; on device, the dense global scan at low opacity fuses into translucent sheets, and without a depth test the walls' sheets drew through everything and stacked in front of the user's face.)

For identifying a mystery shape, XREnvironmentSystem.getDebugSnapshot(world) returns a census of every spawned surface — kind, semantic label, position and approximate radius.

One artifact is handled for you: three.js's depth-occlusion pass draws a head-locked mesh that writes only depth, but leaves its colour output undefined — on Quest hardware that painted the depth camera's view over the screen as faint contour bands that follow the head. XRSessionSystem forces colorWrite off on that mesh every frame, so enabling depthOcclusion costs no visual artifacts.

Lost tracking freezes a surface — it never follows the head

When the headset loses tracking of a surface, WebXR does not stop reporting a pose — it reports an emulated one: the last known pose held relative to the viewer, i.e. glued to the head. Applying that to room geometry would make the surface — and its collider — follow the user around, shoving physics objects out of the way. The engine instead freezes the surface at its last tracked pose, resumes when tracking returns, and never creates a surface from an emulated pose in the first place. The diagnostics census marks such surfaces with emulated: true.

No artificial locomotion in passthrough

While a session composites over the real world, XRLocomotionSystem disables ALL artificial locomotion — snap turn, teleport and smooth movement — whatever the rig says. Virtual content in mixed reality is registered to the real room; rotating or translating the rig swings every virtual object out of alignment while the passthrough room stays put, which feels broken rather than like movement. The user moves by walking. Note that snapTurn is a separate field from locomotion, so locomotion: "none" alone would NOT have stopped the stick from turning the rig — the gate is on the live blend mode.

Finding the real room from a script

Detected surfaces carry XREnvironmentComponent, so they are queryable like any other entity. semanticLabel is what the runtime called the surface — commonly floor, wall, ceiling, table, desk — and is null on runtimes that classify nothing, so always handle that case.

import { Behaviour } from "@relu-interactives/spatial-ecs";
import {
  TransformComponent,
  XREnvironmentComponent,
} from "@relu-interactives/spatial-ecs/components";

export default class FloorReporter extends Behaviour {
  onUpdate() {
    for (const entityId of this.world.query(XREnvironmentComponent)) {
      const surface = this.world.getComponent(entityId, XREnvironmentComponent);
      if (surface?.semanticLabel !== "floor") continue;

      const transform = this.world.getComponent(entityId, TransformComponent);
      if (transform) console.log("real floor at y =", transform.position.y);
    }
  }
}

Surfaces appear over the first few seconds of a session and are revised as the headset re-scans, so query them each frame rather than caching a list at start-up.

Reacting to what the device granted

Every mixed-reality feature is requested OPTIONALLY — requiring one would make requestSession reject outright on any headset that lacks it, which would mean shipping a build that refuses to start. So check what you actually got:

import { Behaviour, type XrCapabilities } from "@relu-interactives/spatial-ecs";

export default class RoomAware extends Behaviour {
  onXRSessionStart(capabilities: XrCapabilities) {
    if (capabilities.mode !== "immersive-ar") {
      // Fell back to VR — draw the virtual environment instead.
      return;
    }
    if (!capabilities.planeDetection && !capabilities.meshDetection) {
      // Passthrough, but no room geometry: keep content within arm's reach.
    }
  }
}

Hit-testing the real room from a script

With hitTest on the rig, XRHitTestSystem keeps a ray from the viewer (and one per hand) cast against the real room. Systems are resolved by INSTANCE, so pass a throwaway one to getSystem:

import { Behaviour, XRHitTestSystem } from "@relu-interactives/spatial-ecs";
import { TransformComponent } from "@relu-interactives/spatial-ecs/components";

export default class PlaceOnRealSurface extends Behaviour {
  onSelectStart() {
    const hitTest = this.world.getSystem(new XRHitTestSystem());
    // "viewer" = where the user is looking; "left"/"right" = hand rays.
    const hit = hitTest.getHitResult("viewer");
    if (!hit) return; // nothing granted, or no surface under the ray yet

    const transform = this.world.getComponent(this.entityId, TransformComponent);
    if (transform) {
      transform.position.x = hit.position.x;
      transform.position.y = hit.position.y;
      transform.position.z = hit.position.z;
    }
  }
}

getHitResult returns null until the runtime grants hit-testing and the ray actually meets a surface — always handle that case.

Anchors

A headset keeps revising where it thinks the room is. Content at fixed scene coordinates does not move with those corrections, so a vase carefully placed on a real table slowly slides off it.

Add XRAnchorComponent (inspector: Add Component → XR → World Anchor) and the entity is pinned to the real world instead: the runtime owns its pose from then on, and it stays put relative to the room.

Because the anchor owns the transform, moving the entity by script or physics has no lasting effect — disable the component first, move it, then re-enable.

From a script the toggle is the whole API:

import { Behaviour } from "@relu-interactives/spatial-ecs";
import { XRAnchorComponent } from "@relu-interactives/spatial-ecs/components";

export default class ReleaseOnGrab extends Behaviour {
  onGrab() {
    // Let the hand move it: release the weld for the duration of the grab.
    const anchor = this.world.getComponent(this.entityId, XRAnchorComponent);
    if (anchor) anchor.enabled = false;
  }

  onRelease() {
    // Re-anchor wherever the user left it.
    const anchor = this.world.getComponent(this.entityId, XRAnchorComponent);
    if (anchor) anchor.enabled = true;
  }
}

Anchors last for the session only. Persisting one across launches needs requestPersistentHandle, which is a draft extension and not supported here.

Notes

• Never paint an opaque background. A skybox covers the whole framebuffer, so it hides the room completely and looks exactly like passthrough having failed. The engine withholds the skybox automatically while a passthrough session runs.
• Blend mode is fixed for a session. passthrough is read when the session starts; it cannot be toggled mid-session. Ending and re-entering is the only way.
• Room geometry arrives late. Nothing is detected on the first frame. Write logic that copes with an empty room and fills in over a few seconds.