VR, Grabbing and Hand Poses

VR projects run on raw WebXR. Behaviours receive XR events through hooks, exactly like onUpdate or onTargetFound — there is no separate XR API to import.

Session hooks

import { Behaviour, type XrCapabilities } from "@relu-interactives/spatial-ecs";

export default class Greeter extends Behaviour {
  onXRSessionStart(capabilities: XrCapabilities) {
    // What the headset actually granted — never guess from the user agent.
    if (capabilities.handTracking) { /* … */ }
  }

  onXRSessionEnd() { /* … */ }
}

XrCapabilities carries mode, handTracking, planeDetection, meshDetection, depthSensing, anchors, hitTest and the raw enabledFeatures array.

Pointer and grab hooks

Hook	Fires when
onSelectStart(input)	Trigger pressed / finger pinched.
onSelectEnd(input)	Released.
onGrab(input)	This entity was picked up.
onRelease(input)	This entity was let go.
onHandPoseStart(event)	A hand pose declared on this entity began.
onHandPoseEnd(event)	That pose ended.

input is an XrInputState: handedness, kind (controller / hand / transient-pointer / gaze), rayOrigin, rayDirection, gripPosition, gripQuaternion, select, squeeze, axes, buttons, and joints for tracked hands.

Grabbing and throwing

Add an XR Interactable component in the inspector. The useful fields:

• Reach — physical (hand must touch it), distance (grab along the pointer ray), or both.
• At Distance — snap brings it to the hand, remote manipulates it where it stands.
• Gesture — pinch, full grip, or any.
• Hold Offset — where the object sits in the hand once held.
• Apply Velocity — a released object keeps the speed your hand was moving at, which is what makes throwing work (on by default).
• Hover — Outline, Grow, Outline + grow, or None, plus the colour and grow amount.

A held object follows the hand exactly and does not collide with the world while held.

To make something throwable, give it a Physics component with a DYNAMIC rigidbody. That is the only requirement — Apply Velocity is on by default. Release velocity is taken from the peak of a short window of hand motion, so throws match the motion the user made rather than the frame they opened their hand.

The pointer ray stops at whatever it hits, and its colour, thickness and visibility are set on the XR Interactor. Show Ray is When interacting (only on the hand pointing at something grabbable, and hidden once that hand is holding it), Always, or Never. Hover feedback tells the user what they are about to pick up.

On hand tracking the same pinch drives both grabbing and teleport aiming, so a hand that is holding something — or merely pointing at something grabbable — has teleporting suppressed on that hand. Passing an object between hands does not teleport you, and grabbing with both hands suppresses both. A teleport also only commits if you actually aimed first, so simply letting go of an object never moves you.

import { Behaviour } from "@relu-interactives/spatial-ecs";

export default class Basketball extends Behaviour {
  onGrab() {
    console.log("picked up");
  }

  onRelease() {
    // The throw itself is applied by the engine — react to it here.
    console.log("thrown");
  }
}

Two hands, hand-off, and attach points

An object decides how many hands may hold it, via Hands:

Setting	Behaviour
handOff (default)	Grabbing with the other hand takes it — passing something between your hands.
two	Both hands hold it. The first anchors it, the second aims it.
single	Stays with whichever hand grabbed first.

Attach points are authored hand-holds: a local position and rotation, optionally restricted to one hand, and given a role. The primary role is the anchoring hand (a rifle's trigger grip) and secondary the steering hand (the foregrip) — setting roles is what makes a two-hand grab assemble correctly no matter which hand grabs first. Among eligible points the nearest to the hand wins, so reaching for the foregrip picks the foregrip.

Leave the list empty and hands grab wherever they touch. Two-hand grabbing and resizing still work in that case, anchored wherever the hands landed.

For a two-handed rifle: Hands two, Second Hand Aims the object, and two attach points — one primary at the grip, one secondary at the foregrip.

The primary hand is anchored and the secondary steers. Twist about the line between the hands comes from the primary hand's own roll, so banking your trigger hand banks the weapon.

Two-hand resizing

Turn on Resize and moving both hands apart or together scales the object. Scale is uniform — two grab points give one distance measurement, so there is nothing to drive a second axis independently — and limited by Min/Max Size, which bound a single gesture relative to the size when the second hand grabbed. Repeated gestures compound.

A small deadzone means ordinary hand drift while carrying something does not resize it.

Releasing one hand does not drop the object. Let go with the support hand and the anchoring hand keeps it; let go with the anchoring hand and the other is promoted, so a two-hand hold released one hand at a time can still be thrown.

Reacting to grabs in script

import { Behaviour, type XrInputState } from "@relu-interactives/spatial-ecs";

export default class Rifle extends Behaviour {
  onGrab(input: XrInputState) {
    console.log("held by", input.handedness);
  }

  onSecondaryGrab() {
    // Both hands are on it now — steady the aim, show a bipod, whatever.
  }

  onSecondaryRelease() {
    // Back to one hand.
  }
}

Every grab setting is a plain field on the component, so a script can change it mid-session — switch on resizing, retune throw strength, or move an attach point in response to what the player is doing.

Vision Pro

Vision Pro drives WebXR entirely through gaze and pinch. There are no controllers, and the input source for a pinch is created when the pinch starts and destroyed when it ends. That has a few consequences worth designing around:

• Look at a thing and pinch to select or grab it. The pick ray starts between the user's eyes and is then carried by the hand, so aiming is "pinch, then move your hand", not "point, then press".
• There is no hover before a pinch — the input source does not exist until then. Pointer rays and hover feedback therefore only appear during a pinch, not while the user looks around.
• Grip/squeeze is never reported. Set an interactable's Gesture to Pinch or Any; Full grip can never trigger on Vision Pro.
• Grabbing does not teleport the user, the same as on hand tracking elsewhere.
• Hand tracking (and therefore hand poses) is available on visionOS 2 and later, but asks the user for a second permission. The OS draws the user's real hands.
• immersive-ar is not supported by Safari on visionOS, so VR projects are the only kind that run there.

Scripting VR interactions

A few patterns that cover most scenes.

Score when a thrown object passes through something. Give the ball a dynamic rigidbody and an XR Interactable; the throw itself is handled for you.

import { Behaviour, type XrInputState } from "@relu-interactives/spatial-ecs";

export default class Ball extends Behaviour {
  onRelease(input: XrInputState) {
    // Fires the moment it leaves the hand — the engine has already applied the
    // velocity, so this is where scoring or audio goes.
    console.log("released by", input.handedness);
  }
}

A panel you point at but cannot pick up. Set pointable: true and grabbable: false, then act on the pointer.

export default class Button extends Behaviour {
  onSelectStart() {
    this.world.emit?.("door:open");
  }
}

React to a two-handed hold. Useful for steadying a weapon, showing a bipod, or gating an action on both hands being committed.

export default class Rifle extends Behaviour {
  private braced = false;

  onSecondaryGrab() {
    this.braced = true;
  }

  onSecondaryRelease() {
    this.braced = false;
  }

  onUpdate() {
    if (this.braced) { /* tighter aim, less sway… */ }
  }
}

Adapt to the device. Branch on what the session granted, never on a device name.

onXRSessionStart(capabilities: XrCapabilities) {
  if (!capabilities.handTracking) {
    // Controllers only — hide anything that depends on finger poses.
  }
}

Two rules worth internalising: move the rig, never the camera (the headset owns the camera while presenting and overwrites it every frame), and give any destructive gesture a longer Hold Time so it cannot fire in passing.

Reading grab state from a script

XRInteractionSystem.getGrabInfo(entityId) reports who is holding an entity — hand count, primary/secondary handedness, live hand separation (two-hand grabs) and the attach-point ids in use. Systems resolve by instance:

import {
  Behaviour,
  XRInputSystem,
  XRInteractionSystem,
} from "@relu-interactives/spatial-ecs";

export default class TwoHandChargeUp extends Behaviour {
  onUpdate() {
    const interaction = this.world.getSystem(
      new XRInteractionSystem(new XRInputSystem()),
    );
    const grab = interaction.getGrabInfo(this.entityId);
    if (!grab || grab.hands < 2 || grab.handSeparation === null) return;

    // Example: pull both hands apart to "charge" — separation is metres.
    console.log("charge level", Math.min(1, grab.handSeparation / 0.8));
  }
}

Returns null while nothing holds the entity, so gate on it every frame.

Hand poses

Add an XR Hand Pose component and list the poses to watch for. Each is either a built-in preset or a custom set of finger curls.

import {
  Behaviour,
  type XrHandPoseEventDetail,
} from "@relu-interactives/spatial-ecs";
import {
  Object3DRef,
} from "@relu-interactives/spatial-ecs/components";

export default class Portal extends Behaviour {
  onHandPoseStart(event: XrHandPoseEventDetail) {
    if (event.poseId === "open-portal") this.show(true);
  }

  onHandPoseEnd(event: XrHandPoseEventDetail) {
    if (event.poseId === "open-portal") this.show(false);
  }

  private show(visible: boolean) {
    const ref = this.getComponent(Object3DRef);
    if (ref) ref.visibility = visible;
  }
}

XrHandPoseEventDetail is { poseId, poseName, handedness, confidence }. Match on poseId — it is the id set in the inspector, not the preset name.

Changing another object's colour from a pose. MaterialComponent.updateValue takes an index and a PATCH OBJECT — updateValue(0, { color }), not updateValue(0, { color: value }).

import {
  Behaviour,
  type EntityInput,
  type WorldEntityView,
  type XrHandPoseEventDetail,
} from "@relu-interactives/spatial-ecs";
import {
  MaterialComponent,
} from "@relu-interactives/spatial-ecs/components";

export default class PoseColor extends Behaviour {
  data = { target: { type: "entity", value: null } as unknown as EntityInput };
  private original = "#ffffff";

  init() {
    const material = this.material();
    const current = material?.values?.[0]?.color;
    if (typeof current === "string") this.original = current;
  }

  onHandPoseStart(event: XrHandPoseEventDetail) {
    if (event.poseId === "approve") this.material()?.updateValue(0, { color: "#0066ff" });
  }

  onHandPoseEnd() {
    this.material()?.updateValue(0, { color: this.original });
  }

  private material() {
    const target = this.data.target.value as WorldEntityView | null;
    return target
      ? this.world.getComponent(target.entityId, MaterialComponent)
      : null;
  }
}

Hand poses require HAND TRACKING. They are read from the 25 tracked finger joints, so nothing fires while the user is holding controllers — put them down and enable hand tracking on the headset. If a scene has poses configured and no tracked hand is seen for a few seconds, the runtime says so in the console.

Presets

fist, open-palm, point, thumbs-up, thumbs-down, peace, pinch, ok, gun, rock, shaka, palm-up.

Custom poses

A custom pose is five curl values, 0 (finger straight) to 1 (fully closed), plus a tolerance. Poses are matched on normalized finger curl rather than raw joint positions, and every distance is scaled by the user's palm width — so a pose authored once matches hands of any size.

Optionally add a direction requirement: which way the palm, fingers, or thumb must point (up, down, forward, back, left, right, toward-head, away-from-head) within an angle tolerance. This is what separates thumbs-up from thumbs-down, which are the same hand shape.

Per-finger direction

Curl says how bent a finger is; direction says where it points. "Index extended, pointing up" and "index extended, pointing at the user" are different gestures with identical curls, so direction is what tells them apart.

In the editor, click a finger on the hand and pick the directions it may point: Up, Down, Forward, Back, Left, Right, At user, Away. Picking several means any one of them counts — that is how "thumb up or diagonally up" stays a single pose. Pick none to leave the finger free.

Angle Tolerance is how far a finger may point from an accepted direction. 45° by default: direction is a coarse signal, and a tight cone makes a pose that looks right to the user fail to register.

Tuning

• Tolerance — raise it if a pose is hard to trigger, lower it if it fires by accident.
• Hold Time — seconds the pose must be sustained before firing. Fingers pass through many shapes on the way to the intended one; a hold time stops those firing.

A pose does not re-fire while held, and survives a brief hand-tracking dropout rather than flickering.

Driving game logic

The hooks fire on the entity that declares the pose, so put the component on the object that should react. A few patterns worth knowing:

Fire once per gesture. onHandPoseStart fires when the pose begins, not every frame, so it is safe to spawn or play something directly.

import { Behaviour, type XrHandPoseEventDetail } from "@relu-interactives/spatial-ecs";

export default class Spellcaster extends Behaviour {
  onHandPoseStart(event: XrHandPoseEventDetail) {
    if (event.poseId !== "cast") return;
    this.spawnPrefab("assets/fireball.prefab", {
      position: this.transform.position,
    });
  }
}

Hold a state while the pose lasts. Use both hooks when the pose should gate something continuous.

export default class Shield extends Behaviour {
  private raised = false;

  onHandPoseStart(event: XrHandPoseEventDetail) {
    if (event.poseId === "open-palm") this.raised = true;
  }

  onHandPoseEnd(event: XrHandPoseEventDetail) {
    if (event.poseId === "open-palm") this.raised = false;
  }

  onUpdate() {
    if (this.raised) { /* absorb damage, drift the shield toward the hand… */ }
  }
}

React per hand. event.handedness is "left" or "right", so one pose can drive two behaviours.

onHandPoseStart(event: XrHandPoseEventDetail) {
  if (event.poseId !== "point") return;
  if (event.handedness === "right") this.aim();
  else this.selectMenu();
}

Require a confident match. event.confidence is 0–1 at the moment the pose fired. Gate on it when a gesture is destructive.

onHandPoseStart(event: XrHandPoseEventDetail) {
  if (event.poseId === "delete-all" && event.confidence > 0.95) this.clearScene();
}

Two things to design around: match on poseId, not the pose name — the id is the contract, the name is a label — and give a destructive gesture a longer Hold Time so it cannot fire in passing.

Locomotion

Movement is a property of the XR Rig, not of scripts: tracking origin (floor or eye level), locomotion mode, move speed, snap turn, and — when locomotion is teleport — the arc's Color, Blocked Color and Thickness. The arc starts a short way out from the head so it never draws through the user's own view, which matters on Vision Pro where the aim ray originates between the eyes. Teleport destinations are Teleport Area entities. Scripts should move the rig, never the camera — while a session is presenting, the headset owns the camera and writes its pose every frame.