Real-Time 21-Point Hand Landmark Estimation & Quaternion Projection in WebAssembly

1. Abstract & Introduction

Monocular 3D hand pose estimation from ordinary RGB webcams represents one of the most computationally demanding challenges in modern browser software. While specialized LiDAR sensors and stereo cameras project structured infrared grids to measure physical depth directly, commodity webcams output a flat 2D pixel array.

To infer 3D spatial coordinates (x, y, z) across 21 anatomical hand joints, Animatrous applications deploy a two-stage deep learning pipeline compiled to WebAssembly with 128-bit SIMD (Single Instruction, Multiple Data) optimizations:

1. Palm Detection: An SSD (Single Shot Multibox Detector) model locates the bounding box and rotation of the palm within the 2D frame.
2. Landmark Estimation: A dense convolutional regression network outputs 21 3D coordinates relative to the wrist keypoint.

2. Palm Normal Derivation via Cross Products

Determining the orientation of a user's hand in virtual space requires computing the normal vector perpendicular to the plane of the palm. Let the three primary anchor landmarks be:

• Landmark 0 (Wrist): W = [x_0, y_0, z_0]
• Landmark 5 (Index MCP): I = [x_5, y_5, z_5]
• Landmark 17 (Pinky MCP): P = [x_17, y_17, z_17]

We construct two planar vectors spanning the palm:

Vector V1 = I - W   (Wrist to Index Metacarpophalangeal)
Vector V2 = P - W   (Wrist to Pinky Metacarpophalangeal)

// Surface normal vector perpendicular to the palm
Normal Vector N = (V1 × V2) / || V1 × V2 ||

3. Quaternion Rotation Representation

Euler angles (pitch, yaw, roll) suffer from mathematical gimbal lock when the palm approaches 90-degree tilts. To ensure smooth continuous rotation without matrix singularities, Animatrous normalizes the palm vector into a unit quaternion:

import * as THREE from "three";

export function computePalmQuaternion(wrist: THREE.Vector3, indexMcp: THREE.Vector3, pinkyMcp: THREE.Vector3): THREE.Quaternion {
    const v1 = new THREE.Vector3().subVectors(indexMcp, wrist).normalize();
    const v2 = new THREE.Vector3().subVectors(pinkyMcp, wrist).normalize();

    // Palm normal vector
    const normal = new THREE.Vector3().crossVectors(v1, v2).normalize();
    
    // Orthogonal Y vector (finger pointing axis)
    const forward = new THREE.Vector3().addVectors(v1, v2).multiplyScalar(0.5).normalize();
    
    // Cross product to form complete orthonormal basis
    const right = new THREE.Vector3().crossVectors(forward, normal).normalize();

    const rotationMatrix = new THREE.Matrix4().makeBasis(right, forward, normal);
    return new THREE.Quaternion().setFromRotationMatrix(rotationMatrix);
}

4. Adaptive Kalman Filtering for Jitter Elimination

To prevent subtle tremors in user fingers from shaking virtual camera controls or 3D sculpt brushes, we implement a 1D discrete Kalman filter across each coordinate axis:

5. Conclusion

By combining MediaPipe WebAssembly execution with quaternion linear algebra and adaptive Kalman filtering, Animatrous delivers console-grade spatial hand tracking across standard consumer laptops without proprietary sensors.

Animatrous Engineering Lab

Dedicated to pioneering client-side AI, WebGPU graphics pipelines, and contactless spatial interaction.