A real-time monocular hand interface that combines MediaPipe geometry, a custom gesture classifier, a safety-aware command interpreter, and a backend-agnostic Franka cube-stacking controller.
Robotics / Vision
July 2026
ML + robotics engineer
Python, OpenCV, MediaPipe, PyTorch, PyBullet
End-to-end prototype
A standard webcam can recognize a hand, but it does not measure physical depth, custom gestures can flicker frame to frame, and directly mapping predictions to robot motion creates false activations and unsafe behavior.
The project converts webcam landmarks into invariant features, runs built-in and custom classifiers in parallel, debounces high-level commands through a safety state machine, and sends typed events into a simulator-independent robot backend.
The visual dashboard is only the front end. The core contribution is the interface between uncertain perception and deterministic robot control.
OpenCV captures RGB frames; MediaPipe returns 21 hand landmarks, handedness, built-in gestures, and relative landmark z.
A canonical 63-D feature vector feeds an optional MLP for finger heart, pinch, OK sign, and unknown.
Confidence thresholds, debounce, cooldown, conflict resolution, and a latched emergency stop convert labels into typed commands.
The same cube-stacking state machine runs on a deterministic mock backend or a PyBullet Franka Panda.
Subtract landmark 0 to remove translation in the image.
Normalize by the distance from index MCP to pinky MCP for scale invariance.
Mirror left hands into a right-hand coordinate frame.
Preserve MediaPipe geometry without presenting it as metric depth.
# 21 landmarks × (x, y, z) → 63-D vector
points = landmarks - landmarks[wrist]
scale = ||points[index_mcp] - points[pinky_mcp]||
points = points / max(scale, epsilon)
if handedness == "Left":
points[:, 0] *= -1
features = points.astype(float32).reshape(63)
The classifier was evaluated on session-disjoint samples. Public-data results are useful for engineering validation, but can be optimistic because classes came from different source datasets.
Open palm triggers a deterministic, latched stop with a shorter confirmation window.
Normal commands must persist for several frames and cannot immediately re-fire.
The robot receives structured emergency-stop, resume, reject, confirm, and gripper-toggle events—not raw strings.
A deterministic state machine moves from home to source, grasp, lift, target placement, release, retreat, and done. The controller is independent of the physics backend, so tests run on a mock robot while the same interface drives a PyBullet Franka.
The colored surface visualizes relative landmark geometry. A one-point palm-size calibration estimates range, but the project never claims an RGB webcam is LiDAR or ToF.
Missing, incompatible, or low-confidence custom checkpoints degrade to MediaPipe-only operation without crashing the camera loop.
Perception and task logic do not import the simulator. This keeps unit tests fast and makes a future real-robot adapter possible.
The next milestone is temporal control: swipe-based Cartesian nudges, push/pull depth commands, two-hand interaction, and validation on a single-camera labeled dataset to measure false activations under realistic domain shift.
Real-time monocular gesture and robot-control prototype