Overview
The HandTrackingModule is a high-level wrapper around Google MediaPipe. It is designed specifically for Control Loops (robots, game controllers, mouse emulation).
Raw MediaPipe coordinates are jittery and tied to the webcam's resolution. This library filters the jitter (EMA smoothing) and maps coordinates to your specific robot's workspace, running asynchronously to keep your loop fast.
Class Reference
Constructor
Initializes the camera model and loads configuration files.
class HandTrackingModule(
smoothing_factor=0.5,
min_detection_confidence=0.5,
gestures_file=None
)| Parameter | Description |
|---|---|
smoothing_factor float |
Range: 0.0 - 1.0. Controls the Exponential Moving Average (EMA) filter.
• 0.1: Heavy smoothing. High lag. (Good for precision drawing).
• 0.5: Balanced (Default).
• 0.9: Raw input. No lag, high jitter.
|
min_detection_confidence float |
Range: 0.0 - 1.0. How "sure" the AI needs to be to recognize a hand. Increase this to 0.7+ if you are detecting "ghost hands" in background noise.
|
gestures_file str |
Absolute path to a custom .yaml file. If None, uses the library default. |
process_frame()
The main worker function. Call this once per loop iteration. It sends the image to the background thread and returns the latest available data.
def process_frame(image: np.ndarray) -> tupleReturn Values (Tuple of 6)
| Idx | Name | Type | Detailed Usage |
|---|---|---|---|
| 0 | annotated_image |
image | A copy of your input frame with the hand skeleton drawn on it. Use this for cv2.imshow() debugging. |
| 1 | gesture |
str |
The name of the detected gesture (e.g. "FIST").
Returns "NONE" if a hand is seen but no gesture matches.
Returns "UNKNOWN" if init is pending.
|
| 2 | norm_points |
list |
For Robots: A list of 21 (x, y) tuples.
• These are NOT pixel coordinates. • They are normalized (0.0 to 1.0) relative to the calibration.yaml box.
|
| 3 | ratios |
list |
Finger Straightness metrics (Thumb to Pinky).
• ~1.0 = Finger is straight.
• < 1.0 = Finger is bent.
Use to debug why a gesture isn't triggering. |
| 4 | pinch_metric |
float |
Distance between Thumb Tip and Index Tip.
• -1.0: Hand not detected.
• 0.0 - 0.20: Gripper Closed (Pinch).
• > 0.30: Gripper Open.
|
| 5 | handedness |
list |
List of strings: ["Right"], ["Left"], or ["Right", "Left"].
|
create_default_configs()
Utility to "eject" the internal config files to your local folder so you can edit them.
HandTrackingModule.create_default_configs()Developer's Guide
1. Correctly Mirroring the Camera
Webcams are usually mirrored for the user (like a bathroom mirror), but robots see the world directly. You must decide how to flip the image before passing it to the tracker.
# 1 = Horizontal Flip (Mirror Mode)
# Use this if looking at a screen (UI interaction)
frame = cv2.flip(frame, 1)
# 0 = Vertical Flip, -1 = Both
# Don't flip at all if the camera is mounted on the robot facing forward.2. Landmark Index Map
When accessing norm_points, use these indices to find specific parts of the hand.
- 0 : Wrist
- 4 : Thumb Tip
- 8 : Index Tip (Pointer)
- 12: Middle Tip
- 16: Ring Tip
- 20: Pinky Tip
Example: Get coordinates of the Index Finger tip for mouse control.
x, y = norm_points[0][8]3. Creating a Custom "Robot Mode"
To control a robot arm, you usually want to ignore the edges of the camera view (which are often distorted or contain clutter). Use calibration.yaml to define a "Safe Box".
# In calibration.yaml
# Ignore the outer 20% of the camera view
top_left: [0.2, 0.2]
bottom_right: [0.8, 0.8]
Result: When your hand is at (0.2, 0.2) in the raw camera frame, norm_points will report (0.0, 0.0).
Live Configuration
main branch. They represent the current default behavior of the library.
gestures.yaml
LIVE# Connecting to GitHub...calibration.yaml
LIVE# Connecting to GitHub...