How to Build Gesture Controls for AR Apps with MediaPipe Hand Tracking on iOS & Android

TL;DR

• You will create: AR demo app with ability to interact with virtual objects with hand gestures (camera of the phone will track your hands).
• You will: Get code for MediaPipe Hand Tracking Install SDK Launch sample app on iOS/Android Add hand tracking functionality to your own AR app Try gestures on the device.
• You will need: No account (open source SDK), smartphone with camera (iPhone or Android (ARCore support)), tools for development (Xcode, Android Studio).

1) What is MediaPipe Hand Tracking?

Fig 1. MediaPipe Hand Tracking example output – 21 hand landmarks (finger joints) are tracked and shown as dots, with brighter dots indicating points closer to the camera.

MediaPipe Hand Tracking is a high-fidelity, real-time hand and finger tracking solution from Google. It uses machine learning to infer 21 3D keypoints (landmarks) of each hand from a single camera frame. It can track multiple hands at once (e.g. both hands) and even determines left vs. right hand automatically. Under the hood, MediaPipe employs a two-step pipeline: first a palm detector finds the hand's location, then a hand landmark model finds the 21 joint positions in that region. This model was trained on 30,000 real-world images plus synthetic hand images to improve robustness – it can handle various hand poses and even partial occlusion of fingers.

What it enables

• Markerless, controller-free input: Use your bare hands as input for AR/VR apps instead of physical controllers. The camera feed is analyzed to track hand movements and gestures in real time, enabling natural interaction.
• 21-point hand skeleton tracking: Get precise 2D/3D coordinates for 21 key hand landmarks (finger joints and tips) per hand. The high-resolution tracking enables to identify complex hand poses and gestures (e.g. pinches, thumbs-up) based on the landmark geometry.
• Cross-platform on-device ML: Runs in real-time on modern mobile devices (Android & iOS) without needing a cloud server. MediaPipe is optimized for mobile GPUs/CPUs, achieving 30fps hand tracking on a phone and even handling two hands simultaneously.

When to use it

• Hands-free AR/VR interactions: Ideal for mobile AR apps or VR experiences where users don't have dedicated controllers. For example, you can build touchless gaming setups or public AR installations where users use hand signs to interact.
• Creative applications, education: Gestures for interactive games, education, art applications. E.g. rehab or exercise games where patients use hand movements as input, or sign language learning apps that recognize hand shapes.
• No-touch user interfaces: In scenarios where touching a screen is impractical (kiosks, automotive HUDs, hygienic environments), hand tracking allows natural control. For UI navigation using mid air gestures (swipe, grab) in AR headsup or smartphone AR, without view obstruction.

Current limitations

• Camera field of view: Always make sure that the camera has a view of the hand. The hand interaction is restricted to what the camera sees. For example, in mobile AR it may only make sense that the user can interact with one hand (at a time) (since they are holding the phone with the other hand). This also means that the gestures should happen within an arm's length of the device.
• Lighting/background: Since it is a vision-based solution, it may suffer when the lighting is poor or when there is a complex background. This problem can be especially problematic in colored or low lighting situations. This might mean that you have to make sure that there is good consistent lighting or that you need to warn the user about not using backgrounds that could confuse the hand detector (e.g. skin colored or very cluttered background).
• No direct depth measurement. MediaPipe gives you the relative 3d coordinates of the hand, but it does not know the absolute distance from the camera. For spatial interactions (putting virtual objects in the world) you'll have to integrate with AR frameworks (ARCore ARKit) to translate 2d hand locations to real-world scale. Depth for hand landmarks is relative (e.g. fingertips vs palm) but it's not an absolute distance from the device.
• Performance considerations: Real-time hand tracking is computationally intensive. Older or low-end devices might achieve lower frame rates. Running this alongside AR rendering can tax the device, so use GPU acceleration if possible and optimize your app's frame processing. (MediaPipe's pipeline avoids redundant work by only re-running the palm detector when tracking is lost to maintain speed.)
• Preview SDK status: MediaPipe Hands (as part of MediaPipe Solutions) was initially released as a preview and beta. As of this writing, the APIs are fairly stable, but still marked as evolving. Check release notes for changes, and be cautious about shipping to production – test thoroughly on target devices and be prepared for updates as the library matures.

2) Prerequisites

Access requirements

• No sign-up required: MediaPipe is open-source. You don't need a special developer account or portal access. All libraries and sample code are publicly available on GitHub and Google's Maven/CocoaPods. (License is Apache 2.0, free for commercial use.)
• MediaPipe model file: You will need the pre-trained hand landmark model file. Google provides a .task model file (e.g. hand_landmarker.task) that the SDK uses. We'll download this in the steps below – no API key required, just a direct download.

Platform setup

iOS

• Xcode 13+ (with iOS 13.0 or later SDK). Ensure you can run apps on a device running iOS 13 or higher (iOS 16+ recommended for ARKit support).
• CocoaPods installed (for integrating the MediaPipe SDK). MediaPipe iOS tasks library is available as a pod. Install the latest Cocoapods and if necessary, run pod repo update
• Physical iPhone/iPad (highly recommended). Hand tracking is using camera, it won't work fully on iOS Simulator (no live camera feed by default in Simulator). You would need to run on physical device to test AR camera features iOS Simulator

Android

• Android Studio Arctic Fox(4.0) with Android SDK 30+ (Android 11+) Make sure you have a recent build environment to match the MediaPipe AAR. If you're using Kotlin, make sure to have
• Gradle 7+, Kotlin 1.5+ MediaPipe's Android solution is available as an AAR library via Gradle. You'd want a somewhat recent version of Gradle required by Android Studio
• Physical Android phone (recommended). It's possible to use an emulator which emulates camera input but it's easier to just run on a physical device which has a camera. The device should support ARCore if you plan to do world AR (see Integration section), but for just hand tracking a device with Android 8.0+ and a decent camera is sufficient.

Hardware or mock

• smartphone: The only thing you need is a regular camera. You don't need any depth sensor or special gloves or wearable. Just the usual "No controllers required" your hand is the only "input device". For AR apps, usually the rear camera is used for world tracking. However, the hand tracking also works with front cameras (e.g. for selfie AR filters). Hand (or recorded video): For testing, you'll use actual movements of
• your hand. Alternatively, for automated testing, you may also use a recorded video of your hand gestures as a mock input (feed into your app) for simulating the hand gestures in a consistent way during development. This may be useful in CI environments or if you cannot actually do gestures yourself.
• Good lighting environment: Not a hardware requirement per se, but ensure the area where you test has adequate lighting. This will significantly improve tracking stability (as discussed, dark or harsh lighting can impair the hand detection).

3) Get Access to MediaPipe Hand Tracking

1. Go to the MediaPipe project repository: Navigate to the official MediaPipe sample projects on GitHub. The hand tracking example code is part of the google-ai-edge/mediapipe-samples repo. You can clone it with Git:

   git clone https://github.com/google-ai-edge/mediapipe-samples.git
   
   

2. Retrieve the hand tracking sample: (Optional) After cloning, you can save time by using a sparse checkout to fetch only the hand tracking examples:

   cd mediapipe-samples
   git sparse-checkout init --cone
   git sparse-checkout set examples/hand_landmarker
   
   

   This will pull down only the hand landmarker sample apps (for iOS and Android). If you prefer, you can also download the repository ZIP and extract it, but the repo is large; sparse checkout is convenient to avoid downloading unnecessary files.

3. No special permissions needed: There is no need to request access or join any beta program the SDK is available publicly. (the AI Edge Portal from Google is just for analytics/benchmarking. Not necessary to use the library) No need to agree to terms (other than open source license you agree to this just by using the code).

4. create project (if for integrating into your existing app): if integrating into your existing app, make sure you create a project in xcode/android studio. you may want to make a copy of example to start with. (skip this if you're just running sample).

5. download model file:

   • ios: download hand landmark model file (e.g. hand_landmarker.task) from google's website. (you can find it in MediaPipe doc, or you can directly through the url for example you can use something like wget see Google's guide). after you download it, then add this file into your Xcode project. (for example, drag it into project navigator make sure you check "copy items if needed") This will bundle the model in your app.
   • Android: Similarly, download hand_landmarker.task and place it in your app's app/src/main/assets/ directory. In the sample code, they use a constant path to load this asset. Make sure the asset file name matches what the code expects (the sample uses MP_HAND_LANDMARKER_TASK which is set to "hand_landmarker.task").

Done when: you have the MediaPipe Hand Landmarker sample code on your machine, and the model file (hand_landmarker.task) is downloaded and added to the project (iOS bundle or Android assets). At this point, you're ready to build and run the sample apps with hand tracking capability.

4) Quickstart A — Run the Sample App (iOS)

Goal

Run the official iOS hand tracking sample app and verify that hand landmark detection works on a real device's camera (and by extension, that gestures can be recognized with your hands in front of the device).

Step 1 — Get the sample

• Clone or open project: If you followed the steps above, open the Xcode project located at mediapipe-samples/examples/hand_landmarker/ios/ (it may be an Xcode workspace if CocoaPods are used). The sample is typically an Xcode project configured with the necessary files.
• Alternatively, download the sample as a ZIP from GitHub. Unzip it and open HandLandmarker.xcodeproj (or .xcworkspace) from the iOS example directory in Xcode.

Step 2 — Install dependencies

• Install MediaPipe SDK via CocoaPods: The sample uses CocoaPods to include the MediaPipe Tasks SDK. Navigate to the ios sample folder in Terminal and run pod install. This will fetch the MediaPipeTasksVision pod (which contains the Hand Landmarker library). After this, open the generated .xcworkspace if not already open.
  • Note: Ensure you have CocoaPods installed (sudo gem install cocoapods if not) and an updated pods repo. The Podfile in the sample should already list pod 'MediaPipeTasksVision', which brings in the hand tracking SDK.
• (SPM alternative): If preferred, you could integrate via Swift Package Manager by referencing the MediaPipe GitHub, but as of now the official docs recommend CocoaPods. Stick to the Pod for simplicity.

Step 3 — Configure app

Before running, check a few configuration details:

• Add the model to the app bundle: If you haven't already, ensure hand_landmarker.task is added to the Xcode project and is included in the app target. In Xcode, you should see this file in the Project navigator (the sample might already include a placeholder or instructions to add it). In code, the sample will load it via Bundle.main.path(forResource: "hand_landmarker", ofType: "task") – if that returns nil, the file isn't bundled correctly.
• Bundle Identifier: Optionally, set a unique Bundle ID for the sample app in the project settings if you intend to run it on your device (especially if an app with the same ID is already installed). For example, change com.google.mediapipe.handlandmarker to something like com.yourname.HandSample and update the provisioning profile if needed.
• Privacy permission: In Info.plist, verify that a NSCameraUsageDescription key is present with a string explaining why the app uses the camera. iOS will require this for camera access. (The sample likely has this, but double-check to avoid a runtime crash when accessing the camera.)
• Other capabilities: No special entitlements are needed since we are just using the camera. (If this were ARKit-based, you'd also enable ARKit capabilities, but the sample focuses on the ML aspect.)

Step 4 — Run

1. Select the target device: In Xcode, choose your iPhone as the run destination (connect your device via USB or network and select it in the scheme dropdown).
2. Build & Run: Hit the Run button (▶️). Xcode will compile the app and install it on the device. The first build may take a bit longer as it compiles the MediaPipe pod.
3. launch and see. app should launch on your iphone. probably open to some camera view with some UI to switch between camera live view and maybe still image view (sample implementation depends)

Step 5 — Connect to camera input

• Give camera access You'll see an iOS permission popup for camera access the first time you run. Please select "Allow". If you miss or select "Don't Allow", you'll have to enable the access in Settings so that the app can work.
• Look at your hand with the camera Position the camera on the device to make sure one of your hands is visible in the full frame of view. You might want to position it in front of a clean background at first.
• (No additional wearable or controller is required. Just use your hand as input. Just ensure the camera can see it clearly.)

Verify

• Landmarks visible: You should see the app drawing hand landmarks on your hand in real-time – typically 21 points overlaying your fingers and palm joints. In some samples, connections between points (a skeleton outline) are drawn. Move your hand around; the landmarks should follow.
• Basic gestures register: Try making a fist, open hand, or a thumbs-up. While the sample might not explicitly label gestures, you can verify the landmarks move accordingly (e.g., points bunch together for a fist, etc.). If the sample has a mode to detect gestures (some MediaPipe examples have a "Gesture Recognizer" mode), you might see text feedback for recognized gestures.
• Multiple hands (if supported): If you have a friend nearby or can manage using the front camera, try getting two hands in view. The model can track two hands simultaneously if configured. The sample might only track one by default, but it's a good stress test – it should at least not crash with a second hand in frame.

Common issues

• build errors: Missing modules/architectures make sure after installing pods you are opening the.xcworkspace file (not the.xcodeproj). Also make sure to build for device (the pod might not support simulator for ARM M1. Run on a device).
• Black camera view: If the app runs but shows a black screen, likely camera permission was denied or not working. Check the device Settings -> Privacy -> Camera and make sure your app has access. Also ensure your device isn't using the camera elsewhere (though iOS generally handles that).
• no hand-points showing up (landmarks not detected): You should be able to see the camera feed but there are no hand points showing.
  • Make sure you have your model file loaded. (check the debug output in Xcode. If it can't find the model file, there will be an error). If you do, add the model file correctly and rebuild.
  • Ensure your hand is within frame and reasonably centered. Maybe the detector doesn't catch your hand that is cut by edge or small/big in the view.
  • Try to improve the light conditions or use a simpler background if the environment is difficult.
• App crash when it starts. Maybe because of missing permission usage description (check Info.plist) or some code problem. Use debugger to get the log. Maybe you forgot to add the model or wrong path. It maybe raise an exception when you init the HandLandmarker. (it should have error handling in the sample code but check it anyway) fix the issue (correct the resource name for example) and try again.

5) Quickstart B — Run the Sample App (Android)

Goal

Launch official Android hand tracking sample app. Make sure it can detect hand landmarks from the phone camera. This confirms the MediaPipe hand tracking is working on Android, so you can then integrate it into an AR app.

Step 1 — Get the sample

• Import project: Using Android Studio, import the Android sample project found in mediapipe-samples/examples/hand_landmarker/android/. If you cloned with sparse checkout, open that folder as a project. It should contain an Android app module (with Gradle files, etc.).
• Gradle sync: Android Studio will likely prompt to sync Gradle. Let it download any Gradle wrapper and plugin updates. The project should be configured with the necessary dependencies (we'll verify in the next step).

Step 2 — Configure dependencies

• MediaPipe Tasks dependency: The sample app uses the MediaPipe Tasks Vision library. In the app-level build.gradle, ensure you see the dependency declaration for Hand Landmarker:

  implementation 'com.google.mediapipe:tasks-vision:latest.release'
  
  

  This brings in the MediaPipe vision AAR which includes hand tracking. The latest.release will resolve to the latest version (e.g., 0.10.+). If Gradle fails to find it, make sure you have Google's Maven repository enabled in your project (repositories { google() ... } in build.gradle). The sample likely has it set already.

• Authentication token (not needed): MediaPipe artifacts are public, so you do not need any API token in gradle.properties. If you see references to needing a token in some contexts, that might be for certain Google services, but not for this library. The dependency should download without credentials.

• Sync project: After confirming the dependency, click "Sync Project with Gradle Files" in Android Studio. It should download the com.google.mediapipe:tasks-vision library and any others. If it completes without errors, you have the SDK ready.

Step 3 — Configure app

• Set applicationId (optional): The sample comes with an applicationId (e.g. com.google.mediapipe.handlandmarker). If this clashes with an existing app on your device, you can change it in the app's build.gradle (applicationId "com.yourcompany.handdemo"). Generally, it's fine as is.

• Add permissions in AndroidManifest: Open AndroidManifest.xml. Ensure it includes:

  <uses-permission android:name="android.permission.CAMERA" />
  
  

  The sample should have this already, as camera access is essential. If the sample allows reading from gallery, it might also have READEXTERNALSTORAGE or the newer READ_MEDIA_IMAGES for Android 13+. For our purposes, camera permission is key.

• AR support (if planning ARCore): (Not in sample by default) If you intend to extend this to an ARCore app, you would add:

  <uses-feature android:name="android.hardware.camera.ar" android:required="true"/>
  
  

  and include ARCore dependencies. The sample doesn't do AR, so this isn't needed just to test hand tracking, but remember this for integration later.

• Check model file placement: Verify that hand_landmarker.task is present in app/src/main/assets/. The sample's code will try to load this asset by name. If you don't see it, add the model file (download as mentioned earlier) into the assets folder. Without it, the app will not be able to initialize the hand tracker.

Step 4 — Run

1. Connect your Android device: Enable USB debugging on your phone and connect it to your PC. Ensure Android Studio recognizes it (you might need to approve the PC RSA key on the phone).
2. Select Run configuration: In Android Studio, select the app module run configuration (usually already selected by default). Choose your device as the target.
3. Run the app: Click the Run ▶️ button. The app will build (Gradle will compile the code and package the APK) and install it on your device. Watch the Run console for any issues.
4. Launch on device: The app should auto-launch on your phone (or you can find the app icon, typically "Hand Landmarker", and open it).

Step 5 — Connect to camera input

• Approve camera permission: Android will prompt you "Allow app to take pictures and record video?" when it first tries to use the camera. Grant this permission.
• Test with your hand: You should see the camera preview on-screen. Put your hand in front of the camera. Try to move it around to see if the app will start marking landmarks on your hand
• (you don't need any Bluetooth/wearable, just the camera. If the sample has a mode selector (image vs live) switch, make sure it's on live camera mode to continuously track)

Verify

• hand landmarks: App should put points (and maybe lines between them) on your hand in camera feed. Maybe you'll see colored dots at each knuckle and fingertip. That's a sign your hand tracking ML model is working.
• Real-time tracking: Try moving your hand at moderate speed or changing your gestures. It should track with little delay. (Should be ~30 FPS on modern phone.) If you see a significant delay or choppy updates, the device might be struggling check logcat for warnings (e.g., fallback to CPU if no GPU available can slow it down).
• Gesture functioning: Try a thumbs-up or peace sign. The sample might not display gesture names (unless you're running a GestureRecognizer variant of the sample), but visually confirm that the landmarks align with the gesture (e.g., for thumbs-up, you should see the thumb's points extended upward, other fingers curled).
• Multiple hands: If you have the ability, test two hands in view. The default config might only track one (to save resources). If only one hand is being tracked at a time, that's expected with default numHands=1. You could increase it in code to test, but be aware of performance. At least ensure the presence of a second hand doesn't break tracking of the first.

Common issues

• Gradle build fails (dependency not found): If Gradle cannot resolve com.google.mediapipe:tasks-vision ensure you have mavenCentral() or google() in the repositories. Also check your internet connection for Gradle. You're dependent on google's Maven (no need to log in). Sometimes you'll need to add google() to project and module build.gradle.
• App crashing on startup. Look in logcat for errors. Often your assets are missing the model file. It may throw an exception trying to load the model. You'll see "asset not found" or a file open error in the logcat. If that's the case, add the hand_landmarker.task to the assets and try again. Another possibility is that the camera permission is missing in the manifest (should've done that in step 3). On android 11+ it'll crash if you forget that when trying to get the permission. Add the <uses-permission> and install again.
• No camera feed/blank screen. If you allowed the permission but aren't seeing a camera preview you may not have a device that supports the camera api you're using or it may be in use by another app. Close other camera apps. Also, some devices (or if using an emulator without camera) will show black. Use a real device with a working camera. Check logcat for any camera-specific errors.
• Landmarks flicker or not stable: If the hand is at the edge of the frame or partially out, the detector may lose it and re-detect repeatedly. This can cause flicker. To improve stability, keep the hand fully in view and not too small. Also, rapid motion can momentarily break tracking until the palm detector catches up.
• Low performance on older device: If running on a lower-end Android, you might get low FPS. Ensure the sample is using GPU if available (the MediaPipe Tasks library usually uses CPU by default on Android unless you enable acceleration via options). You can try enabling the Android GPU delegate in the code (if not already) via HandLandmarkerOptions.builder().setExecutor(…something…) or simply be aware that older phones will have some lag. Some
• app says "Connected=false" or something: There may be something appearing in some examples. If the status says 'not connected' or 'waiting' it might be waiting for a second device or a sensor. But in the Hand Landmarker example, it should just work. If it has something that displays the status, check what it says on the screen or see if you need to press something to trigger the camera starting. It should normally just start automatically.

6) Integration Guide — Add MediaPipe Hand Tracking to an Existing Mobile AR App

Goal

Integrate the MediaPipe Hand Tracking SDK into your application to enable a hand gesture-based feature in an AR experience. We'll walk through how to integrate the hand tracking pipeline into your app (either iOS or Android) and use it to enable an example interaction (eg. gesture-based control of a virtual object).

Architecture

Think of the integration in layers:

• Camera AR Feed Hand Tracker Gesture events App logic. I guess you have probably ARKit (for iOS) or ARCore (for Android) setup for the camera feed and world tracking in your app. Then you will access your camera feed (or a separate one) to send image to the MediaPipe. On each frame you have a MediaPipe HandLandmarker which will detect the hand landmarks. Then you will check for some gesture given the hand landmarks. Then you will trigger some app's UI or game logic, e.g. "pinch fingers" grab/select AR object.

A possible structure:

• App's UI. e.g. Unity scene or native AR view where you see virtual objects.
• MediaPipe Hand Tracker (SDK client). This one will handle the ML model. Process the camera frames to get landmarks. Some
• gesture recognizer logic. This one will translate landmarks to high level gestures. e.g. make a fist, do a thumbs up.
• Application state controller: takes gesture events and performs actions in the AR scene (e.g., toggling an object, moving something, or taking a photo, depending on your feature).

Step 1 — Install the SDK

Bring MediaPipe into your project as a dependency:

iOS (Xcode project):

• Add the MediaPipe Tasks SDK via CocoaPods or Swift Package. E.g., in your Podfile:

  pod 'MediaPipeTasksVision'
  
  

  then pod install. This gives you access to HandLandmarker and related classes in Swift/ObjC. If using Swift Package Manager, point it to Google's repo (if supported) or add the .xcframework manually (advanced).

• Import the library in code with import MediaPipeTasksVision.

Android (Gradle):

• In your app module build.gradle, add:

  implementation "com.google.mediapipe:tasks-vision:0.10.29"  // or latest version
  
  

  (Replace latest.release with an explicit version for stability, e.g., 0.10.29 which is a 2025 release.) Ensure minSdkVersion of your app is at least 19 (as required by MediaPipe, usually). Sync Gradle to pull the library.

• Import classes in your Kotlin/Java code:

  import com.google.mediapipe.tasks.vision.handlandmarker.HandLandmarker
  import com.google.mediapipe.tasks.vision.handlandmarker.HandLandmarkerResult
  // etc.
  
  

  These classes come from the tasks-vision package.

Step 2 — Add permissions

Integrating into an AR app means your app likely already has camera permissions, but double-check:

iOS (Info.plist):

• Ensure NSCameraUsageDescription is present with a user-facing reason (e.g., "This app uses the camera to track your hand and enable AR interactions."). Without this, iOS will block camera access.
• If you plan to use the front camera, also consider adding NSFaceIDUsageDescription if needed (only if using TrueDepth or face data; for simple hand tracking with back camera, not needed).
• NSMicrophoneUsageDescription – not needed for hand tracking unless your app uses audio.
• Privacy - Motion usage – not needed here.
• Bluetooth permission – not needed (no external device).

Android (AndroidManifest.xml):

• The app must have <uses-permission android:name="android.permission.CAMERA" />.
• If using Android 13+, and you want to pick images from gallery as fallback, add READ_MEDIA_IMAGES permission (or legacy READ_EXTERNAL_STORAGE for older Android).
• If your AR app uses other sensors or features, those permissions should already be in place (e.g., coarse location for ARCore's geospatial features, if any not related to hand tracking though). And
• You might also include <uses-feature android:name="android.hardware.camera.ar" ...> if distributing an ARCore-dependent app so that only ARCore-capable devices install it.

Step 3 — Create a thin client wrapper

It's a best practice to isolate hand tracking logic into a component to not pollute your UI code with ML stuff. You'll need classes/services like:

• HandTrackingManager (analogous to WearablesClient): Initializes HandLandmarker SDK and takes care of camera frames processing, e.g. opens the camera (unless it's already opened as part of AR session), subscribes to AR frame updates. Exposes methods like startTracking(), stopTracking(). May expose state (e.g. is there currently detected hand)
• GestureRecognitionService: Based on data provided by HandTrackingManager (landmarks), this service performs gesture interpretation. For instance, it could have logic to detect a "pinch" by measuring distance between thumb tip and index tip landmarks, or detect a "fist" by comparing curled finger landmark angles. This can be as simple or complex as needed initially, implement one gesture for your feature.
• PermissionsService: (if you don't have one) – to handle requesting camera permission and any other permission gracefully at runtime. Ensure this is called before starting camera/hand tracking.

Integrate these in your app's lifecycle:

• Initialize HandTrackingManager (and load the model) on app launch or when you enter the AR feature screen. E.g., on iOS, you'll load the model into memory by creating HandLandmarker with the model path; on Android, call HandLandmarker.createFromOptions(context, options) to get an instance. This may take a brief moment (it loads TFLite model).
• Connect to the camera feed: If using ARKit/ARCore, you can get the camera frames from those APIs (ARKit's ARFrame.capturedImage, ARCore's frame.acquireCameraImage() or via OpenGL texture). Or simpler, start a secondary CameraX or Camera2 pipeline just for hand tracking. Choose one to avoid conflicts (ARCore can sometimes share camera if configured for CPU image output).
• Each frame (or each AR update): pass the image to MediaPipe. On iOS, you might use HandLandmarker.detectAsync (if available) or the appropriate API to feed a CMSampleBuffer or UIImage each frame. On Android, use HandLandmarkerResult result = handLandmarker.detect(videoFrame) or if using live stream mode, set a listener that gives results asynchronously.
• Handle the result: You'll get a set of hand landmarks (and possibly a handedness label, and a flag if hand present). If landmarks are present, feed them to your GestureRecognitionService to see if a target gesture is performed.
• Emit events: If a gesture of interest is detected, have the system call back into your UI/game logic, e.g., via a delegate or LiveData/Flow (Android) or a Swift delegate/closure (iOS).
• Manage lifecycle: Start tracking when the AR view appears (and permission is granted), stop tracking when view disappears (to free camera/CPU). Also handle errors: e.g., if model fails to load or camera fails to open, show an error to user.

Definition of done:

• MediaPipe hand tracker initializes without errors when the feature starts (e.g., no missing model file, no incompatible device issues).
• The camera feed is successfully being processed by the hand tracker (you get landmark results in your logs or callbacks).
• When your hand is in view, the system detects it (even if you're not yet acting on it, you can log "Hand detected" to verify).
• When you perform the chosen gesture, the system recognizes it (e.g., you log "Pinch gesture detected").
• Clean up happens on exit – the camera and tracker stop so they don't consume resources in other parts of the app.

Step 4 — Add a minimal UI screen

Design a simple interface or feedback mechanism for the hand-controlled feature. This can be in your AR view or overlaying your AR view:

• Enable Hand Control toggle: Have a UI switch or button to turn on/off (if not always on) hand tracking. This could call your HandTrackingManager start/stop.
• Connection/status: As there's no external device, this could just be an icon or text on-screen e.g. Tracking" when it detects a hand. Just some way to show you're detecting a hand. e.g. you could have a small green dot or "Hand: appear when there's a hand and gray or "Hand: when there isn't.
• Virtual object or target UI: If you're controlling an object, put that object in the AR scene. Maybe just a 3D cube for demo purposes or a balloon that pops in for a quick demo.
• Gesture action button (if needed): In some cases, you might still include a traditional UI button for fallback. But ideally, the gesture itself triggers the action. However, having a debug button "Perform Action" can help testing alongside the gesture.
• Result display: E.g., if the gesture triggers capturing a photo or spawning an object, show the result. For instance, if a screenshot is taken, show a thumbnail. If an object is placed, visibly show it in scene or list it.

For our running example (say, pinch to spawn a virtual object), the UI could be:

• A short text: "Pinch your index and thumb to place a cube."
• A counter or indicator that an object was placed.
• Perhaps a "Reset" button to remove all spawned objects (for testing repeatedly).

Keep the UI minimal so as not to clutter the AR view – use translucent overlays or small icons. The focus is that the user's hand is doing the main interaction.

7) Feature Recipe — Use a Pinch Gesture to Toggle an AR Object

Goal

Implement a specific gesture-driven feature: when the user pinches their thumb and index finger together (like a "air click" gesture), the app will spawn a virtual object in the AR scene (or toggle its visibility). Pinch again, and the object disappears. This simulates a basic "select" action using hand gesture, without touching the screen.

UX flow

1. Connected & ready: The app is running the camera and has hand tracking active (user's hand is being tracked).
2. User performs pinch: They bring thumb and index finger together in view of the camera.
3. Gesture recognized: The app detects the pinch gesture.
4. Action triggered: The app places a virtual object (e.g., a 3D cube) at a predefined location in the AR scene, or toggles it on/off.
5. Feedback given: The object appears (if toggled on), possibly accompanied by a brief confirmation (sound or visual flash). The user opens their hand or un-pinches.
6. User can repeat: Each distinct pinch (thumb-to-index touch) will toggle the object's presence or state.

(This is a simple recipe – in a real scenario, you might raycast from the hand to place objects where the user is "pinching", but that involves more AR logic. Here we keep it straightforward.)

Implementation checklist

• Hand tracking running: Ensure the HandTrackingManager is updating landmarks every frame.
• Permissions okay: Confirm camera permission was granted (otherwise this feature should be disabled or prompt user).
• Detect pinch gesture: In your gesture recognizer, implement logic to detect pinch:
  • Identify landmark indices for thumb tip and index finger tip (in MediaPipe, thumb tip is landmark 4, index tip is 8 in the 21-point scheme).
  • Calculate the normalized distance between those two points. For example, in 2D image coordinates, get dx = x_thumb - x_index, dy = y_thumb - y_index, distance = sqrt(dx^2 + dy^2).
  • Determine a threshold: when fingers are apart, distance is larger; when pinched, distance becomes very small. You might set a threshold like "distance < 0.1 (in normalized units)" to consider it a pinch. Tune this if needed.
  • Additionally, ensure it's a deliberate pinch: maybe require that other fingers are relatively straight or not confound (for basic, distance is fine).
• Debounce gesture. Don't want repeated toggling when fingers are together. Keep a boolean state (pinchActive). If distance is below threshold, but pinchActive was false, it means just started new pinch event. Trigger the action, and pinchActive true. If distance above threshold (fingers released), pinchActive false. So only one event per pinch (press-release) happens.
• Trigger AR action. When you recognize pinch event (just thumb touch index).
  • If object is not in scene put it there. If object is in scene take it out. (For example, if using ARKit/ARCore, you might add or remove an ARAnchor with a model. Or if using SceneKit/Sceneform, toggle node visibility.)
  • As feedback, perhaps change the object's appearance briefly or play a sound.
• Handle multiple pinches: Each time the user pinches again (after unpinching), the toggle happens again.
• Edge cases: If the user keeps holding pinch for a long time, your debounce logic ensures it doesn't spawn multiple items. If the user's hand goes out of view while pinched, you might consider that a canceled gesture – but in this simple toggle, it's okay; it will just remain in whatever state last toggled.

Pseudocode

Here's a simplified pseudocode for the pinch detection logic within a frame update loop:

// Swift-style pseudocode (could be similar in Kotlin/Java)

var pinchActive = false // state to track if currently pinching

var objectVisible = false

func onNewHandLandmarks(landmarks: [Point]) {

    guard landmarks.count == 21 else { return } // need full hand

}

let thumbTip = landmarks[4]

let indexTip = landmarks[8]

// compute squared distance in normalized coords for simplicity

let distSq = (thumbTip.x - indexTip.x)^2 + (thumbTip.y - indexTip.y)^2

let threshold: Float = 0.005 // tuned threshold (squared) for "touch"

if distSq < threshold {

if !pinchActive {

// Pinch just began

pinchActive = true

objectVisible.toggle() // flip state

if objectVisible {

placeVirtualObject() // show object in AR

showFeedback("✅ Object placed")

} else {

removeVirtualObject() // hide object

showFeedback("❌ Object removed")

}

}

} else {

// fingers apart

pinchActive = false

}

This assumes onNewHandLandmarks is called each time we get an update (perhaps every video frame or every few frames).

You would integrate this into your HandTrackingManager's callback. For example, on Android if using live stream mode, you set a ResultListener that gives you HandLandmarkerResult each frame; from that you extract the first hand's landmarks and apply logic. For iOS, you can synchronously call handLandmarker.processFrame() to get results, or you can use delegate if you want to live stream.

Troubleshooting

• Pinch not detected consistently: perhaps you need to change the threshold. Too low and you'll miss small gaps, too high and you might have false positives if you just touch with your fingers. Print the distance while you try and fiddle to get the right value. Also make sure you're not too far from the camera. If your hand is really small on the image it could mean that the normalized distance is always small. Maybe use a ratio (say the distance from thumb to index vs the distance between index MCP (knuckle) and wrist as a scale).
• Gesture misfires when not intended: Sometimes if you bring other fingers close, or your hand rotates profile, the thumb/index might appear close in 2D even if not pinching (occlusion). If you get false triggers, you can add an extra check: for example, also ensure that the angle of the index finger is bent (if doing more complex geometry), or that other fingertip distances are larger (to ensure it's specifically thumb-index touching). For basic usage, you might live with an occasional false positive but document that best practice is to face the camera directly.
• No response in AR scene: If the object isn't appearing/disappearing, verify that the gesture logic is actually toggling objectVisible. Add logging or on-screen text when gesture triggers to ensure that part works. If it does, then the issue is likely with how you place/remove the AR object:
  • Check that you are adding the AR anchor on the main thread (for ARKit/ARCore).
  • If using a fixed position, ensure it's within view. For example, you might just place the object 1 meter in front of the camera for simplicity.
  • If you don't see anything, try dumbed down versions of it like instead of AR just draw 2D icon on screen to verify that your toggle is working and then debug your 3D placement. Expectations for
• continuous gesture (hold pinch). For us we toggle it on when you start pinch. If you wanted something like "hold pinch to hold an object" then you would do something else (like spawn when you start pinch, maybe move until pinch stop). For now we toggle. So that should be obvious to user (maybe something like "pinch again to remove")
• Multiple gesture interference. When you add other gestures later on make sure you either prioritize or make them distinguishable. Like maybe you have open palm gesture vs pinch. If user is switching then one gesture should end before you start another gesture so there is no confusion. Using state machines can help (not needed for single pinch gesture).

8) Testing Matrix

now that you have gesture controls using your hand, you need to test your app in all sorts of different scenarios to ensure it is working well:

of confidence). All of this helps to ensure good robustness of your gesture control feature. Pay special attention to the background/lighting conditions, as those are the main factors affecting camera-based tracking.

9) Observability and Logging

For production maintenance and debugging of the feature we want to put in some logging/analytics for key events to get a sense of how users are engaging with it and potential issues: log

• Startup events: Log when hand tracking starts and stops.
  • hand_tracking_start – when the camera and model are activated.
  • hand_tracking_stop: when stopping hand tracking (camera and model are turned off) (e.g., exit AR mode)
• log if we have permission issues for camera (not granted) or user prompted for it.
  • hand_tracking_permission_denied: (maybe this is just a one time log if user permanently denied? so you can see of users that can't use it) log
• Hand detection events:
  • hand_found – each time a hand is detected after not being present.
  • hand_lost – each time tracking loses the hand (e.g., hand leaves frame). You might also log duration the hand was present.
• Gesture attempt events:
  • pinch_start – when a pinch gesture is first recognized (even if it doesn't lead to a toggle because maybe already pinched).
  • pinch_end – when the pinch is released.
  • If you implement other gestures, similarly log fist_start, etc.
• Gesture action outcome:
  • pinch_toggle_on – object was shown.
  • pinch_toggle_off – object was hidden.
  • If you have more complex outcomes, log those (e.g., if pinch selects one of several objects, log which one).
• Performance metrics: It could be useful to log the average processing time or frame rate:
  • hand_tracking_fps (maybe periodically log how many frames per second the tracking runs, or log if it drops below a threshold).
  • hand_tracking_inference_ms – average milliseconds per frame for hand landmark model. If you have access to this (maybe not easily via the high-level API), could instrument.
• Error logging: If the MediaPipe throws an error (e.g., model load failed, camera failed):
  • hand_tracking_error with details (exception messages, etc.).
  • If using a logging service, surface these so you know if many users are hitting an issue.

By analyzing these logs, you can answer questions like: How often do users engage with hand gestures? Are there many false triggers (e.g., hand_found but no pinch_start)? Does tracking frequently stop and start (maybe indicating users moving out of frame a lot)?

Additionally, if you have analytics, you might track a funnel:

• How many users enter the AR feature vs how many actually perform a gesture successfully. This can gauge usability.

Make sure logging does not spam too much (e.g., don't log every single frame). Focus on state changes and user-triggered events. Logging and analytics will ensure you can observe the feature's performance in the wild and iterate on improvements (like adjusting gesture recognition parameters or updating the model version in future).

10) FAQ

• Q: Do I need any special hardware (depth camera, gloves, etc.) to use MediaPipe hand tracking?

  A: No special hardware is needed – just a standard RGB camera on your device. MediaPipe Hand Tracking works with the regular camera feed, using AI to infer depth and landmarks. You don't need depth sensors or markers on the hand. This makes it usable on most smartphones. Just be mindful of lighting and camera positioning for best results.

• Q: Which devices and platforms are supported?

  A: MediaPipe Hand Tracking supports Android and iOS devices. It's been demonstrated to run in real time even on mid-range phones. For Android, devices should ideally support ARCore if you're doing AR (for world tracking), but the hand tracker itself doesn't require ARCore. On iOS, any device running iOS 13+ with a camera (iPhone 6s or later) can run it; newer devices (with A12 Bionic or later) will perform better thanks to neural processing units. There is also a web version and Python version of MediaPipe Hands for other environments, but for this guide we focus on mobile apps.

• Q: Can I use this in a production app?

  A: Yes, many apps use MediaPipe for hand tracking (e.g., some filter apps, games). The code is Apache-licensed. However, note that as of late 2023 it was labeled preview, so you should test thoroughly. The models are quite robust (trained on many images) and work in real time, but you'll need to account for edge cases (lighting, etc.) in your UX. Also, keep the SDK updated – Google continues to improve it (the version 0.10+ in 2025 is much improved over earlier iterations). Always comply with platform privacy rules (e.g., if you use the camera in background, disclose it, etc.). In short, it's production-capable, but do your due diligence with testing and user guidance.

• Q: What gestures can it recognize out of the box?

  A: By default, the Hand Landmarker gives you raw landmark positions – it doesn't label gestures. You (the developer) interpret those for gestures you care about. Google provides a Gesture Recognizer task as well, which builds on the hand landmarks to classify a set of common gestures (like "Closed Fist", "Open Palm", "Pointing", "Victory/Peace", etc.). If you use that, you can get high-level gesture names directly. The set of gestures is somewhat limited (mainly basic ones and some letter shapes, depending on the model). For custom gestures (say a unique sign or a complex movement), you would need to implement logic or even train a custom model. MediaPipe's flexibility allows you to use the landmarks as inputs for your own classification algorithm (even a simple rules-based as we did for pinch, or a machine learning model if needed).

• Q: Can I track multiple hands and do interactions between them?

  A: Yes, MediaPipe can track multiple hands (you can configure the max number of hands, e.g., 2). It will give you separate landmark sets for each hand and an identifier for left vs right. You can definitely enable two-hand interactions (like pinch your hands together, or one hand controls something the other hand reacts to). Keep in mind performance will be a bit lower with two hands and more complex logic. Also, if the hands overlap in view, tracking can be momentarily confused (one hand covering the other might cause one to disappear until visible again). But many demos (and research) have used two-hand tracking successfully. Just set numHands to 2 and design your gestures accordingly (and test the edge cases of hands switching or overlapping).

• Q: How does this compare to platform-specific solutions (ARKit Hand Tracking or others)?

  A: Apple's ARKit as of iOS 17 does not provide full hand skeleton tracking – it offers finger skeleton for the index finger if using LiDAR (for hand interactions in VisionOS) and basic pinch detection in ARKit (2D pinch detection with ARSCNView). But it doesn't give 21 joints of the hand like MediaPipe does. Apple's Vision framework has a HandPose request (introduced in iOS 14) which does detect hand joints in images, similar to MediaPipe. If you target only iOS and want a native approach, you could use Vision's VNDetectHumanHandPose request, which provides up to two hands' joint locations. MediaPipe's advantage is cross-platform and potentially more optimized for real-time video (Vision might be a bit slower or have lower frame rate, though it's improved with Apple Neural Engine). On Android, there isn't an official equivalent (there are third-party libs like ManoMotion or Google ML Kit had some gestures but not full hand skeleton). MediaPipe fills that gap on Android. So if you need one codebase for both, MediaPipe is a great choice. If you're on a single platform, you can consider these trade-offs. In this guide, we chose MediaPipe for broad applicability and proven performance.

• Q: Can I use hand tracking for complex gestures like sign language recognition?

  A: MediaPipe provides the raw landmarks which is a great starting point for sign language recognition research. However, recognizing full sign language involves not just static hand pose but motion and both hands, plus facial expressions sometimes. You would need to build a classification system on top of the landmarks (like a sequence model – e.g., an LSTM or Temporal CNN – to interpret sequences of hand poses into signs). There are academic projects doing exactly this. The MediaPipe team's gesture recognizer isn't that advanced to do full sign language out of the box, it's more for basic gestures. So yes, you can use MediaPipe as a foundation (and it's likely one of the best real-time hand trackers to use for this), but the gesture recognition for sign language would be your custom implementation or model. Expect to gather training data and develop an ML model if you aim for comprehensive sign language interpretation.