How to Implement an Audio-Guided Pick/Pack Workflow with Smart Glasses on iOS & Android

TL;DR

• You’ll build: a warehouse picking app that connects to camera-enabled smart glasses, streams what the wearer sees, and provides spoken pick/pack instructions in real time.
• You’ll do: Get developer access → Install the Smart Glasses SDK → Run the official sample app → Integrate the SDK into your own app → Test with a wearable device or simulator.
• You’ll need: a developer account (Meta Wearables preview), a pair of supported smart glasses (or a mock device kit), and a development environment (Xcode 15+ for iOS, Android Studio for Android).

1) What is the New Solution (Smart Glasses + Phone Compute)?

Smart glasses paired with a smartphone unlock a hybrid “vision + voice” picking experience. In this setup, the glasses act as eyes and ears on the warehouse floor, while the phone app provides the brains and connectivity. What does this enable?

Vuzix's warehouse-focused smart glasses combine voice prompts with visual scanning for hands-free picking (voice + vision workflow). They offer at-a-glance pick data, integrated barcode scanning, and voice control, allowing workers to boost productivity without juggling paper or handheld scanners.

What it enables

• Hands-free vision picking: The glasses' built-in camera lets workers scan barcodes and confirm items just by looking at them, eliminating handheld scanners. This head-up approach led to a ~15% productivity boost in DHL's trials, with faster picks and fewer errors.
• Real-time audio guidance: The phone delivers step-by-step instructions via the glasses' speakers (or bone-conduction audio). This is similar to traditional voice picking systems, but now enhanced with visual context from the glasses.
• AI-assisted workflows: The phone can run AI/vision algorithms on the glasses' feed. For example, it can recognize items or read labels ("tell me what I'm looking at," AI and knowledge transfer) and then immediately relay the info to the worker. This bridges knowledge gaps on the floor with on-demand guidance.

When to use it

• Order picking & packing: Ideal for warehouse pick/pack operations that previously relied on voice-only picking or paper lists. The hybrid solution shines in complex orders where visual confirmation complements audio prompts (e.g. verifying similar-looking products).
• No-hands scenarios: Environments where workers need both hands free (for heavy items or safety) benefit greatly. Smart glasses allow eyes on task while the system whispers directions. In loud or low-light areas where voice-only or screen devices falter, audio plus visual cues keep workers efficient.
• Onboarding & training: New hires can follow intuitive audio prompts and see what to pick without extensive training. The system can even walk them through tasks with images or snapshots of items to grab, reducing onboarding time onboarding halved with vision picking.

Current limitations

• Supported hardware (preview phase): As of now, only certain smart glasses are supported by the SDKs. For example, Meta's Device Access Toolkit (preview) works with Ray-Ban Meta and Oakley Meta glasses (camera + audio), but not yet with every AR headset. Additionally, these first devices have no see-through HUD display, so all feedback is via audio (or the phone screen).
• Permissions & platform constraints: The solution requires working with Bluetooth and camera streams. This means handling permissions: camera/mic access (for capturing imagery and voice commands) and Bluetooth permissions on mobile. iOS apps must include usage descriptions for Bluetooth and microphone, and Android apps need new Bluetooth permissions (e.g. BLUETOOTH_CONNECT, BLUETOOTH_SCAN) due to Android 12+ requirements. Also, continuous streaming from glasses may pause when the app is backgrounded (OS limitations), so design for active use (e.g. keep the app in foreground during picking sessions).
• Preview SDK gaps: Current SDKs are in developer preview, not yet production-ready (developer preview status). For instance, Meta's toolkit allows photos/streaming from glasses, but it doesn't yet expose the glasses' display for custom AR overlays. Advanced inputs like voice/gesture input are also off-limits in this preview. In short – great for prototyping and internal pilots, but not for public app store releases.

2) Prerequisites

Before diving in, ensure you have the necessary accounts, hardware (or simulators), and tools set up.

Access requirements

• Create or sign in to the Meta Wearables Developer Portal (Meta's hub for smart glasses SDK) – available to supported countries.
• Join or create a Wearables developer organization on the portal. This will allow you to manage app projects and test releases.
• Enable preview access if prompted. Agree to any beta program terms for the Wearables Device Access Toolkit (being a developer preview, you must accept Meta's developer terms and use policy).
• Create a Wearable App project in the portal (and a test Release Channel if needed). This will generate an App ID for your mobile app and allow your app to be authorized with the glasses.

Platform setup

iOS

• Xcode 15+ with iOS 15.2 or later SDK system requirements. Your deployment target must be iOS 15.2 or above (the minimum OS that supports the Meta glasses toolkit).
• Swift Package Manager (built into Xcode) or CocoaPods set up, to fetch the Smart Glasses SDK. Internet access is required to pull the package from GitHub.
• A physical iPhone (iOS 15.2+). Simulator is not recommended – the app needs Bluetooth to talk to glasses, which isn’t available on iOS simulators.

Android

• Android Studio (Giraffe or newer) with Android SDK 31+ (Android 12.0+). The app should target Android 13 (API 33) ideally, and support Android 10+ as minimum.
• Gradle (8.+) and Kotlin (1.8+). The toolkit is distributed via Maven; you’ll configure Gradle to fetch the SDK with a GitHub Packages token.
• A physical Android phone (Android 10 or higher). Emulators are not recommended – while you can compile, you’ll need real Bluetooth hardware to connect to the glasses.

Hardware or mock

• Supported smart glasses – e.g. Ray-Ban Meta smart glasses (Gen 2) or Oakley Meta HSTN. These have an 8MP camera and open-ear audio; they pair with the Meta AI mobile app. OR use the Mock Device Kit if you have no hardware Mock Device Kit instructions. The SDK includes a simulator that can emulate a glasses device, letting you test capture/stream workflows on your dev machine without physical glasses.
• Meta AI app installed on your phone and your glasses paired to it. The Meta AI app (available on iOS/Android) acts as a bridge app for smart glasses. Ensure you can connect the glasses to your phone via this app at least once.
• Bluetooth enabled on your mobile device, and awareness of permission prompts. The first time your app tries to connect to glasses, the Meta AI companion will ask for approval. Also, be ready to grant the app mic access if using voice commands.

3) Get Access to the Smart Glasses SDK

To start, you need to get the SDK and credentials from the provider’s developer portal.

1. Go to the Wearables Developer Center: Visit the Meta developers portal for wearables at developer.meta.com/wearables. Sign in with your Meta developer account. If you're new, create an account (it's free) and ensure your country is supported for AI glasses.
2. Request access: If the Device Access Toolkit is listed as a Developer Preview, click to join the preview. This may involve agreeing to preview terms and developer confirmation.
3. Accept terms: Make sure to accept the Meta Wearables Developer Terms and Acceptable Use Policy when prompted developer terms. You must comply with privacy requirements (since the glasses capture user-facing camera and audio).
4. Create a project: In the Wearables Developer Center, create a new Glasses App project. Provide a name (e.g. "WarehousePickerApp"). The portal will assign a unique Application ID for your app app ID assignment details. You might also set up an Organization (if not already) under which this project lives.
5. Download credentials/config: In this preview, there may not be a classic “API key”, but you do need to configure your app with the provided ID and settings:
   • iOS: No separate key file – you will integrate via Swift Package and set the App ID in code or Info.plist if required by the SDK.
   • Android: No JSON config file; instead, you'll add the Application ID meta-data in your AndroidManifest.xml GitHub documentation and supply a GitHub token for fetching the SDK (see next steps).

Done when: you have access to download the SDK (or its GitHub repo), and you’ve noted your App ID (from the portal) for use in your app. You should also see your project listed on the developer portal dashboard, possibly with options to manage releases (for now, you’ll just be testing locally or via a closed channel).

4) Quickstart A — Run the Sample App (iOS)

Goal

Run the official iOS sample app to verify that your glasses can connect and that you can capture a photo/stream using the SDK. This confirms your setup before you integrate into your own app.

Step 1 — Get the sample

• Option 1: Clone the SDK and sample from GitHub. The Meta Wearables SDK for iOS is open-sourced on GitHub at facebook/meta-wearables-dat-ios. Clone the repo, and open the Xcode workspace for the sample (for example, samples/CameraAccess/CameraAccess.xcodeproj included in the repo).
• Option 2: Download a pre-built sample if available from the developer portal. (As of now, the GitHub sample is the main source. The portal’s “Getting Started” might also offer a link to a sample app zip.)

Step 2 — Install dependencies

The sample uses Swift Package Manager to include the Wearables SDK:

• Open the sample project in Xcode. Go to File > Add Packages.... Enter the package URL: https://github.com/facebook/meta-wearables-dat-ios. When prompted, select the latest version (e.g. 0.3.0 preview) and add it to the project's target GitHub documentation.
• Xcode will fetch the package. Ensure that the MetaWearablesDAT frameworks are now visible in the project’s Swift Package dependencies.
• If the sample uses CocoaPods (less likely), run pod install to fetch the SDK. (SPM is recommended by the docs.)

Step 3 — Configure the app

A few settings to update in the sample for your setup:

• App ID: Locate where the sample sets the Wearables Application ID. This might be in the code (initialization call) or an Info.plist entry. Replace it with the App ID from your project in the developer portal so that Meta's system recognizes your app.
• Bundle ID: If you created an App ID on the portal tied to a specific bundle ID, ensure the Xcode project’s bundle identifier matches it. (If the portal didn’t require a fixed bundle ID, you can skip this.)
• Permissions: Open your app’s Info.plist and verify it has usage descriptions for NSBluetoothAlwaysUsageDescription (explain that the app uses Bluetooth to connect to wearable scanners), and NSMicrophoneUsageDescription (if the sample listens to voice). If not, add them to avoid runtime permission issues.
• Entitlements: In Signing & Capabilities, you might need the “Bluetooth LE Access” background mode if continuous connection is needed even in background (optional – most often not needed unless your use-case runs with screen off).

Step 4 — Run

1. Select the sample app scheme (e.g. CameraAccess) in Xcode.
2. Choose a target device – select your iPhone (connected via USB or Wi-Fi).
3. Build and Run the app on the iPhone. Install any provisioning profiles if prompted (for device deployment).
4. The app should launch on the phone. It will likely start in a “disconnected” state waiting for glasses.

Step 5 — Connect to the glasses

• Put on your smart glasses and ensure they are paired to your phone via the Meta AI app. In the Meta AI app, enable Developer Mode for the glasses (there’s usually a toggle in settings to allow third-party apps to connect).
• In the sample app on your phone, tap the Connect button. The first time, it will redirect you to the Meta AI companion to authorize linking your app. Approve the connection (you might see a prompt in the Meta AI app or a notification).
• Once authorized, the sample app will register and connect to the glasses. The status should indicate Connected.

Verify

• The app shows a connected status (no errors). It might display the glasses name or battery level.
• You can initiate a camera stream or capture. For example, tap “Start Stream” if available – you should see live video from the glasses camera on the phone screen (with a slight latency due to Bluetooth). Alternatively, tap a “Capture Photo” button – the glasses should snap a picture and the image appears in the app.
• Audio link works: If the sample has an audio component (like a “Talk” or TTS feature), test that the glasses play the sound. For instance, some samples might allow recording audio from glasses mic or playing a test prompt to the glasses speaker.

Common issues

• Build errors (iOS): If you get an architecture or signing error, ensure you opened the .xcproj/.xcworkspace and not a playground. Also confirm Xcode is using a team/provisioning that allows installing on your device (for personal team, add your Apple ID in Xcode preferences).
• SDK not found: If Xcode complains it cannot find MetaWearablesDAT package, check your internet and that the SPM reference is correct. You might need to add a GitHub auth token in Xcode if rate-limited (in Preferences > Accounts).
• Connection fails: If tapping Connect does nothing or errors:
  • Make sure Developer Mode is enabled in the Meta AI app.
  • Ensure the Meta AI app is running on the phone (it handles the pairing in background).
  • If you see a “Permission Denied” message, check that your App ID is correctly set and that the glasses are listed in your developer portal project (some platforms might require registering the device or serial number for dev mode).
• No video feed: If connected but no video, the Bluetooth bandwidth might be an issue. Remember the stream max is 720p30 over BT UploadVR. Try moving closer to the phone or check if another app is using the connection. Also, iOS might need you to accept a pairing request for "glasses camera" as an accessory – check Bluetooth settings.

5) Quickstart B — Run the Sample App (Android)

Goal

Run the official Android sample app to verify end-to-end functionality with your glasses. You’ll confirm you can connect, stream the camera, and receive images using the Android SDK.

Step 1 — Get the sample

• Clone the repository: The Android SDK and a sample app are available on GitHub at facebook/meta-wearables-dat-android. Clone this repo to your machine. The sample app (often named CameraAccess or similar) can be found under samples/ or app_client/ in the repo.
• Open in Android Studio: Launch Android Studio and Open an existing project, navigating to the cloned repository’s sample app folder. Let Gradle sync the project. (If prompted to upgrade the Gradle plugin, you can accept, but ensure compatibility with the SDK.)

Step 2 — Configure dependencies

The Wearables SDK is pulled from GitHub Packages, which requires authentication:

• Open the project's settings.gradle.kts. Add the GitHub Maven repository as shown in the docs, including credentials. For example GitHub documentation:

  maven {
      url = uri("https://maven.pkg.github.com/facebook/meta-wearables-dat-android")
      credentials {
          username = "" // not needed
          password = System.getenv("GITHUB_TOKEN") ?: localProperties.getProperty("github_token")
      }
  }
  

  This means you need a GitHub personal access token (with read:packages scope). Easiest: create a token on GitHub, then put github_token=<YOUR TOKEN> in your local.properties file.

• Add the SDK dependencies. The sample's Gradle might already include them, but double-check build.gradle.kts for lines adding mwdat-core, mwdat-camera, mwdat-mockdevice. If not present, add the libraries as per docs. For example GitHub documentation:

  # In libs.versions.toml
  mwdat = "0.3.0"
  mwdat-core = { group = "com.meta.wearable", name = "mwdat-core", version.ref = "mwdat" }
  mwdat-camera = { group = "com.meta.wearable", name = "mwdat-camera", version.ref = "mwdat" }
  mwdat-mockdevice = { group = "com.meta.wearable", name = "mwdat-mockdevice", version.ref = "mwdat" }
  

  And in app module dependencies:

  implementation(libs.mwdat.core)
  implementation(libs.mwdat.camera)
  implementation(libs.mwdat.mockdevice)
  

• Hit “Sync Project” in Android Studio. It should download the SDK. If prompted for credentials, ensure your local.properties token is set.

Step 3 — Configure app

Now prepare the app for your specific setup:

• Application ID: Open AndroidManifest.xml. Insert a meta-data tag with your App ID from the portal GitHub documentation:

  <application ...>
      <!-- Required: Your application ID from Wearables Developer Center -->
      <meta-data
          android:name="com.meta.wearable.mwdat.APPLICATION_ID"
          android:value="YOUR_APP_ID_HERE" />
      ...
  </application>
  

  This ensures your app is recognized by the glasses and Meta’s system.

• Package name: Make sure the app’s package name matches any registration you did. If the developer portal tied the App ID to a package (often not strictly required in preview), keep it consistent.

• Required permissions: In AndroidManifest.xml, add:

  • <uses-permission android:name="android.permission.BLUETOOTH_CONNECT" /> (for connecting to the glasses).
  • <uses-permission android:name="android.permission.BLUETOOTH_SCAN" /> (for discovering devices, if needed).
  • <uses-permission android:name="android.permission.RECORD_AUDIO" /> (if your app will use the glasses’ mic or voice commands).
  • <uses-permission android:name="android.permission.CAMERA" /> – not strictly needed for the glasses camera, since it’s external, but if you plan to use the phone camera as fallback or scan QR codes on phone, include it.
  • Also, ensure your app targets SDK 31+ and include the <uses-permission android:name="android.permission.ACCESS_FINE_LOCATION" android:maxSdkVersion="30" /> if supporting older devices (Bluetooth scan on Android 10 required location permission).

• Enable Bluetooth permissions (runtime): If your targetSdk >= 31, you’ll need to request the BLUETOOTH_CONNECT (and SCAN if used) at runtime. The sample likely does this, but verify in code. If not, be prepared to prompt the user on first run.

Step 4 — Run

1. In Android Studio, select the sample app’s run configuration (usually already present).
2. Choose your Android device (connect via USB or use wireless debugging). Ensure Bluetooth is on.
3. Click Run. The app will build and install on the phone. Grant any installation prompts and then launch it.
4. On first launch, the app may ask for permissions (e.g. “Allow to record audio?” for mic, or Bluetooth scan permission). Approve these so it can function.
5. The app UI should appear, likely with a “Connect” button similar to the iOS sample.

Step 5 — Connect to wearable/mock

• If you have the real glasses: Open the Meta AI companion app on the phone and enable Developer Mode for the glasses (if not already). Then, in your sample app, tap Connect. You should see a flow where it registers your app (possibly a quick switch to Meta AI app for approval). After a moment, it should indicate a successful connection.
• If using the Mock Device Kit: The Android SDK includes a mock device simulator. The sample might let you choose a "Mock" mode (perhaps a toggle or it auto-falls back if no real device). Follow the sample's instructions to activate the mock device – it might simply connect to a virtual device that streams a sample video feed or images. This is useful if you have no hardware; you'll still be able to test capture and streaming logic on the phone itself Meta Developer documentation.
• Once connected (real or mock), test the core feature: start a camera stream or capture a photo. The live feed from the glasses (or a placeholder feed in mock mode) should display on screen.

Verify

• Connected status is shown in the app (e.g. a “Connected to Glasses” message or icon). The glasses might also indicate connection via an LED or sound.
• Video stream works: Initiating a stream shows the scene in front of the wearer on the phone screen. Move around to see the video updating. There will be some lag (the SDK caps at 720p/30fps due to Bluetooth bandwidth), but it should be continuous.
• Photo capture works: When you press a capture button, the glasses’ camera shutter should trigger (often you’ll hear a click sound from glasses), and the captured image appears in the app (perhaps as a thumbnail or a new screen). Try pointing the glasses at a barcode or object and capture it.
• Audio link: If applicable, play a test audio prompt from the app. For example, the app might have a “Play Prompt” that uses Android’s AudioManager to route sound to the glasses (which act as a Bluetooth headset). Verify you hear it through the glasses. Similarly, test speaking (if there’s a voice command demo) – your voice should be picked up by the glasses mic and recognized by the app (common demo: ask a question and see if the app transcribes or responds).

Common issues

• Gradle authentication error: If Gradle fails to fetch mwdat-android with a 401 error, it means the GitHub Packages token was not provided or incorrect. Double-check your local.properties for github_token, or set the environment variable before building GitHub documentation.
• Manifest merge conflict: If your project’s manifest already had a <application> meta-data that conflicts, you might see an error. Resolve by merging entries or removing duplicates. Also ensure you didn’t accidentally declare the Application ID meta-data outside the application tag.
• Connection timeout: If the app cannot find the glasses:
  • Make sure the glasses are powered on and paired to the phone via the companion app.
  • The Meta AI app must be running (it usually starts automatically when glasses connect to phone via Bluetooth, but if you killed it, reopen it).
  • Ensure your phone’s Bluetooth is on and the glasses are in range. If the connection still fails, try toggling Bluetooth off/on.
  • If using Mock mode and nothing happens, verify that the sample app’s mock mode is properly enabled (some samples may require shaking the phone or tapping a “Use Mock” in developer settings).

6) Integration Guide — Add Smart Glasses Support to Your Warehouse App (iOS & Android)

Goal

Integrate the Smart Glasses SDK into your existing warehouse app, so that you can trigger an end-to-end picking feature: e.g., capturing a product photo via glasses and playing an audio confirmation to the user. By the end, your app will be “glasses-aware” and ready to deploy in a pilot.

Architecture

Your mobile app will serve as the client connecting to the glasses:

• Mobile App UI – includes buttons (connect, capture, etc.) and status displays.
• Wearables SDK Client – manages the connection to glasses and data streaming (camera frames, etc.).
• Wearable Device (glasses) – captures images/video and sends to phone; receives audio from phone.
• Application logic – your own code that triggers captures or processes images (e.g. barcode decoding or sending to server) and then uses phone’s TTS or prompts for feedback.
• Callbacks/Events – the SDK will provide events like “photo received” or “device disconnected” which your app will handle to update UI or retry actions.

In practice, you’ll add a WearablesClient class to abstract the glasses connection, a FeatureController for specific tasks (e.g. PhotoCaptureController), and hook these into your existing architecture (which might include a Warehouse Management System API).

Step 1 — Install the SDK in your app

iOS

• Using Swift Package Manager, add the package facebook/meta-wearables-dat-ios to your app project (just like in the sample). Choose the latest tag (e.g. 0.3.0). Xcode will embed the needed frameworks.

• Import the module in code where needed, e.g. import MetaWearablesDAT (actual module name might be based on SDK docs).

• If your app uses CocoaPods, alternatively add:

  pod 'MetaWearablesDAT', '~> 0.3.0'
  

  (Check the SDK docs for the correct pod name if available.)

Android

• Add the GitHub Maven repo for com.meta.wearable:mwdat to your buildscript (as shown earlier in Quickstart B) GitHub documentation. Configure the GitHub token for Gradle.

• In your app module build.gradle, add dependencies:

  implementation("com.meta.wearable:mwdat-core:0.3.0")
  implementation("com.meta.wearable:mwdat-camera:0.3.0")
  implementation("com.meta.wearable:mwdat-mockdevice:0.3.0")
  

  (Using the version you got access to. You can also use the TOML centralized versions approach.)

• Sync Gradle to fetch the libraries.

Step 2 — Add required permissions

Ensure your app has the necessary permissions declared and user prompts handled:

iOS (Info.plist)

• NSCameraUsageDescription – Although the glasses have their own camera, include a generic camera usage string (in case you fallback to phone camera or the SDK still expects a description).
• NSMicrophoneUsageDescription – Explain that the microphone (on glasses) is used for voice commands or audio notes.
• NSBluetoothAlwaysUsageDescription – Explain that Bluetooth is used to connect to wearable scanners/glasses. iOS will present this text when your app first accesses BT for the glasses connection.
• (If you want background operation: add UIBackgroundModes with external-accessory or bluetooth-central if you plan on maintaining connections when the app is backgrounded, but use with caution as continuous background use can drain battery.)

Android (AndroidManifest.xml)

• <uses-permission android:name="android.permission.BLUETOOTH_CONNECT" /> (required for connecting to paired BT devices at runtime on Android 12+).
• <uses-permission android:name="android.permission.BLUETOOTH_SCAN" /> (if your logic needs to scan for nearby BT devices – often not needed if connecting via companion).
• <uses-permission android:name="android.permission.RECORD_AUDIO" /> (if voice input from glasses will be used).
• <uses-permission android:name="android.permission.CAMERA" /> (optional, if using phone camera as well).
• If targeting Android 13, also declare <uses-permission android:name="android.permission.POST_NOTIFICATIONS" /> because the companion might use a notification for device status (and your app might need to show a notification if running a foreground service for long-running tasks).
• Note: The Wearables SDK might internally request needed Bluetooth permissions, but it’s good practice you handle it. On Android, call ActivityCompat.requestPermissions for BLUETOOTH_CONNECT when initializing, so the user approves it.

Step 3 — Create a connection manager (thin client wrapper)

Organize your integration by creating helper classes:

• WearablesClient (or GlassesManager): This class uses the SDK to connect/disconnect to the glasses and exposes high-level states (connected, battery, etc.). For example, it will have methods like connect() which under the hood registers your app with the Meta AI service, and disconnect(). It should listen to events from the SDK (like onConnectionFailed, onDeviceDisconnected) and forward those to your app (perhaps via a delegate or LiveData/Flow).
• FeatureService or Controller: Depending on your app’s architecture (MVC, MVVM, etc.), create a controller for the specific feature, e.g. PhotoCaptureController. This will use the WearablesClient to initiate a photo capture. In the Meta SDK, that might be an API call like glassesCamera.capturePhoto(). The controller can also handle receiving the image and then perform your app-specific logic (e.g., decode barcode, send to server, etc.).
• PermissionsService: Implement a utility to check/request permissions at runtime. For example, on app launch or when user tries to connect, ensure Bluetooth and audio permissions are granted. If not, request them and only proceed once granted.

Integrate these into your app’s flow:

• On app start, you might initialize WearablesClient (but not connect yet).
• Provide a UI button “Connect Glasses” that calls wearablesClient.connect().
• Ensure proper tear-down: when app exits or user signs out, call disconnect() to release resources.

Definition of done:

• SDK initialized without errors when app runs. (E.g., on iOS, no crash on startup — if the SDK requires any setup code, it’s called; on Android, maybe initialize in Application class if needed).
• Connect/disconnect lifecycle handled: The app can connect on demand and survive temporary disconnections (e.g., if glasses go out of range and reconnect when back). Implement retries or user prompts on failure (“Please turn on your glasses”).
• User feedback on errors: If connection fails or an action errors (e.g., photo capture fails), the app should show a toast/alert like “Glasses connection lost. Please check Bluetooth.” Also log these events for debugging.

Step 4 — Add a minimal UI for the glasses features

Design a simple interface in your app for the new functionality:

• A “Connect Glasses” button (toggle). When tapped, if not connected, attempt to connect and change to “Disconnect” when connected. Show an icon or status text (🔌 Connected or ❌ Disconnected).
• A status indicator – could be just text or an icon that shows if the glasses battery is low or if the feed is active. (Optional: display the glasses name or battery % if SDK provides it.)
• A “Capture” or “Scan Item” button. When tapped, it will trigger the glasses camera to take a photo (or start a short video stream) of whatever the user is looking at (the item or location label).
• A result display – e.g., an ImageView for the photo thumbnail or a text field for decoded barcode/recognized text. After the capture, show a small thumbnail of what was captured so the user knows it was recorded. If you decode a barcode/QR from the image or get an AI recognition result, display that (e.g., “Item 12345 detected”).
• Optionally, a log or message area for events (useful in testing: e.g., “Connected to glasses”, “Photo received”, “Error: Bluetooth off”).

In terms of integration, add the new UI elements in your existing app screens (maybe a dedicated “Wearable” screen or if it’s specifically for picking, integrate into your picking workflow screen).

7) Feature Recipe — Trigger Photo Capture from Wearable into Your App

To cement the integration, let’s implement a specific use-case step by step: taking a photo with the glasses when the user taps a button, then using it in the app. In a warehouse scenario, this could be used to scan an item or confirm a pick.

Goal

When the user taps “Capture” in the app:

1. If the glasses are not yet connected, prompt to connect (or do so automatically).
2. If permissions (camera/mic/Bluetooth) are not all granted, request them.
3. Send a capture command to the glasses via the SDK.
4. Glasses take a photo (or a few video frames) of whatever the worker is looking at (e.g., a barcode on a package).
5. The phone app receives the image bytes via the SDK callback.
6. The app processes the image (e.g., decodes a barcode or simply saves it as proof).
7. Provide feedback: show the image thumbnail in the UI and play a voice prompt confirming the action (e.g., “Item captured!”).

UX flow

1. Ensure connected: The Capture button should be disabled or say “Connect first” if no device is connected. The user must pair/connect the glasses initially.
2. Tap Capture: User taps the button in app once they have the target item in view.
3. Show progress: The UI might show a spinner or “Capturing…” message while the image is being taken (this might take a second or two).
4. Receive result: When the image comes in, hide the spinner. Show the thumbnail on screen. If the app extracts info (like scanning a barcode from the image), display the decoded result (e.g., “Scanned SKU: 123456”).
5. Display/save confirmation: Possibly save the image to local storage or send to server as needed, and notify the user (“Saved ✅” or an auditory confirmation).

Implementation checklist

• Connected state verified: In the capture button’s click handler, check if (!wearablesClient.isConnected) { showAlert("Please connect your smart glasses first."); return; }. This prevents calls when not ready.
• Permissions verified: If your app might not have gotten mic or storage permissions yet by this point, ensure to check and request. (Bluetooth should have been handled on connect; camera/mic handled on app start.)
• Capture request issued: Use the SDK’s API to take a photo. For example, on Android it might be wearablesClient.getCamera().capturePhoto() and on iOS something like GlassesCamera.shared.capturePhoto(completion:). This will trigger the glasses to snap a picture.
• Timeout & retry: Implement a timeout in case the operation hangs (e.g., if the Bluetooth connection is lost mid-capture). For instance, if no callback in ~5 seconds, show an error and allow retry.
• Result handling: In the SDK callback (e.g., onPhotoReceived(imageData)), promptly handle the data:
  • Convert the image bytes to a bitmap or UIImage.
  • Save it to the app’s storage or a temporary location.
  • Update the UI on the main thread: set the ImageView’s image to this new photo and perhaps animate a “flash” to mimic a camera effect.
  • Optionally, run recognition (if you have a barcode scanner library or ML model, feed it the image here).
• UI feedback: After processing, show a small toast or label “Image saved!” and if appropriate, use text-to-speech to read out a confirmation. (On Android, use TextToSpeech or MediaPlayer with a recorded prompt; on iOS, use AVSpeechSynthesizer or play a sound).

Pseudocode

@IBAction func onCaptureTapped(_ sender: UIButton) {
    guard glassesClient.isConnected else {
        showAlert("Connect your smart glasses first.")
        return
    }
    // Ensure required permissions are granted (pseudo-code)
    if !permissionsService.hasRequiredPermissions() {
        permissionsService.requestPermissions { granted in
            if granted { self.onCaptureTapped(sender) }
        }
        return
    }
    showLoading("Capturing…")
    glassesClient.capturePhoto { result in
        DispatchQueue.main.async {
            self.hideLoading()
            switch result {
            case .success(let photo):
                self.imageView.image = photo
                self.saveToCameraRoll(photo)
                self.showToast("Photo captured and saved ✅")
                self.speak("Image saved")
            case .failure(let error):
                self.showAlert("Capture failed: \(error.localizedDescription)")
                self.log("Capture error: \(error)")
            }
        }
    }
}

And for Android (Kotlin-style pseudocode):

fun onCaptureClicked() {
    if (!glassesManager.isConnected) {
        toast("Please connect glasses first")
        return
    }
    if (!permissionsHelper.hasAllPermissions()) {
        permissionsHelper.requestPermissions(activity) // handle result in callback
        return
    }
    showProgress("Capturing...")
    glassesManager.capturePhoto { result ->
        runOnUiThread {
            hideProgress()
            result.onSuccess { bitmap ->
                imageView.setImageBitmap(bitmap)
                saveImage(bitmap)
                Toast.makeText(context, "Saved ✅", Toast.LENGTH_SHORT).show()
                textToSpeech.speak("Image saved", QUEUE_FLUSH, null, null)
            }
            result.onFailure { ex ->
                AlertDialog.Builder(context).setMessage("Capture failed: ${ex.message}").show()
                Log.e(TAG, "Capture failed", ex)
            }
        }
    }
}

(The above pseudocode assumes the SDK provides a high-level capture API with callback. In reality, you might need to subscribe to a photo event rather than get a direct completion, depending on SDK design.)

Troubleshooting

• Capture returns empty: If the photo result is null or zero bytes, check that the glasses actually took the photo. Sometimes a known issue is missing camera permissions – however, since the glasses camera doesn't use the phone's camera permission, the likely culprits are Bluetooth issues. Check logs: if you see something like "No frames received", the connection might have dropped due to Bluetooth bandwidth issues. Try to reconnect the glasses and capture again. Also ensure the glasses weren't busy (some devices can't handle back-to-back commands too quickly).
• Capture hangs (no response): Implement that timeout! If after X seconds no success or failure callback, abort the request. Call glassesClient.cancelCapture() if available or simply show a “Failed to capture – please retry” message. It could be the glasses were not ready. It’s safe to let the user retry.
• “Instant display” expectation: Users might expect an AR overlay immediately. Since these glasses don’t have a see-through display (in this preview), manage expectations by providing good feedback in the app UI. Show a placeholder or loading indicator during capture so they know something is happening. If they need to keep moving, play an audio like “Got it!” as soon as the photo is captured to let them continue without glancing at the phone.

8) Testing Matrix

When you think you’re done, test the solution under various conditions to ensure robustness:

9) Observability and Logging

When deploying a pilot, it’s crucial to log key events for both debugging and measuring performance. Instrument your app with logs or analytics for:

• Connection events: Log "connect_start", "connect_success" (with time taken), and "connect_fail" (with error details). This helps identify if connections are flaky or taking too long.
• Permission states: Log whether the user granted all required permissions or if any were missing. For example, "permission_bluetooth_granted": true/false on first launch.
• Capture actions: Log events like "photo_capture_start", "photo_capture_success" or "photo_capture_fail". If you send images to a server, log when that upload starts and ends too.
• Performance metrics: You can timestamp and measure latency, e.g., add a log for "photo_roundtrip_ms" – time from user tapping capture to image received. Similarly, measure how long audio prompts take to play after an event.
• Error tracking: If using a crash reporting tool, catch exceptions from the SDK or your integration and report them (with context). E.g., if capturePhoto throws an exception, log the exception message (common ones might be “device not connected” or “bluetooth error”).
• Reconnect counts: If you implement auto-reconnect, log each attempt ("reconnect_attempt" and result). This will show if the glasses are frequently disconnecting (perhaps a sign of Bluetooth interference).
• User actions: Even though the focus is on the wearable, log normal app actions in context – e.g., "pick_confirmed_by_voice": true/false if the user used a voice command to confirm a pick. These data points will help demonstrate the benefit of the solution (e.g., “X% of picks were confirmed via glasses”).

By monitoring these logs, you can gather insights like “Audio prompt acknowledged in 2 seconds on average” or “Reconnection succeeded 90% of the time without user intervention”, which are valuable for convincing stakeholders of the solution’s viability.

10) FAQ

• Q: Do I need the actual hardware to start developing?
  A: Not immediately. You can begin with the provided Mock Device Kit to simulate glasses. This lets you write and test integration code on your laptop or phone without the glasses. However, before final deployment, you'll want to test on real smart glasses to tune the user experience (especially the comfort of audio prompts and camera framing).
• Q: Which smart glasses are supported by this solution?
  A: Currently, the toolkit supports Meta's AI Glasses lineup – specifically Ray-Ban Meta and Oakley Meta HSTN models (both have cameras) supported devices. Support for devices with displays (like Ray-Ban Meta Display edition) is coming, but in preview you can only access their camera, not draw AR overlays display support. Other brands (Vuzix, RealWear, etc.) are not supported by Meta's SDK. Those have their own SDKs, but the general approach in this guide would be similar.
• Q: Can I deploy this feature to production (e.g. on App Store/Play Store) now?
  A: Not yet for broad distribution. The current SDK is in developer preview – meant for testing and prototyping. Meta does not allow shipping apps using this SDK to general consumers yet (developer preview status). You can certainly use it in a closed pilot within your company. Keep an eye on Meta's announcements; full production support will likely come once the glasses themselves are widely available and the SDK graduates from preview.
• Q: Can I push information or AR overlays to the glasses (like to guide the user visually)?
  A: At the moment, no custom visual output to the glasses is available in the SDK display support. The first iterations focus on camera and audio capabilities. That's why this guide emphasizes audio prompts for feedback. In the future, as the glasses with displays open up, you might be able to send text or images to show on the lens. For now, use the phone screen or audio as the output mediums for any guidance.
• Q: Can I use voice commands through the glasses?
  A: Partially. The glasses have microphones which you can access through the phone (the glasses act like a Bluetooth microphone). This means you can record audio or run speech-to-text in your app to implement voice commands (e.g., saying "Next Item" to trigger the next pick instruction). However, you cannot use the glasses' built-in wake word ("Hey Meta") or their onboard assistant for custom commands via the SDK voice/gesture input. Any voice control will be through your own app's voice processing. Make sure to give clear instructions to users and handle noisy environments (the glasses do have noise reduction mics, but always test how accurate recognition is in a bustling warehouse).

11) SEO Title Options

• “How to Get Access to Smart Glasses SDK and Run a Warehouse Picking Demo (iOS/Android)” – covering dev portal sign-up and trying the sample app.
• “Integrate Smart Glasses into Your Warehouse App: A Step-by-Step Guide for iOS & Android” – focusing on adding the glasses capture + audio prompt feature into an existing app.
• “Trigger Photo Capture from AR Glasses in Your App (Hands-Free Scanning Tutorial)” – highlighting the photo capture use case with glasses and phone.
• “Smart Glasses Troubleshooting for Warehouse Apps – Pairing, Permissions, and Common Errors” – a problem-solution styled piece targeting those who hit snags implementing this.

12) Changelog

• 2025-12-25 — Verified on Meta Wearables DAT SDK v0.3.0 (Developer Preview). Tested on iOS 17.2 (Xcode 15.1) and Android 13 (API 33) with Ray-Ban Meta smart glasses (2nd Gen) and the Mock Device Kit. Updated instructions for new Bluetooth permission model on Android and added Meta AI app steps for device pairing.