How to Auto-Cut POV Footage into B-Roll Clips for TikTok/Reels with Creator Mode on iOS & Android

Step-by-step: get access, run the sample app, and integrate Creator Mode into an existing mobile app to automatically generate B-roll highlights from POV camera footage.

TL;DR

  • you’ll make? A basic mobile app functionality to work with smart glasses and generate automatic small clips of B-roll video from first person (POV) to be used for TikTok or Reels sharing. What
  • you’ll do: Access to Wearables SDK Install SDK in Xcode/Android Studio Run sample app in iOS and Android Implement Creator Mode SDK in your app Test it with wearable device or mock simulator.
  • You’ll need: a Meta developer account (Wearables Toolkit preview access), a pair of Ray-Ban Meta smart glasses (or SDK mock device), and a development environment (Xcode 15+ for iOS, Android Studio for Android).

1) What is Creator Mode?

What it enables

  • Hands-free POV content capture: Leverage smart glasses' 12MP camera and mics to film from a first-person perspective without holding a camera with hands-free capture capabilities. This provides a natural "eyesight" view for your videos.
  • Automatic highlight detection: The SDK can identify interesting moments in long, unstructured first-person videos and extract the best segments as highlights – saving you from manual editing.
  • Instant B-roll generation: It quickly cuts raw POV footage into short, shareable B-roll clips (vertical format, with smooth transitions) that are ideal for TikTok and Instagram Reels, speeding up your content creation workflow.

When to use it

  • Vlogging & daily life capture: Record your day hands-free using glasses and automatically get the most engaging snippets to post on social media.
  • Action sports and travel: For activities like hiking, biking, or sightseeing, capture hours of POV video and let Creator Mode auto-pick the scenic or exciting moments as B-roll.
  • IRL streaming & UGC creation: Streamers and UGC creators can use it to generate extra cutaway shots or behind-the-scenes clips from a first-person view, enriching their videos with minimal effort.

Current limitations

  • Device availability: Currently supports Meta's Ray-Ban AI glasses (developer preview); no broad device support yet. If you don't have the glasses, you'll rely on the provided mock device simulator in the SDK.
  • Developer preview only: The SDK is in early access – apps can be tested internally, but public app store releases are restricted until general availability in 2026. This is a prototype/MVP feature.
  • Processing constraints: Automatic highlight detection is still basic in this MVP. It may miss some key moments or include shaky footage. It doesn't (yet) add music or captions – you'll need to use editing tools (like CapCut's AutoCut) for that final polish.
  • Permissions & platform: Continuous video capture can be battery-intensive. iOS apps cannot access the glasses when completely in background (so keep the app active during capture). Voice-command control of the glasses isn't available in this version with no voice/AI integration yet.

2) Prerequisites

Access requirements

  • Meta Developer Account: Create or log in to the Meta Wearables Developer Portal (the Ray-Ban Meta glasses dev center).
  • Join the Preview Program: Request access to the Wearables Device Access Toolkit preview. You may need to agree to preview terms since this feature is in beta.
  • Organization setup: If working with a team, set up or join a Developer Organization/Team on the portal. Ensure all members have the necessary permissions.
  • Enable Wearables project: Create a new Wearables Project/App ID in the developer console. Select the Wearables Device Access (glasses) integration and note the generated App ID or credentials. (This ties your mobile app to the glasses SDK.)

Platform setup

iOS

  • Xcode 15+ with iOS 16.0 or later SDK (iOS 17 recommended).
  • Swift Package Manager (built-in to Xcode) or CocoaPods (if available) for adding SDK.
  • Physical iPhone on iOS 16+ (needed for bluetooth camera access). Simulator isn't supported for wearables (bluetooth).

Android

  • Android Studio Flamingo/Giraffe or higher with Android SDK 33+ (Android 13.0+ or higher).
  • Gradle (7.0+ preferred) Kotlin (1.8+ or higher) for project.
  • Physical Android phone (Android 12 or higher). Usually can't pair with bluetooth wearables in emulators, so physical phone preferred.

Hardware or mock device

  • Ray-Ban Meta Smart Glasses (which is the supported wearable device that comes with the POV camera and mics (developer preview edition))
  • OR Mock Device: If you do not own the glasses, use the mock device simulator (for development purposes) that is included with the SDK. It will simulate camera data, and sensor data.
  • Bluetooth enabled: Ensure your phone's Bluetooth is on. Understand the platform's Bluetooth permission prompts (on iOS, the app needs Bluetooth permission; on Android, the app needs BLUETOOTH_CONNECT permission to communicate with the glasses).

3) Get Access to Creator Mode

  1. Go to the Developer Portal: Visit the Meta Wearables Developer Center at wearables.developer.meta.com and sign in with your Meta developer credentials.
  2. Request Preview Access: Find the Device Access Toolkit section and opt into the developer preview. This may involve filling out an application or simply enabling the feature in your account. (Only approved developers or partners can use the glasses SDK at this stage.)
  3. Accept Terms: Review any beta agreements or Wearables Developer Terms and accept them. You must agree to usage policies (e.g. data collection rules) before proceeding.
  4. Create a Project: In the portal, create a new Wearables project for your app:
    • Give it a name (e.g. "CreatorModeDemo").
    • Register your app's Bundle ID (iOS) or Application ID (Android) if prompted – this should match your actual app's identifier.
    • Add any team members to the project if needed.
  5. Obtain Credentials: After project setup, download or note any credentials:
    • iOS: It could be a config file (think GoogleService-Info.plist etc), or an App ID/Key that you need to include in your Xcode project. Store any.plist or keys provided.
    • Android: It could be an API key, Client Token... Typically you'll want to add it into your gradle.properties or string resource. (Also, you should have a github pat for getting the SDK packages. See next section)
    • If there are no explicit keys provided, then your App ID and developer account association will be used under the hood after integrating the SDK. When

done: You should have something that shows access (e.g. App ID or project ID in the portal) You should see your new project in the portal. You should be able to see a dashboard or settings for your Wearables project, confirming you're enrolled in the preview.


4) Quickstart A — Run the Sample App (iOS)

Goal

Try running the official iOS sample app to make sure you can get connected to the glasses and use the camera. This will make sure that Creator Mode (taking a photo, streaming video, etc) will work correctly with an actual (or simulated) device.

Step 1 — Get the sample

  • Option 1: Clone the repo. Clone the Meta Wearables iOS SDK repo

    git clone <https://github.com/facebook/meta-wearables-dat-ios.git>
    
    

Open Xcode project in the repository: samples/CameraAccess.

  • Option 2: Download as ZIP. Download repository as zip from GitHub Unzip it. Open in Xcode samples/CameraAccess/CameraAccess.xcodeproj There is very little code in

the sample CameraAccess app. It will find the glasses, connect and display camera output.

Step 2 — Install dependencies

The sample uses Swift Package Manager (SPM) to include the Wearables SDK:

  • In Xcode, open the Project Navigator and select the project. Check that the Package Dependencies section already references meta-wearables-dat-ios. If not, add it:
    1. Go to File > Add Packages...
    2. Enter the package URL: https://github.com/facebook/meta-wearables-dat-ios
    3. Choose the latest version (e.g. 0.x.y preview).
    4. Add the package to the sample app target.
  • If the package requires authentication (it shouldn't, as it's public), ensure you have a GitHub account set in Xcode preferences. (CocoaPods is not needed since SPM is used. There is no Podspec as of this preview.)

After adding, Xcode will fetch the SDK. You should see MetaWearablesDAT (or similarly named) frameworks in the project.

Step 3 — Configure app

Before running, adjust a few settings:

  • Bundle ID: Change the sample app's bundle identifier to your registered one (from step 3). In Xcode, select the project target, go to Signing & Capabilities, and set Bundle Identifier to the value you set in the dev portal (e.g. com.yourname.CreatorModeDemo). This ensures the glasses recognize your app as authorized.
  • Add config file: If the portal provided an iOS config .plist (for API keys or project info), add it to the Xcode project (usually by dragging into the project and ensuring it's included in the app target). For example, a WearablesConfig.plist might contain your App ID or OAuth credentials.
  • Capabilities/Entitlements: Enable required capabilities:
    • In Signing & Capabilities, click "+ Capability" and add Background Modes → enable "Uses Bluetooth LE Accessories" (if your app will maintain BLE connections even in background).
    • If provided, add any custom entitlement file from Meta (not common for this SDK).
  • Info.plist permissions: Add usage descriptions for any system permissions the app will request:
    • NSBluetoothAlwaysUsageDescription – "Allow connection to wearable camera devices via Bluetooth."
    • NSCameraUsageDescription – "Allow app to use camera." (Not strictly needed for external camera, but include if you plan to use phone camera too.)
    • NSMicrophoneUsageDescription – "Allow app to use microphone for audio from glasses."

Step 4 — Run

  1. Build Settings: Select the CameraAccess sample app scheme in Xcode. Connect your iPhone via USB or use wireless debugging.
  2. Run on device: Choose your iPhone as the run target (the sample cannot run on Simulator due to Bluetooth). Click Build & Run ▶️.
  3. Allow permissions: On first launch, iOS will ask for Bluetooth permission (and microphone, if used). Grant these so the app can find and communicate with the glasses.

The app should launch on your iPhone.

Step 5 — Connect to wearable/mock

  • Power on the glasses: Make sure your Ray-Ban Meta glasses are powered and in range. If they haven't been paired before, put them in pairing mode (usually by pressing the power or capture button for a few seconds until an LED indicates pairing).
  • Connect via app: The sample app will likely automatically scan for the device. If you see a device list, pick your glasses model. If not, it might just connect to the first glasses
  • Grant any prompts: If iOS prompts for Bluetooth pairing or confirmation to connect to the device, accept it. Also ensure the glasses are not already connected to another app (only one app at a time can usually hold the connection).

Once connected, the sample app's UI should indicate a successful connection (e.g. a "Connected" status or the camera viewfinder might appear streaming the glasses' view).

Verify:

  • the app like 'Connected' (e.g. status label or icon turns green, etc). You might see something on the glasses themselves indicating they're connected (LED, sound,
  • etc). Do something with the camera from the app. Like tap on a 'Capture Photo' button. The glasses should take a photo and you should see that photo in the app. Or if the sample has a live preview, you see real-time video from the glasses on your phone screen.

Common issues

  • Build Error (Module not found/codesign issues): This issue may occur due to a problem with the resolution of the Swift package or incorrect signing. Make sure to include Meta Wearables package and try Product Clean Build Folder, then rebuild the project. Also, make sure to use the correct Development Team and iPhone provisioning profile (Xcode should help with this if you have Apple ID added).
  • No device: If the app cannot find your glasses, make sure that Bluetooth is turned on and the glasses are in pairing mode. You may need to pair your glasses with Bluetooth settings on your phone first. Make sure you keep your glasses close to the phone. If using the mock device ensure you included the mock device module and that the sample app has an option to simulate a device (some samples include a toggle to use a virtual device).
  • Permission denied: If the app isn't connecting, check Settings > Privacy on the iPhone:
    • Bluetooth permission must be enabled for your app. If you accidentally denied it, re-enable it.
    • The sample might not explicitly request microphone if it doesn't use audio, but if it does, ensure microphone permission is granted.
    • Restart the app after changing permissions.

5) Quickstart B — Run the Sample App (Android)

Goal

Run the official Android sample app to verify the glasses' integration on Android. You will confirm that you can connect to the wearable and use its camera from an Android app.

Step 1 — Get the sample

  • Clone the repository:

    git clone <https://github.com/facebook/meta-wearables-dat-android.git>
    
    

    Open the project in Android Studio, and locate the sample app under samples/CameraAccess. (If Android Studio prompts to import Gradle project, proceed.)

  • (Alternatively, download the repo as a ZIP from GitHub and open it.)

The sample CameraAccess app (Android) contains minimal code to connect to the glasses and stream from the camera.

Step 2 — Configure dependencies

The Android SDK is distributed via GitHub Packages, so you'll need to add the Maven repository and authentication token:

  • Add Maven repository: In the project's settings.gradle or settings.gradle.kts, add the Meta GitHub Maven URL. For example, in Kotlin DSL:

    dependencyResolutionManagement {
        repositories {
            maven {
                url = uri("<https://maven.pkg.github.com/facebook/meta-wearables-dat-android>")
                credentials {
                    username = "" // not needed
                    password = System.getenv("GITHUB_TOKEN") ?: localProperties.getProperty("github_token")
                }
            }
            // ... other repos ...
        }
    }
    
    

    This snippet (from Meta's docs) configures Gradle to use the GitHub token for access.

  • Provide GitHub token: Generate a GitHub Personal Access Token (classic) with at least read:packages scope. In your ~/.gradle/gradle.properties or the project's local.properties, add:

    github_token = YOUR_GITHUB_PAT_HERE
    
    

    Gradle will use this token to authenticate when downloading the Wearables SDK artifacts.

  • Sync Gradle: After adding the repository, click "Sync Project with Gradle Files" in Android Studio. Gradle should fetch the Wearables Toolkit libraries. (Ensure internet access and that the token is correct – if not, you'll see errors in the Sync console.)

Step 3 — Configure app

  • Application ID: Open the app module's build.gradle. Set the applicationId to the one you registered in the portal (e.g. com.yourname.creatormodedemo). This should match the App ID from step 3 to satisfy any backend checks.

  • Add SDK dependencies: The Meta SDK is modular. In the sample's Gradle config, you might see something like:

    dependencies {
        implementation("com.meta.wearable:mwdat-core:0.3.0")
        implementation("com.meta.wearable:mwdat-camera:0.3.0")
        implementation("com.meta.wearable:mwdat-mockdevice:0.3.0")
    }
    
    

    If not present, add the necessary modules:

    • mwdat-core (core connectivity and data)
    • mwdat-camera (camera control and streaming)
    • mwdat-mockdevice (for simulating a device if needed) Use the latest version (e.g., 0.3.0 as of Dec 2025).
  • AndroidManifest permissions: Ensure the app has required permissions in AndroidManifest.xml:

    • <uses-permission android:name="android.permission.BLUETOOTH_CONNECT" /> (for Bluetooth communication, Android 12+).
    • <uses-permission android:name="android.permission.BLUETOOTH_SCAN" /> (for finding devices, Android 12+; requires <uses-permission android:name="android.permission.ACCESS_FINE_LOCATION" /> as well due to scanning).
    • <uses-permission android:name="android.permission.RECORD_AUDIO" /> (if you will capture audio from the glasses' mics).
    • (Optional) <uses-permission android:name="android.permission.CAMERA" /> – not needed for the wearable's camera, but include if your app also uses the phone camera.
    • <uses-feature android:name="android.hardware.bluetooth_le" android:required="true"/> for BLE support.
  • File handling: If you're storing videos/clips to storage, you'll need to add the necessary storage permission WRITE_EXTERNAL_STORAGE or use scoped storage APIs. (For API 33+ you may not need it if you're using the MediaStore).

Rebuild your project to make sure that everything compiles and your SDK imports are picked up in the IDE.

Step 4 — Run

  1. Connect device. Attach your Android phone (USB debugging enabled) or make sure its connected with ADB on WiFi. Choose it as your target device on Android Studio.
  2. Run app. Click Run The app will be built and installed on the phone. Make sure to notice if you're getting any runtime permissions requested when you start the app.
  3. Permissions. You'll most likely be asked for Bluetooth permissions when you first run the app on Android. For Android 13+, it will specifically prompt "Allow this app to find, connect to, and determine position of nearby devices?" choose Allow If prompted for microphone (for audio) or storage (if writing files), allow those as well.

The sample app should open on your Android device now.

Step 5 — Connect to wearable/mock

  • Pair or discover the glasses: If your Ray-Ban glasses are already Bluetooth-paired with the phone (through the companion app or OS settings), the sample might detect them. Otherwise, ensure the glasses are in pairing/discoverable mode.
  • In-app connection: Use the sample's UI to connect:
    • Some samples auto-connect to the first available wearable. Others provide a Scan/Connect button. Tap it to start scanning for nearby devices.
    • When you see your glasses (e.g. "Ray-Ban AI Glasses") in the list, tap to connect. You might get a system pairing request – confirm pairing by comparing PIN if shown.
  • Grant permissions: If you hadn't already, the app might request BLUETOOTH_SCAN permission when scanning (on Android 12+ this triggers a location permission dialog). Grant it to allow finding the device.
  • Mock mode (if no device): If using the mock simulator, the sample app might have a developer setting or automatically use a fake device when no real one is found. In this case, it will simulate a connection and feed dummy camera data (like a test pattern or sample video).

Once connected, the app should indicate success (e.g., "Connected to device"). The glasses' camera may start streaming to the app if the sample is coded to do so, or there may be a button to start the camera.

Verify:

  • The app UI shows Connected status to the wearable. You might see device info or an LED on the glasses confirming connection.
  • The key feature works end-to-end. For example, if you press "Start Preview" in the app, you begin seeing the POV camera feed from the glasses on-screen. Or if you tap "Capture Photo", a photo is taken and displayed. The round-trip (command to glasses → action → result back to app) should function.

Common issues

  • Gradle authentication error: If the build failed to download the SDK (Could not resolve com.meta.wearable:...), it means the Maven repo access failed. Double-check your GitHub token configuration. Make sure the token is valid and set in local.properties (github_token) or as GITHUB_TOKEN env var, and that the maven { url "<https://maven.pkg.github.com/>..." } block is in the top-level settings.gradle(.kts). Re-sync after fixing.
  • Manifest merger conflict: If your app's manifest had a uses-sdk or other entries that conflict with the SDK's, you might see a merger error. Usually, the sample is configured correctly. Resolve by adjusting manifest or Gradle settings (e.g., if prompted to add tools:remove for a duplicated attribute).
  • Device connection timeout: If the app can't find or connect to the glasses:
    • Ensure the glasses are not connected to another app (e.g., the official Facebook View app or another phone).
    • Restart Bluetooth on the phone and try again.
    • If pairing for the first time, go to Bluetooth settings on the phone and pair the glasses there, then return to the app.
    • Use the mock device as a fallback to continue development until you can use the hardware.
  • Permissions (Android): If you mistakenly denied a permission, the feature will fail silently or throw an error (check Logcat). Go to Settings > Apps > [Your App] > Permissions and enable the required ones, then retry. (For BLE scan on Android 13, you might also need to enable Location if it was off, as BLE scanning is tied to location services.)

6) Integration Guide — Add Creator Mode to an Existing Mobile App

Goal

Now that you've tried the samples, let's integrate the Creator Mode SDK into your own app. We'll see how to add the SDK, connect to the glasses, and do one of the end-to-end features (auto-cut highlight clip) So at the end, your app will be able to communicate with the wearable and do basic POV capture stuff.

Architecture

think about it as a layered integration

  • UI Layer (Your App): Buttons and screens to allow a user to connect their glasses and ask for a capture.
  • Client Layer (SDK wrapper): You have a WearablesClient in your app that will handle the connection to the glasses. (Connect, Disconnect, check status, etc)
  • Feature Layer: Services/Controllers that do specific actions. Let's say you have a CaptureService that will initiate a video capture and process it or a PhotoService that does still shots.
  • Device SDK (Meta Wearables SDK): The SDK provided by Meta that actually interfaces with the glasses hardware (handles Bluetooth communication, camera control, etc.), providing callbacks/events for your app.
  • Wearable Device: The Ray-Ban glasses themselves, which capture the video/audio and send data to the phone.

Flow of events: App UI → calls into WearablesClient (SDK wrapper) → glasses hardware performs action → results (video, photo, status) come back via SDK callbacks → Your App updates UI or saves data accordingly.

Step 1 — Install the SDK

Add the Creator Mode (Wearables Toolkit) SDK to your project if you haven't already:

iOS (Swift):

  • Using Swift Package Manager, add the dependency as in Quickstart A. In your existing Xcode project, do File > Add Packages... and enter facebook/meta-wearables-dat-ios repo URL. Choose the latest tag and add it to your app target.
  • If you prefer CocoaPods (and if Meta provides one in the future), add to your Podfile:pod 'MetaWearablesDAT', '~> 0.3.0' (hypothetical example; as of now, use SPM).
  • After integration, import the relevant modules in your code: e.g. import MetaWearablesDAT (and submodules like MetaWearablesCamera if applicable).

Android (Kotlin/Java):

  • In your app's Gradle scripts, add the Maven repo and dependencies as described in Quickstart B:

    • In settings.gradle: include the maven.pkg.github.com/facebook/meta-wearables-dat-android repository and GitHub token credentials.

    • In app/build.gradle: add dependencies:

      implementation "com.meta.wearable:mwdat-core:0.3.0"
      implementation "com.meta.wearable:mwdat-camera:0.3.0"
      implementation "com.meta.wearable:mwdat-mockdevice:0.3.0" // remove if only using real device
      
      
    • Sync the project to download the libraries.

  • Import classes in your code as needed, e.g. import com.meta.wearable.mwdat.Camera (actual package names may vary; check SDK docs).

Step 2 — Add necessary permissions

Ensure your app's permission setup covers the wearable use case:

iOS (Info.plist):

  • Add NSBluetoothAlwaysUsageDescription with a user-facing message, since the app needs Bluetooth to communicate with the glasses.
  • If your app will record audio from the glasses, add NSMicrophoneUsageDescription.
  • (Camera usage may not be needed for external camera, but adding NSCameraUsageDescription doesn't hurt for completeness.)
  • Optionally, add a background mode capability in the project if you want the connection to persist while the app is backgrounded (this is advanced and not always allowed; for initial integration, you can skip background support).

Android (AndroidManifest.xml):

  • Include the Bluetooth permissions:

    <uses-permission android:name="android.permission.BLUETOOTH_CONNECT" />
    <uses-permission android:name="android.permission.BLUETOOTH_SCAN" />
    <uses-permission android:name="android.permission.BLUETOOTH" /> <!-- for older devices -->
    
    

    If using BLUETOOTH_SCAN, also include location permission as mentioned.

  • Include microphone permission if capturing audio:

    <uses-permission android:name="android.permission.RECORD_AUDIO" />
    
    
  • (No need for android.permission.CAMERA specifically for glasses, but include if your app also accesses phone camera.)

  • If you intend to keep the connection in background, you might need to request android.permission.ACCESS_BACKGROUND_LOCATION (for BLE scanning in background) and handle foreground service for long-running connections. This is beyond the basic integration (consider for advanced use).

Step 3 — Create a thin client wrapper

To keep things organized, implement small classes to encapsulate the SDK's functionality:

  • WearablesClient (Connection Manager): This class will:
    • Initialize the SDK (if required) when your app starts.
    • Scan and connect to the glasses, or attach to an already paired device.
    • Expose connection state (connected/disconnected) to the rest of your app.
    • Handle reconnection logic (e.g., try to reconnect if connection drops).
    • for instance methods: connect(deviceId) disconnect() property: isConnected
  • CaptureService/FeatureService can contain a high level logic for capture and processing,
    • for instance a BRollCaptureService which gets triggered. This in turn would send a command to glasses to start recording video. Then grab the video data. And invoke auto-cut algorithm to get highlight video. For instance,
    • It might use the SDK's camera API (e.g. start video stream or send a "record" command) and listen for completion or frames.
    • It should also handle timeouts or errors during capture.
  • PermissionsService: especially for android. a small helper class to check and ask runtime permission(Bluetooth, audio etc) before doing actions. iOS may not require a separate class but you need to call that CBCentralManager authorization for bluetooth(if needed).

Have this as separate components for a cleaner/maintainable code.

Definition of done:

  • The Wearables SDK is initialized (for instance, some SDKs require a context or config to be set up – check if Meta's needs initialization or if using any singleton from it).
  • You can call a method like wearablesClient.connect() and it successfully connects to the glasses, updating a state (and UI) to "connected". You handle the full lifecycle: connect on demand, disconnect on app exit or when user toggles off, and attempt to reconnect if needed (e.g. device goes out of range and comes back).
  • Errors (e.g. "device not found" or "connection lost") are caught and either shown to the user or logged. The app should not crash on any exceptions coming from the SDK – handle them gracefully (use try/catch or SDK callbacks).

Step 4 — Add a minimal UI screen

Design a simple interface in your app for the Creator Mode feature. It could be as basic as a single screen with a few controls:

  • "Connect Glasses" button: Initiates scanning/connecting using your WearablesClient. It can change to "Disconnect" when already connected.
  • Connection status indicator: Show an icon or text ("Connected ✅ / Not Connected ❌") so the user knows if the wearable is linked.
  • "Capture Highlight" button: When pressed, this triggers the process to capture a B-roll highlight clip from the glasses. (We will implement this in the next section.)
  • Result preview: A small video player or image view to display the output. For example, after capturing, show a thumbnail or the actual video clip so the user can preview what was recorded/cut. Alternatively, a text log area could confirm actions (e.g. "Saved clip to gallery").

Keep the UI minimal for now – just enough to test the integration. You can always expand it later with more bells and whistles (like a gallery of multiple clips, or controls for different capture modes).


7) Feature Recipe — Automatically Capture a B-Roll Highlight Clip from the Wearable

Goal

So for your core user story: User clicks on a "Capture Highlight" button in your app glasses capture short video from user POV your app gets this video, automatically finds most interesting part (highlight) and extracts it short video clip is shown/saved and ready for use as B-roll.

The magic of Creator Mode is that the raw first-person video gets converted into a usable snippet (with one click).

UX flow

  1. Pre-check If glasses are not connected, prompt user to connect before (don't try to capture when not connected).
  2. User press "Capture Highlight". UI shows the capture is being performed (eg. "Recording..." or spinner)
  3. Recording phase: The app instructs the glasses to record a video (for example, a 10-second clip). Maybe even a progress bar if it's a fixed duration video.
  4. During the process: After raw video is sent into app, perform the processing to identify the best moment. During this brief moment of analysis you could show UI "Processing..."
  5. Result: After highlight is ready, show it! Thumbnail or autostart clip in small video view. Also confirm that you have saved it. For example "Clip saved to Photos" or share button to send to TikTok.
  6. After capture: Reset the UI state (hide spinners and un-grey buttons etc.) so the user can capture a new one, or disconnect if necessary.

Implementation checklist

  • Connected state verified: In the onCaptureHighlight handler, first check if !isConnected (for iOS, or for Android, if your WearablesClient says not connected). If not, maybe open the connection dialog or show "Please connect your wearable first." This avoids false starts.
  • Permissions verified: Ensure the app has the needed permissions at the moment of capture. E.g., on Android check BluetoothAdapter.isEnabled() and for any needed runtime permissions (audio recording if capturing audio). If something is missing, request it and do not proceed with capture until granted.
  • Capture request issued: Use the SDK's camera API to start a capture. Depending on the SDK:
    • If there's a convenience method like GlassesCamera.captureVideo(durationSeconds: Int), call that.
    • Or you might have to start a stream and manually stop it after X seconds. In that case, start recording and use a timer to stop after the desired duration.
    • The SDK might also allow grabbing a continuous stream of frames; for simplicity, use whatever highest-level method is available for recording a clip.
  • Timeout & retry handled: Set a reasonable timeout for the operation. If no response from device in, say, 5–10 seconds, assume something went wrong. Stop any recording attempt and show an error ("Capture failed, please try again"). Also handle if the device disconnects mid-capture (the SDK should throw an error or notify; you then reconnect and prompt user to retry).
  • Highlight extraction: Once you have the video data (file or buffer), run the "auto-cut" logic:
    • In this MVP, this could be a simple heuristic (for example, choose the middle 3 seconds of the 10s clip assuming that's when the camera was steady).
    • Or integrate a highlight detection algorithm: e.g., use an AI model or an open-source library to score frames and pick the highest-scoring segment. For now, a simple approach is fine.
    • Trim the video to the highlight segment (using AVFoundation on iOS or Android's MediaCodec/FFmpeg).
  • Persist & update UI: Save the highlight clip:
    • On iOS, write it to the Photos library or app Documents directory.
    • On Android, save to MediaStore (so it appears in Gallery).
    • Then update the UI: show a thumbnail or play the video. When the save is successful (Saved message as in our pseudocode below). Or
    • if save fails due to storage space not available etc. Let user know where it is saved or if they need to grant permission to store the data.

Pseudocode

Here's a simplified pseudocode (mixup of concepts, just for illustration

func onCaptureHighlightTapped() {
    guard wearablesClient.isConnected else {
        showAlert("Connect your glasses first!")
        return
    }
    if !permissionsService.allPermissionsGranted() {
        permissionsService.requestNeededPermissions()  // asynchronous
        return
    }

    showStatus("Capturing…")
    do {
       // Start recording a 10-second video
       let videoData = try wearablesClient.camera.captureVideo(duration: 10)
       showStatus("Processing…")
       // Simple highlight: take middle portion
       let highlightClip = VideoUtils.extractBestSegment(from: videoData)
       VideoUtils.saveToGallery(highlightClip)
       thumbnailView.image = VideoUtils.generateThumbnail(highlightClip)
       showStatus("Saved ✅")
    } catch {
       showStatus("Capture failed 😢")
       log("Error during capture: \\(error)")
    }
}

purposes): Android (kotlin style)

fun onCaptureHighlightClicked() {
    if (!wearablesClient.isConnected) {
        Toast.makeText(this, "Please connect your glasses first.", Toast.LENGTH_SHORT).show()
        return
    }
    if (!permissionsService.hasAllPermissions()) {
        permissionsService.requestPermissions(this)
        return
    }

    statusText.text = "Capturing…"
    wearablesClient.camera.captureVideo(duration = 10,
        onSuccess = { videoFile ->
            statusText.text = "Processing…"
            val highlightFile = VideoUtils.extractBestSegment(videoFile)
            VideoUtils.saveToGallery(context, highlightFile)
            runOnUiThread {
                thumbnailImage.setVideoURI(highlightFile.toUri())
                statusText.text = "Saved ✅"
            }
        },
        onFailure = { error ->
            statusText.text = "Capture failed 😢"
            Log.e(TAG, "Capture failed", error)
        }
    )
}

(Note that I'm assuming synchronous capture here for simplicity. Realistically, I expect the SDK should expose asynchronous callbacks/suspend functions for capture)

Troubleshooting

  • Empty or boring highlight. If you don't have fancy extractBestSegment code it may select boring part (like sky, ground) etc. You can try to:
    • log sensor data or motion data. Maybe if camera was very shaky you should remove parts like this, it's probably not that interesting.
    • You should also check if you recorded actually what you were expecting. (Maybe if user wasn't pointing head on anything interesting highlight will reflect that).
    • You can always allow the user to manually select a segment as a fallback if auto selection isn't great.
  • Capture hangs or times out: Sometimes the glasses might not stop recording when expected:
    • Always implement a timeout. If you command "record 10s" and you don't get a confirmation back in, say, 12s, send a stop command or throw an error to recover.
    • Make sure bluetooth is stable. If you walk too far away from phone or some interference, it could take longer.
    • When using mock device, if you don't send the "end of video" event in your simulation, your app may just wait forever.
  • "I want it right away!": Some people may think that as soon as they click, the highlight should show up right away.
    • Tell people there's a short period to record. You can use feedback to reduce impatience (e.g. countdown "Recording 10..9..seconds").
    • Or you may just record less to get faster (e.g. 5s). But you may miss something important. So it's a tradeoff between speed and content quality.
    • As an advanced option, you might continuously buffer video on the glasses (if hardware allows) and on tap, just grab the last few seconds – this would feel instantaneous. That's more complex, but worth exploring in the future.

8) Testing Matrix

test all your integration scenarios for robustness:

ScenarioExpected ResultNotes
Mock device (simulator)Feature works with dummy data (e.g., a test pattern video)Useful for CI or if hardware is unavailable. Make sure your app can operate in this mode without crashing.
Real device – close rangeLow latency connection, smooth captureBaseline happy path. E.g., glasses 1 meter from phone. Expect minimal delay in starting capture and transferring video.
Real device – far/obstructedPossible slight lag or connection drop if out of rangeTest walking away with glasses or having a wall between. The capture might fail mid-way – ensure graceful handling (error message and ability to retry).
App in background/lockedDefined behavior: likely capture stops or is not allowedOn iOS, if the app goes background during recording, the session may pause (since no background mode by default). Document that the app must stay active. On Android, a background service would be needed to continue – not covered in basic integration.
Permission denied mid-flowClear error and recovery pathE.g., user denies microphone permission when prompted – the app should show "Cannot capture without audio permission" and let them enable it (don't just fail silently).
Disconnect mid-captureGraceful abort, maybe automatic reconnect attemptIf the glasses power off or Bluetooth disconnects during a capture, your app should handle the exception. It might cancel the operation and notify "Connection lost." Test by turning off the glasses during a recording. No crashes should occur.

Using a combination of real device tests and the mock environment will give confidence that your feature works in all conditions.


9) Observability and Logging

If you're going to deploy and/or debug the Creator Mode feature you'll want to make sure you have a good log analytics set up around it. Important events/metrics to log:

  • Connection events: connect_start (user initiate pairing), connect_success (connected to device), connect_fail (timeout or error) (timestamps and device ids for debugging purposes)
  • Permission state: log permission_state when app starts or when attempting to capture something (e.g. "Bluetooth=granted,Audio=denied"). Helpful to know if failures are from not having permissions.
  • Capture lifecycle: (if using the highlight capture feature) log highlight_start when you begin recording. highlight_success if you successfully cut and save the clip. highlight_fail if anything fails. (sub-steps if any) record_start record_end process_start process_end
  • Performance metrics: How long does it take to do each step? highlight_ms (total ms for user tap to result ready). This helps identify bottlenecks (e.g., if processing is too slow). Also track average video size, etc.
  • Reconnect attempts: If you implement auto-reconnect, log each attempt and outcome. A reconnect_count metric per session can tell you if the connection is unstable.
  • User actions: (If relevant) Log if user cancels or retries, to see if UX adjustments are needed.

By monitoring these logs (and sending anonymized analytics events if appropriate), you can continuously improve the reliability and speed of Creator Mode.


10) FAQ

  • Q: Do I need the actual hardware to start development? A: Not initially. You can use the provided mock device simulator in the SDK to simulate a glasses connection and fake camera feed. This is great for early development and CI tests. However, to truly validate the experience (especially the quality of auto-cut highlights), testing with real glasses is highly recommended before release.
  • Q: Which wearable devices are supported by Creator Mode? A: Currently, it's built for the Meta Ray-Ban AI Glasses (2023 model). These have the camera and audio hardware that the SDK interfaces with. Future Meta wearable models should work as well. Other brands' glasses or cameras are not supported in this SDK. (If you have a GoPro or other camera, you'd need a different integration – Creator Mode is specific to Meta's device and SDK.)
  • Q: Can I ship this feature to production apps now? A: Not yet. The Wearables Device Access Toolkit is in developer preview, which means it's for testing and prototyping only. Only select partners can publish apps using it to the public at this time. General availability is expected in 2026, at which point you should be able to release your Creator Mode-enabled app to all users (subject to Meta's terms).
  • Q: Does the SDK let me push content or remote commands to the glasses (like an AR display)?A: The current SDK is primarily focused on accessing the glasses' sensors (camera, microphones, etc.) and not on sending visuals to the user. Note that Ray-Ban Meta glasses have no built-in display— they're not AR glasses, they are camera and audio enabled. You can send commands (like "start recording" or "take photo") and get data back, but you can't, for example, make them display an image (since there's no lens display). Audio playback (like using the glasses' speakers) is possible through standard Bluetooth audio profiles, but that's outside the SDK's special scope.
  • Q: How does the highlight detection work? Can I customize it?A: In this MVP, highlight detection is basic. It might use simple rules (or a lightweight AI model) to choose a segment. You can absolutely customize or improve it – the SDK gives you the raw footage, and it's up to your app how to process it. For example, you could integrate an AI service or your own algorithm to detect excitement (perhaps via audio volume, or detecting faces/landmarks in the video, etc.). The provided logic is just a starting point.
  • Q: What about aspect ratio and editing for TikTok/Reels?A: TikTok and Reels favor vertical 9:16 video. The glasses capture in a wide format (likely horizontal or square). As part of the post-processing, you should crop or rotate the video into 9:16 portrait orientation if intending to post directly. You might take the center of the frame or use a smart crop to focus on the subject. Also, adding music, captions, or effects is not handled by Creator Mode itself – you would do that either in-app (if you have editing features) or by exporting the clip and using an app like CapCut. The goal of Creator Mode is to give you a raw clip that's already cut to the best moment; you can then quickly polish it for publishing.

12) Changelog

  • 2025-12-31 — Verified with Meta Wearables SDK v0.3.0 on iOS 17.1 (Xcode 15.2) and Android 13 (Pixel 6). Tested using Ray-Ban Meta Smart Glasses (Dev Preview) and the SDK mock device. Updated instructions for latest Gradle setup and iOS permission requirements.