OTA Update System

This document covers the over-the-air update mechanism with A/B versioning and Ed25519 signature verification.

Key Files

• crates/bf_mobile/src/ota.rs - OTA asset management system
• crates/bf_resident/src/handlers/ota.rs - Backend OTA endpoints
• crates/bf_types/src/ota.rs - Build info and signed types
• Native Lynx plugin - Download and callback handling

Overview

The OTA system enables seamless frontend updates without app store releases:

┌─────────────────────────────────────────────────────────────────────┐
│                       OTA Update Flow                               │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  ┌───────────────┐                     ┌───────────────┐           │
│  │  Mobile App   │──── GET /mobile ───►│   Backend     │           │
│  │               │     version check   │   bf_resident │           │
│  └───────┬───────┘                     └───────┬───────┘           │
│          │                                      │                   │
│          │   ◄─── Multipart Response ──────────┘                   │
│          │        [BuildInfo JSON][Zstd Data]                      │
│          │                                                         │
│          ▼                                                         │
│  ┌───────────────────────────────────────────────────────────────┐ │
│  │                    Verification Steps                         │ │
│  │                                                               │ │
│  │  1. Parse BuildInfo JSON from response                        │ │
│  │  2. Verify Ed25519 signature against public key               │ │
│  │  3. Verify SHA-256 file hash matches                          │ │
│  │  4. Write data.zstd to alternate directory (A or B)           │ │
│  │  5. Remove invalid.marker on success                          │ │
│  │  6. Emit ReloadWebview event to frontend                      │ │
│  └───────────────────────────────────────────────────────────────┘ │
│          │                                                         │
│          ▼                                                         │
│  ┌───────────────────────────────────────────────────────────────┐ │
│  │                    Asset Serving                               │ │
│  │                                                               │ │
│  │  Request: /index.html                                         │ │
│  │     │                                                         │ │
│  │     ▼                                                         │ │
│  │  OtaAssets::get(key)                                          │ │
│  │     │                                                         │ │
│  │     ├─► OTA version exists? ─► Decompress & serve from A/B    │ │
│  │     │                                                         │ │
│  │     └─► No OTA version ─────► Serve embedded assets           │ │
│  └───────────────────────────────────────────────────────────────┘ │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘

A/B Versioning Strategy

Two directories provide zero-downtime updates:

data_dir/
├── version.check     # App version marker (detects native app updates)
├── A/
│   ├── version.json  # SignedBuildInfo
│   ├── data.zstd     # Compressed assets
│   └── invalid.marker (deleted on successful verify)
└── B/
    ├── version.json
    ├── data.zstd
    └── invalid.marker

Version Selection

// crates/bf_mobile/src/ota.rs:393

pub fn get_current_version(data_dir: &PathBuf) -> AssetVersion {
    let version_a = read_version_from_dir(&AssetVersion::A(0).dir(data_dir), false);
    let version_b = read_version_from_dir(&AssetVersion::B(0).dir(data_dir), false);

    match (version_a, version_b) {
        (Some(a), Some(b)) => {
            if a.data.version >= b.data.version {
                AssetVersion::A(a.data.version)
            } else {
                AssetVersion::B(b.data.version)
            }
        }
        (Some(a), None) => AssetVersion::A(a.data.version),
        (None, Some(b)) => AssetVersion::B(b.data.version),
        (None, None) => AssetVersion::None,
    }
}

Version Alternation

New versions write to the opposite directory:

pub enum AssetVersion {
    None,
    A(u32),  // Version number
    B(u32),
}

impl AssetVersion {
    pub fn alternate(&self) -> AssetVersion {
        match self {
            AssetVersion::None | AssetVersion::B(_) => AssetVersion::A(0),
            AssetVersion::A(_) => AssetVersion::B(0),
        }
    }
}

Build Info Types

BuildInfo

// crates/bf_types/src/ota.rs

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct BuildInfo {
    /// Sequential version number
    pub version: u32,
    /// SHA-256 hash of the compressed asset file (base64)
    pub file_hash: String,
    /// Minimum required native app version
    pub min_app_version: String,
    /// Build timestamp
    pub built_at: String,
}

SignedBuildInfo

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct SignedBuildInfo {
    /// The build information
    pub data: BuildInfo,
    /// Ed25519 signature of JSON-serialized BuildInfo (base64)
    pub data_signature: String,
}

Server Build Manifest

The backend stores live builds in builds_manifest.json in the MOBILE_BUILDS R2 bucket (the pre-manifest current_build.json is kept as a read-only fallback and migrated into a one-entry manifest on first load):

// crates/bf_resident/src/handlers/ota.rs
struct BuildsManifest {
    next_version: u32,           // monotonic counter across all builds
    entries: Vec<SignedBuildInfo>,
}

Version selection (GET /mobile/version): the server returns the highest-version build whose min_app_version the client's native app version satisfies. A device on an older native app keeps receiving the newest bundle that still supports it, rather than being cut off when a newer build raises min_app_version. "App too old" is only returned when the client is below every build's minimum.

Retention (pruned on each deploy): entries older than 3 months are dropped, then entries are collapsed to one per min_app_version, keeping the highest-version build for each. The newest build overall is always retained, however old, so the manifest never goes empty. Because pruned manifests can still reference build files up to 3 months old (and the newest indefinitely), the R2 bucket must not have an expiry lifecycle rule on v* files.

Deploys pass min_app_version from the BROADFORD_MIN_APP_VERSION env var when set (validated as plain x.y.z), falling back to the exporting build's own tauri.conf.json version. So each store release naturally starts a new track in the manifest, and the override lets you deploy a fix into an older track (e.g. for fleets still on a previous store release) without checking out old source.

Signature Verification

Mobile apps maintain two sources of trusted public keys, tried in order:

1. JWKS cache — keys fetched at startup from https://assets.broadfordlife.com/.well-known/jwks.json and cached in the app's data directory with ETag support. All keys in the JWKS are tried during verification.
2. Compile-time fallback — BF_MOBILE_OTA_PUBLIC_KEY baked in via dotenv_codegen::dotenv!(). Used only if the JWKS cache is empty (fresh install that never reached the network, or dev builds).

The JWKS is shared across staging and production by design — see Shared JWKS Across Environments below. Every app (staging backend or prod backend) fetches the same key set and accepts bundles signed by any key in it.

Ed25519 Verification

// crates/bf_mobile/src/ota.rs (verification uses JWKS first, fallback second)

let public_keys = if !self.jwks_keys.is_empty() {
    self.jwks_keys.clone()
} else {
    // Compile-time fallback — only used when JWKS cache is empty
    let fallback = dotenv_codegen::dotenv!("BF_MOBILE_OTA_PUBLIC_KEY");
    vec![ed25519_compact::PublicKey::from_slice(&BASE64.decode(fallback)?)?]
};

// Serialize BuildInfo and verify against any trusted key
let data_json = serde_json::to_string(&version_info.data)?;
let signature = ed25519_compact::Signature::from_slice(&BASE64.decode(&version_info.data_signature)?)?;

let verified = public_keys
    .iter()
    .any(|pk| pk.verify(data_json.as_bytes(), &signature).is_ok());

Shared JWKS Across Environments

OTA signing keys deliberately break the "separate prod from staging" convention used elsewhere in the platform. The reasoning:

• A mobile app build is a single binary that can point at either backend at runtime (via VITE_BF_MOBILE_BACKEND_URL). If staging and prod had disjoint trust sets, switching backends would mean either embedding both key sets or re-fetching a different JWKS — extra moving parts for a system whose whole job is to deliver a frontend bundle safely.
• The OTA trust model already spans the whole app population (one bad key rotation affects every install). Environment split adds complexity without extra safety.

Who writes the JWKS

Only one worker is allowed to mutate the trusted key set: staging bf-resident.

Aspect	Staging (bf-resident-staging)	Production (bf-resident)
Build features	default (mock enabled)	--production (no mock)
POST /mobile/keys route	compiled in	compiled out (#[cfg(feature = "mock")])
ota_signing_keys D1 table	authoritative writer	not used
Shared ASSETS R2 bucket (assets.broadfordlife.com/.well-known/jwks.json)	writes JWKS	read-only (via mobile clients)
BF_MOBILE_OTA_PRIVATE_KEY secret	signs staging bundles	signs prod bundles (same key value)

This is why crates/bf_resident/src/handlers/ota.rs's register_ota_key is gated on #[cfg(feature = "mock")] — it's the "only staging owns rotation" enforcement, not a debug-only feature.

Consequences for rotation

• scripts/rotate-secrets.sh ota POSTs to both staging and production for historical/symmetry reasons; the production POST 404s and the script warns but continues. Staging's success is what actually rotates the JWKS for all clients.
• Deploy the new BF_MOBILE_OTA_PRIVATE_KEY to both --env staging and --env production so both workers sign bundles with the matching key.
• The 14-day coexistence window in list_recent_ota_signing_keys retires old keys automatically — no manual delete step.

File Hash Verification

// Compute SHA-256 of the data file
let mut hasher = Sha256::new();
hasher.update(binary_data);
let file_hash = BASE64.encode(hasher.finalize());

// Compare with expected hash
if version_info.data.file_hash != file_hash {
    return Err("File hash verification failed");
}

Asset Bundle Format

Multipart Response

Backend returns concatenated format:

[BuildInfo JSON}{Zstd compressed data]

The double brace }} marks the boundary:

fn find_second_brace(data: &[u8]) -> Option<usize> {
    let mut count = 0;
    for (i, &byte) in data.iter().enumerate() {
        if byte == b'}' {
            count += 1;
            if count == 2 {
                return Some(i);
            }
        }
    }
    None
}

Asset Compression

Assets are double-compressed:

1. Brotli - Individual file compression (at build time)
2. Zstd - Bundle compression (level 22 for OTA transfer)

Serving decompresses Brotli on-demand:

fn get(&self, key: &AssetKey) -> Option<Cow<'_, [u8]>> {
    if let Some(displaying) = self.displaying.as_ref() {
        displaying
            .assets
            .get(key.as_ref())
            .map(|data| {
                let mut buf = Vec::with_capacity(data.len());
                brotli::BrotliDecompress(&mut data, &mut buf)
                    .map(|()| buf)
            })
            .and_then(Result::ok)
            .map(Cow::Owned)
    } else {
        self.default_assets.get(key)
    }
}

Update Check Loop

Periodic Checking

// crates/bf_mobile/src/lib.rs

// iOS: Check every 5 minutes
tauri::async_runtime::spawn(async move {
    tokio::time::sleep(Duration::from_secs(1)).await;
    let mut interval = tokio::time::interval(Duration::from_secs(60 * 5));

    loop {
        interval.tick().await;
        if !ADDING_NEW_VERSION.load(Ordering::Relaxed) {
            check_latest_version_fn().await;
        }
    }
});

// Android: Check once (to minimize battery)
tauri::async_runtime::spawn(async move {
    tokio::time::sleep(Duration::from_secs(1)).await;
    if !ADDING_NEW_VERSION.load(Ordering::Relaxed) {
        check_latest_version_fn().await;
    }
});

Initial Version Check

Synchronous check on first open:

#[tauri::command]
pub async fn check_initial_version() -> bool {
    // Skip if already have OTA version
    if has_ota_version() {
        return false;
    }

    // Reset flag and trigger check
    INITIAL_VERSION_CHECK_DONE.store(false, Ordering::Relaxed);

    let token = get_token().await;
    lynx::check_latest_version(token, base_url);

    // Wait for callback (with 10s timeout)
    while !INITIAL_VERSION_CHECK_DONE.load(Ordering::Relaxed) {
        if start.elapsed() > Duration::from_secs(10) {
            return false;
        }
        tokio::time::sleep(Duration::from_millis(100)).await;
    }

    has_ota_version()
}

App Version Change Detection

Native app updates clear OTA data:

// crates/bf_mobile/src/ota.rs:569

fn add_app_handle(&mut self, handle: AppHandle) {
    let current_app_version = handle.package_info().version.to_string();
    let version_check_path = data_dir.join("version.check");

    match std::fs::read_to_string(&version_check_path) {
        Ok(previous_app_version) => {
            if previous_app_version != current_app_version {
                // App version changed - delete OTA assets
                std::fs::remove_dir_all(&version_a_dir)?;
                std::fs::remove_dir_all(&version_b_dir)?;
                std::fs::write(&version_check_path, &current_app_version)?;
            }
        }
        Err(NotFound) => {
            // First run - clean slate
            std::fs::write(&version_check_path, &current_app_version)?;
        }
    }
}

After the wipe the app serves its (newer) embedded assets and reports current=0 on the next version check, so it re-downloads from the manifest. The bundle it receives may carry a min_app_version older than the app — that is expected and valid: min_app_version is a floor, not an exact match, and the server picks the highest-version build with min_app_version <= client app. The client never inspects min_app_version (only signature and hash), so it accepts such a bundle. This is also why a hotfix deployed into an older track (via BROADFORD_MIN_APP_VERSION) is served to newer apps too if it has the highest version number they qualify for — after an old-track hotfix, deploy the new track as well (a normal branch push does this) so newer apps aren't left running the older track's frontend.

Invalid Marker Pattern

Atomic downloads via marker file:

// 1. Create invalid marker before writing
std::fs::File::create(new_dir.join("invalid.marker"))?
    .write_all("mark".as_bytes())?;

// 2. Write version.json and data.zstd
std::fs::File::create(new_dir.join("version.json"))?
    .write_all(build_data)?;
std::fs::File::create(new_dir.join("data.zstd"))?
    .write_all(binary_data)?;

// 3. Verify signature and hash
public_key.verify(data_json.as_bytes(), &signature)?;
if file_hash != expected_hash { return Err(...); }

// 4. Remove marker only on success
std::fs::remove_file(new_dir.join("invalid.marker"))?;

Directories with invalid.marker are ignored during version selection.

CSP Hash Collection

Content Security Policy hashes are preserved:

pub struct RuntimeAssets {
    assets: Vec<(String, Vec<u8>)>,
    /// Hashes for all HTML files
    global_hashes: Vec<RuntimeCspHash>,
    /// Per-file specific hashes
    html_hashes: Vec<(String, Vec<RuntimeCspHash>)>,
}

pub enum RuntimeCspHash {
    Script(String),  // script-src hashes
    Style(String),   // style-src hashes
}

OTA Bundle Creation

For deployment, the app can export its assets:

// --export-assets <environment> flag
if let Some(env_name) = std::env::args()
    .windows(2)
    .find(|w| w[0].trim() == "--export-assets")
    .map(|w| w[1].clone())
{
    let zip = ota::make_data();  // Zstd-compressed bundle

    // Sign a short-lived JWT with MOBILE_SERVICE_AUTH_SECRET
    let token = sign_jwt(&auth_secret, "bf_mobile");

    // Upload to backend
    client
        .put(format!("{}/mobile/deploy?min_app_version={}", backend_url, app_version))
        .header("Authorization", format!("Bearer {}", token))
        .body(zip)
        .send()
        .await?;
}

Version String Format

pub fn get_version(&self) -> String {
    // Format: "1.2.3" or "1.2.3+42" (with OTA version)
    let mut app_version = handle.config().version.unwrap_or("0.0.0");

    match &self.displaying_version {
        AssetVersion::A(x) | AssetVersion::B(x) => {
            app_version.push('+');
            app_version.push_str(&x.to_string());
        }
        _ => {}
    }

    app_version
}

Commands

Get Version

#[tauri::command]
pub fn get_version() -> String {
    // Returns "1.2.3" or "1.2.3+42"
}

Reset to Embedded

#[tauri::command]
pub fn reset_remote() {
    // Delete A/ and B/ directories
    // Reload webview with embedded assets
}

CI/CD Deployment

OTA bundles are deployed automatically via GitLab CI on push:

• staging branch → deploy-ota-staging job → uploads to staging backend
• main branch → deploy-ota-production job → uploads to production backend

How It Works

Each job builds a Linux desktop binary (Tauri --no-bundle), then runs it with --export-assets <env>. The binary:

1. Packages embedded frontend assets (CDR + Zstd compression)
2. Signs a short-lived HS256 JWT using MOBILE_SERVICE_AUTH_SECRET
3. Uploads the bundle to PUT {backend_url}/mobile/deploy?min_app_version={version} — {version} is BROADFORD_MIN_APP_VERSION if set, else the build's tauri.conf.json version

Required GitLab CI/CD Variables

Variable	Scope	Description
CLOUDFLARE_API_TOKEN	Global	Wrangler auth for worker deploys
MOBILE_SERVICE_AUTH_SECRET	Environment: ota-staging	HMAC secret for signing staging deploy JWTs
MOBILE_SERVICE_AUTH_SECRET	Environment: ota-production	HMAC secret for signing production deploy JWTs

Compile-Time Variables (automatic)

These are read from the committed .env file via dotenv_codegen::dotenv!() at compile time — no CI configuration needed:

• BF_MOBILE_OTA_PUBLIC_KEY — Ed25519 public key for on-device signature verification
• VITE_CLERK_PUBLISHABLE_KEY / VITE_CLERK_PUBLISHABLE_KEY_DEV — Clerk auth keys
• VITE_BF_MOBILE_BACKEND_URL — JSON array of environment URLs (also read at runtime via dotenvy)

Linux Build Prerequisites

Tauri requires system libraries at compile time even though the binary exits before running the GUI. The CI jobs install:

nodejs npm clang libwebkit2gtk-4.1-dev build-essential
libssl-dev libayatana-appindicator3-dev librsvg2-dev

See Also

• Mobile Architecture - App structure
• Client Caching - Asset caching
• Notification System - Update notifications