Skip to main content

Key Rotation Plan

How Secret Updates Work​

All secrets are Cloudflare Worker secrets loaded via env.secret() at the start of each Worker request or Durable Object invocation. When a secret is rotated via wrangler secret put:

  • In-flight requests continue using the old secret until they complete
  • The next request to the Worker picks up the new secret
  • Durable Objects must be evicted/shut down before they receive updated secrets — the next alarm or request after eviction uses the new secret
  • Retrying within the same request will NOT get the new secret — the retry must come from the caller (mobile app, next alarm, queue retry)

Rotation Resilience Strategy​

MechanismHow it handles rotation
Mobile app withRetry()Retries once on 500/502 after 2s delay — new request gets fresh secret
DO alarm schedulingFailed alarm records error and reschedules — next alarm is a fresh invocation
Email queueFailed send triggers message.retry() — queue consumer is a fresh invocation
Fire-and-forget loggingAcceptable to lose a few log lines; DO logger retains buffer on auth failure
User-initiated actionsUser retries by clicking again — new request gets fresh secret

Rotation Script​

Use scripts/rotate-secrets.sh to deploy secrets to all relevant workers in parallel:

# Rotate a specific secret
./scripts/rotate-secrets.sh service-auth --env production

# Rotate all secrets interactively
./scripts/rotate-secrets.sh all --env production

Per-Secret Details​

SERVICE_AUTH_SECRET / AUTH_SERVICE_AUTH_SECRET​

Workers: bf_resident, bf_notify, bf_smart, bf_user, bf_storage, bf_email_forwarder, bf_auth

All workers must be updated simultaneously to avoid auth mismatches. The rotation script handles this by deploying in parallel.

  • Service tokens are 5-minute-lived JWTs (HS256)
  • The mismatch window is the time between the first and last worker picking up the new secret
  • DO alarm tasks (lifecycle checks) are resilient via alarm rescheduling
  • Mobile app withRetry() covers transient 500 errors from service auth failures

YALE_API_KEY​

Workers: bf_resident, bf_notify, bf_user, bf_smart

Call originResilience
Mobile → device control (lock/unlock)withRetry() retries once on 500/502
DO alarm → device refresh (hourly)Alarm reschedules on failure, next alarm gets new secret
DO alarm → activity log refreshSame as above
Yale → bf_notify webhook verificationYale re-sends failed webhooks
Web app → IT setup (manual admin action)User retries manually

CLOUDFLARE_TURN_KEY_ID / CLOUDFLARE_TURN_KEY_TOKEN​

Workers: bf_resident only

Mobile VideoFeedPlayer retries TURN credential fetch once before falling back to STUN-only. Single handler, single worker.

POSTHOG_API_KEY​

Workers: bf_resident, bf_notify, bf_user

ComponentResilience
bf_resident / bf_notify (fire-and-forget)Acceptable loss, console.log fallback exists
bf_user DO logger (buffered flush)Retains buffer on 401/403, next flush in fresh invocation retries

OPENROUTER_API_KEY​

Workers: bf_auth, bf_resident

Both usages (verification code extraction, floor plan analysis) are user-initiated with existing 3-attempt retry. While retries within the same request use the stale secret, the user can re-trigger the action, which creates a new request with the fresh secret.

RESEND_API_KEY​

Workers: bf_resident only

Emails are sent via the EMAIL_QUEUE (Cloudflare Queue). The queue consumer loads RESEND_API_KEY fresh on each invocation. Failed messages are retried by the queue (up to 3 retries), with each retry being a new Worker invocation that picks up the new secret. Dead-letter queue (email-queue-dlq) catches permanently failed messages.

CLERK_SECRET_KEY​

Workers: bf_resident only

Used for invite CRUD and admin management — all user-initiated synchronous actions. If the secret is stale, the operation fails and the user retries by clicking again. The next request gets the new secret.

Note: Rotate in the Clerk Dashboard first, then deploy the new secret via wrangler secret put.

BF_ENCRYPTION_KEY​

Do NOT rotate. This key encrypts stored data. Rotation would make existing encrypted data unreadable without a migration.

Mobile OTA Keys (BF_MOBILE_OTA_PRIVATE_KEY)​

Workers: bf_resident only

OTA bundles are signed with an Ed25519 private key on the backend and verified by the mobile app using public keys served from a JWKS endpoint.

Cross-environment design (intentional):

Unlike most secrets, OTA signing keys are shared across staging and production by design. Mobile apps — regardless of which backend they point at — trust the same JWKS, so a bundle signed in staging verifies in production and vice versa. The trust set is single-sourced to avoid fragmenting the mobile app's trust anchors per environment.

To enforce "one writer" for the shared JWKS, the registration endpoint is only compiled into the staging worker:

  • Staging (bf-resident-staging) — built with the default mock feature. It exposes POST /mobile/keys, owns writes to the ota_signing_keys D1 table, and publishes the JWKS to the shared ASSETS R2 bucket. This is the single source of truth for the trusted key set.
  • Production (bf-resident) — built with --production (no mock feature). The POST /mobile/keys route is compiled out. Production only signs bundles with BF_MOBILE_OTA_PRIVATE_KEY; it never touches the JWKS.
  • Shared artifacts — the ASSETS R2 bucket (served at https://assets.broadfordlife.com/.well-known/jwks.json) and the BF_MOBILE_OTA_PRIVATE_KEY secret value are the same across both environments. Staging writes, both envs sign and verify.

Architecture:

  • Public keys are stored in the ota_signing_keys D1 table (staging only) with a kid and created_at timestamp — crates/bf_resident_db/migrations/008_ota_signing_keys.sql
  • POST /mobile/keys (crates/bf_resident/src/handlers/ota.rs, gated by #[cfg(feature = "mock")] → staging-only; protected by MOBILE_SERVICE_AUTH_SECRET) accepts a new public key, inserts it into the database, rebuilds the JWKS file from all keys created in the last 14 days, and uploads it to the shared ASSETS R2 bucket at .well-known/jwks.json
  • The JWKS is served publicly at https://assets.broadfordlife.com/.well-known/jwks.json and consumed by all mobile clients (staging + production + dev builds pointed at a real backend)
  • The mobile app fetches and caches the JWKS at startup (with ETag-based conditional requests), storing the cache in the app's data directory — crates/bf_mobile/src/ota.rs (fetch_and_cache_jwks)
  • OTA signature verification tries all keys from the cached JWKS; if no JWKS keys are available (fetch failed or dev environment), it falls back to the compile-time embedded BF_MOBILE_OTA_PUBLIC_KEY in .env
  • In dev environment, the JWKS fetch is skipped entirely

Rotation procedure:

./scripts/rotate-secrets.sh ota

The script offers two options:

  1. Generate a new Ed25519 keypair automatically (via openssl)
  2. Provide your own private key and public key (base64-encoded)

It then:

  1. Deploys the private key (as a 64-byte seed+pubkey pair) to bf-resident via wrangler secret put. Run once with --env staging and once with --env production so both environments sign with the matching key.
  2. Registers the public key via POST /mobile/keys against staging (app-staging.broadfordlife.com), which writes the authoritative JWKS to the shared R2 bucket. The script also POSTs to app.broadfordlife.com for legacy/symmetry, but that route is compiled out in production and will return 404 — this is expected and the script treats it as a non-fatal warning. Staging alone is what actually updates the JWKS that prod clients consume.

Resilience: The 14-day key window means old and new keys coexist in the JWKS. Bundles signed with the old key remain verifiable until the old key ages out. The mobile app caches the JWKS locally, so even if the fetch fails on a subsequent launch, the last-known keys are used. The compile-time fallback key provides a safety net for apps that have never successfully fetched the JWKS.

Queue Setup​

Before first rotation involving email, ensure the queues exist:

# Production
wrangler queues create email-queue
wrangler queues create email-queue-dlq

# Staging
wrangler queues create email-queue-staging
wrangler queues create email-queue-staging-dlq