Cloud Run Migration — Fleet Off Cloud Functions
🏁 Complete
ADR-024 · 26 services migrated · Gateway routing waves · Health probes · Gen2 workflows deleted · Artifact Registry
CF
Before
→
Run
Now
July was the month the backend fleet left Cloud Functions. Twenty-six services were migrated to Cloud Run behind the Nimbly-Go-API gateway in staged routing waves, each with real startup and liveness probes, capped max-instances, and tuned Mongo connection settings.
The old Gen1/Gen2 deploy workflows have been deleted, inter-service calls now resolve to Cloud Run URLs, and the container registry moved to Artifact Registry — no more cold-start ceiling, no more 32 MB response limit, and per-service scaling we actually control.
The old Gen1/Gen2 deploy workflows have been deleted, inter-service calls now resolve to Cloud Run URLs, and the container registry moved to Artifact Registry — no more cold-start ceiling, no more 32 MB response limit, and per-service scaling we actually control.
Services Migrated
- api-reports, api-users, api-issues, api-schedules, api-questionnaires, api-organizations, api-statistics, api-departments, api-sites, api-skus, api-lms, api-nps, api-sales, api-public, api-auth, api-user-roles, api-file-repository, api-soft-upgrade, api-bulk-operations, api-report-issues, api-miscellaneous, api-vertex-ai, api-irf-sat, api-customer-feedback, api-whatsapp and the CF Workers gateway
- Gateway routing waves 1–3 in nimbly-go-api — miscellaneous, vertex-ai, nps, soft-upgrade, questionnaires, sales, skus, lms, sites, bulk-operations, statistics and public all now resolve to Cloud Run
- /health, /ready endpoints plus startup and liveness probes added across the fleet
- Cloud Functions Gen1/Gen2 deploy workflows removed; Cloud Run image registry moved to Artifact Registry
Hardening & Fallout Fixes
- Gateway connection pooling, transport hardening and CORS preflight caching; token exchange now verifies against the auth service and fails closed on a miss
- Express body-parser limit raised to 32 MB fleet-wide; multipart uploads fixed for Cloud Run in api-miscellaneous, api-sales and api-bulk-operations (raw stream piped to busboy)
- Inbound host/entity header forwarding removed — it was causing 404s on questionnaire save, site off-days and cross-service PubSub calls
- serverSelectionTimeoutMS lowered to 5 s with maxConnecting=2 and max-instances capped at 50–100 across services
- Memory bumps where Cloud Run exposed limits — api-nps 256Mi→1Gi, api-miscellaneous 512MB→1GB; static/font dirs copied into production images
- Node pinned to 24.16.0-slim to dodge a v24.17.0 GCP SDK regression
Impact
26 Services on Cloud Run
Gateway Waves 1–3
Health Probes Fleet-Wide
Gen2 Workflows Deleted
Artifact Registry
32 MB Limit Gone
QC Performance Dashboard — Shipped ✨ New
Renamed from QC Dashboard · Live API · Formula Tooltips · Deep-Links · Pass/Fail Filter
Shipped This Month
- QC Dashboard renamed to QC Performance across UI copy, menu and route in both admin-lite and audit-admin; tab gated behind the QC_PERFORMANCE_TAB feature flag
- Live QC analytics API wired end to end — mock-data fallback removed, filters connected, null-safety and i18n gaps closed
- Per-report QC Reports table added to admin-lite at parity with audit-admin
- Info tooltip on every widget explaining the underlying formula — teams can now see how each score is derived
- Issue widgets deep-link straight into the audit-admin issue tracker with the qCategory filter pre-applied
- Pass/fail (passStatus) filter added to score-based widgets; issue widgets now respect the questionnaire filter
- Mock data served for the Nimbly demo org so sales can demo without customer data
Scoring Correctness
- Checklist questions now scored via checklistsV2 instead of the legacy field; question-level point weighting aligned with the PDF report
- Reports whose answers are stored as numeric strings now score correctly; a zero score is no longer mistaken for "no score"
- Peers RCR count no longer hits a 30 s socket timeout — dead $addFields stage removed
Impact
QC Performance Live
Formula Transparency
Issue Deep-Links
Nimbly Echo — AI Recording Platform
ElevenLabs Scribe v2 · Live Recording · Product Launch · Store Releases 1.0.2 & 1.0.3
Transcription & Pipeline
- ElevenLabs Scribe v2 transcription backend dark-deployed, then hardened — slot-gated pump, reconciler and overlap merge delivered roughly 22× throughput
- webhook_error resubmit loop stopped and an attempt cap enforced on requeue — runaway retries eliminated
- Live on-upload report generation shipped for demo, with a toast confirming generation after a Live Mic upload
- Deprecated gemini-2.0-flash-001 replaced with gemini-2.5-flash; three-grader transcript review added
- Modinity composite daily report (product-launch detector) and weekly report chain delivered; Kyochon Malaysia and Nestlé × Shopper 360 report mockups produced
App, Admin & Access
- App renamed to Nimbly Echo; versions 1.0.2 and 1.0.3 released to the stores
- Per-org feature flags API plus a super-admin Feature Access UI; Live Recording tab gated per org and live_recording split from live_report
- Live recording now asks which mic before it starts; recording date derived from filename rather than import day
- Full localization sweep — all locales translated, hardcoded copy removed, and a CI guard added to keep it that way
- Product Launch admin config section and /product-launches API; iOS recorder deletion fixed, ending the permanent "file could not be found" retry loop
Impact
Scribe v2 · 22× Throughput
Per-Org Feature Flags
Store Release 1.0.3
Mobile — Offline, Storage & Upload Reliability
ADR-034 / ADR-046 · Offline Storage Controls · Camera · Geofencing
- Offline data usage now tracked with a manual clear option, a low-storage flag, and a storage threshold check before a report is started or downloaded
- Offline report promotion fixed — attachments were being lost and promoted reports pointed at offline-scoped paths
- Background upload reliability (ADR-034 / ADR-046) — stall recovery, zombie-upload resurrection stopped, stale signed URLs refreshed, transport errors correctly classified as retryable
- Batch update-answers endpoint adopted for cloud-path reconcile, eliminating 409 conflict storms during autosave
- Camera fixes — Android "No camera device available" after capture, iOS flash overexposure, 0.5× ultra-wide zoom, and geotag/timestamp watermark baked into the saved photo
- Confidence-gated geofencing shipped (ADR-036 / ADR-044) with the server-side location-required gate and geofence UI
- Global query-cache error handler (ADR-030) landed; SOP and reference doc links now work offline
- Onboarding infinite loop fixed; Continue card self-heals for deleted schedules
Impact
No More 409 Storms
Storage Controls
Upload Stall Recovery
Cron Migration & RTDB Decommission
ADR-40/41/42/43 Cloud Tasks · ADR-052 · ADR-058 Phases 0–3
Cloud Scheduler + Cloud Tasks
- Report overdue (ADR-41), report recap (ADR-40), issue daily recap (ADR-42) and issue escalation (ADR-43) all migrated off Firebase Cloud Functions onto Cloud Scheduler + Cloud Tasks
- Issue daily-recap WhatsApp now dispatched via pubsub-bulk; report-due push and WhatsApp routed through Pub/Sub (ADR-28)
- Legacy queue-makers and handlers deleted across audit-functions, nimbly-cloud and nimbly-web-api; recap schedule corrected to 5:03 AM Jakarta
Firebase RTDB Retirement
- ADR-058 Phases 0–3 executed — siteQuestionnaireSchedule leg, siteScheduleHistory writes, multi-site schedule RTDB writes, auditorSiteIndex writers and the backwardHook Firestore→RTDB mirror all removed
- ADR-052 — audit-admin and audit-web now activate users via API instead of writing to RTDB; RTDB→Mongo sync triggers deleted; api-users dual-writes user mutations to Mongo
- api-file-repository organization reads migrated from RTDB to MongoDB (ADR-053)
- Six dead report endpoints, the report onCreate/onDelete hooks and the initRoleOnNewUser trigger removed
Impact
4 Crons on Cloud Tasks
RTDB Phases 0–3 Done
Dead Code Purged
Security & Access Control
Org Scoping · RBAC Middleware · Privilege Escalation · Credential Permission
- Vertical privilege escalation closed in api-users user create/update; email changes blocked on activated accounts
- Org-scoped access enforced on user query endpoints and findByID; sensitive fields stripped from user API responses
- RBAC middleware added to all api-issues mutating routes; org scoping enforced on issue tracker and indicator endpoints
- Auth and org-scoping added to api-organizations endpoints; SUPERADMIN correctly exempted from the org match check; 403 returned instead of 401 for authorization failures
- New admin:user:credentials RBAC resource in nimbly-common — password changes and the audit-admin password field are now gated behind it
- CS utility routes in api-nps restricted to superadmin, with read-only routes opened to non-superadmins
- Gateway token exchange now fails closed when the auth service returns no token, instead of silently falling back
Impact
Escalation Closed
Fleet Org-Scoped
Credential RBAC
Quality Engineering — ADR-48 & Automation
48 Coverage PRs · 33 Contract-Test PRs · Maestro E2E · Web-Admin Regression Suite · AI Reviewer
ADR-48 Test Coverage Programme
- PR test-existence check rolled out via the shared github-workflows repo to ~25 services and both frontends — a PR touching business logic without a test is now flagged at review time
- Unit-test coverage raised across api-users, api-reports, api-issues, api-nps, api-statistics, api-organizations, api-departments, api-questionnaires, api-skus, api-lms, api-sales, api-soft-upgrade, api-user-roles, api-attachment-gallery, api-customer-feedback, api-bulk-operations and api-report-issues
- Pact broker health check added before provider verification across the fleet, with retries to absorb transient broker blips — the single biggest source of red CI in June
- catch-untested-code Claude skill shipped (ADR-48 Part 3) so untested changes get caught before the commit, not after
- Offline-first integration coverage added to audit-lite across auth, home, schedule and questionnaire flows
E2E Automation
- Web-admin regression suite built out — create schedule (all repeat presets, custom, ad-hoc-only, until-date, until-occurrences), create user, site, department and questionnaire with edit/block/activate/archive/restore coverage
- Daily scheduled sanity runs for both mobile and web admin (3am + 2pm JKT), with Discord start and result notifications
- Broad flakiness campaign — animation waits, search-pattern stabilisation, robust back navigation, and suites that continue past a single scenario failure
- Staging build/OTA lane added to the audit-lite release-sanity workflow; each EAS Update now tagged as its own Sentry release
- AI reviewer upgraded to a three-stage reviewer with anchoring and refute-to-drop verification, hardened against malformed model responses
Impact
48 Coverage PRs
Test Gate on ~25 Repos
Daily Sanity Runs
Pact CI Stabilised
Performance, Product & Internal Tooling
Query Timeouts · Bulk Ops UX · Reports Hub · Radar · Compass
Performance & Reliability
- Issue tracker date range no longer fails at the BSON document limit — range is bounded and resolved without a per-issue $lookup, with the pre-query projecting issueID before grouping
- api-public GET /issue/ 60 s timeout eliminated — null-key $lookup COLLSCAN removed, departments lookup indexed, rows pruned by siteID early
- Report aggregate writes made monotonic and 409-free under contention, with delta version read from primary on retry
- Attachment gallery query timeout and array-param bug fixed; attachment refs now matched by identity instead of a 4-character prefix
Product & Admin
- Delete Missed Reports now supports multi-select across sites, questionnaires and auditors, fanning out across multi-site mappings
- Cause Analysis, Corrective Action and Additional Attachments columns added to Reports Hub and audit report Excel exports; recurring issues included in date-filtered issue exports
- Asset Tracker bulk QR code export shipped in admin-lite, with Unicode filename support and a fail-fast 10k limit
- Drag-and-drop reordering for audit-admin home navigation cards — admins can lay the landing page out to match how their team actually works
- Org-level noFixPolicy setting added to report settings; phone numbers editable on activated auditor accounts
- Report push notifications fixed — wrong recipients, a supervisor crash, and a zero APNs badge that was clearing the notification tray
- audit-admin recovers from stale-deploy chunk load errors; TypeScript upgraded 4.9 → 7.0 in three steps
Internal Tooling
- Radar — new User Lookup utility with a unified, timezone-aware timeline and enriched Sentry breadcrumbs for support triage
- Compass — semantic search via Ternlight embeddings, full doc set served with auth on docs.hellonimbly.com, plus QC Performance and escalation documentation
- Claude skills packaged as a plugin marketplace with automated onboarding and auto-update; report deploy skills added for internal and public hosting
Impact
60 s Timeouts Gone
Bulk Multi-Select
QR Export
Radar User Lookup
Compass Semantic Search