Disclaimer: The following text is an AI-generated summary of the design decisions and evolution of the Google Health data pipeline built with Swamp and Claude. All work was done iteratively with claude-code handling implementation while I approved steps and provided direction.

Choosing the Data Source

The starting prompt was simple: “how to extract my health metrics from zepp mobile app, I want to stream them into my victoria metrics.”

I wanted my wearable data in my own time-series database — VictoriaMetrics on my Unraid box — instead of renting it back through an app. The first candidate was an old Amazfit paired to the Zepp app. Every path into it failed:

The move that mattered was stopping: “actually lets pause here for a moment, how to extract data from pixel watches.” Switching the source beat grinding the tactic. A @magistr/fitbit model was built for the legacy API and deleted the same day — registration is closed, the API sunsets September 2026. The Google Health API (health.googleapis.com/v4) is the only road forward, and in June it was weeks old and thin on docs.

Everything became one Swamp extension model — @magistr/google-health, with authorize, exchange, probe, sync, and derive methods — so every answer the API ever gave landed as versioned, queryable data instead of scrollback.

The OAuth Tax

All Google Health scopes are Restricted. The consent screen started in “Testing” mode, and in that mode Google expires the refresh token after ~7 days. For weeks the pipeline demanded a manual re-auth ritual — open consent URL, approve, paste a 4/0AdkVLP... code into exchange.

The fix was one console click, found late: publish the OAuth consent screen to Production. An unverified single-user app is fine; the refresh token then persists until revoked. The weekly tax was self-inflicted.

Lesson: read the token-expiry rules of the publishing status before accepting a re-auth ritual as the cost of doing business.

The DataPoint Shape — Numbers Are Strings

There is no value field. Each DataPoint nests its reading under a camelCased key named after the data type, and every number arrives as a string:

{
  "dataSource": { "...": "..." },
  "heartRate": {
    "sampleTime": { "physicalTime": "2026-06-20T07:14:03Z" },
    "beatsPerMinute": "61"
  }
}

Where the timestamp lives depends on the type’s cardinality:

Two more traps: type ids are kebab-case in URLs but snake_case in filter params, and units are baked into field names — distanceMillimeters, weightGrams — deliberately, to avoid losing precision.

Rather than guess, sync uses tolerant extractors (take the one object that isn’t dataSource, read the first scalar that isn’t time or metadata, Number() it) and stores the first DataPoint of every type as a raw-<type> resource. swamp data get google-health raw-exercise shows the exact shape the API really returns before the mapping gets trusted.

Lesson: verify shapes against live data, not docs. The raw resources are the audit trail — I can still query the sample that proved the numbers arrive as strings.

Probing a Closed Enum

“probe api what else there is exist and undocumented and could be usefull”

At build time there was no index of data types and no list endpoint (GET users/me/dataTypes 404s). The API is a closed enum, so a probe method was added: give it candidate type ids, it fetches a small sample of each and stores the raw shape. The status code is the answer:

• 200 with points — real, and I have data
• 200 empty — valid, but no device or log for it (blood-glucose, nutrition)
• 403 — real, needs a scope I haven’t granted (that’s how ECG was found)
• 400 "data type ID not supported" — not a thing
• 400 "List is not supported" — rollup-only type, different endpoint entirely

The probe sweep paid off beyond the obvious heart-rate/steps/sleep: daily-heart-rate-variability (deep-sleep RMSSD — the readiness input), daily-heart-rate-zones (my real zone cutoffs: 112/135/163), daily-sleep-temperature-derivations (skin temp with a 30-day baseline already computed), active-zone-minutes, respiratory-rate-sleep-summary, run-vo2-max.

The most important probe result was negative: every derived score — readiness, cardio load, stress, sleep score — returns 400. Google computes them client-side and does not expose them. Stress (cEDA “Body Response”) has no data type at all. If I wanted those numbers, I had to rebuild them from raw.

Backfill and the VictoriaMetrics Gotchas

“backfill data to nov 2025”

Pixel Watch heart rate is high-frequency — ~26 samples/minute, about 8 million HR points over 7 months. Backfill paginates newest-first via nextPageToken until points predate the target date, flushing to VM every 200k lines to stay under the 64MB ingest limit. A full run takes 20–40 minutes.

Three bugs came out of this phase, none of them where they first appeared:

Bug 1 — the timezone day-shift. Daily metrics were bucketed with new Date(y, m, d) — local midnight, which is 22:00 UTC the previous day. Every daily point landed in the wrong UTC day. Fix: Date.UTC(). The server had already normalized the civil date; re-localizing it was the bug.

Bug 2 — the re-derive no-op. Recalibrated derivation formulas, re-ran derive, nothing changed. Suspected stale bundles, suspected schema defaults — both misattributions. A clean repro showed the real cause: VictoriaMetrics does not overwrite samples at identical timestamps. Old midnight-stamped values simply won. Fix: derive now deletes its own output series before re-pushing. Idempotency required delete-then-write.

Bug 3 — the query 422. query_range rejects requests where (end − start) / step exceeds 30k points per series. Per-minute HR over months must be chunked — 14-day windows in derive.

One non-bug mattered as much: per-metric history depth genuinely differs. Phone-sourced metrics reach Nov 2025; watch-only intraday HR starts ~Feb 2026; intraday SpO2 turned to garbage after the March Pixel Watch feature drop — a documented Google bug, not mine. The dashboard shows daily SpO2 only.

Lesson: don’t “fix” missing data. First establish whether it ever existed.

Reverse-Engineering the Readiness Score

“now explain all magical constants in your formulas”

The derived metrics were rebuilt from first principles: Cardio Load as Banister TRIMP over per-minute HR (only counting HR above the 112 bpm fat-burn floor, like the app), ACWR as the 7-day/28-day load ratio, Readiness as a composite of deep-sleep HRV, resting HR, and sleep z-scores against a trailing baseline.

Then came calibration. I supplied 8 days of anchor values read off the app screen, and the fit collapsed the mystery:

• Cardio Load was 3× under-scaled; loadScale=1.35 matches the app within ±3.
• Readiness is essentially linear in one input: readiness ≈ 2.85 × last-night-deep-sleep-RMSSD − 31, r ≈ 0.97 against the app. Sleep duration and resting HR barely move it, despite what Fitbit’s own docs imply. Six of eight anchor days matched within 1–3 points.

Lesson: the proprietary score they won’t sell you back is a straight line through a single measurement you already own.

Sessions, Running Form, and ECG

“each training session and run has additional metrics and metadata check and pull them from api”

The exercise session type carries a metricsSummary — duration, calories, distance, pace, per-zone durations — and, on runs only, a nested mobilityMetrics block: cadence, stride length, vertical oscillation, ground contact time. Real biomechanics behind a summary screen. 178 sessions since November came back carrying it, each pushed to VM tagged by exercise type and stored as a queryable session resource.

The all-time trend was the story: Nov/Dec runs at 8 km/h @ HR 165, spring runs at 5 km/h @ HR 130. I had dropped all high intensity and ramped easy volume — which is exactly what suppressed HRV.

“build the ecg pipeline, store files on locally for now”

Probing electrocardiogram returned 403 until the ecg scope was added and re-consented. The watch then hands over the raw waveform: single Lead I, 250 Hz, 7500 samples (30 seconds), with a millivoltsScalingFactor to convert. An ecg-export method writes each reading as CSV + JSON metadata; a containerized NeuroKit2 extractor delineates the waveform and pushes QTc/PR/QRS/HRV features back into the same VM stack. Single-lead is screening, not diagnosis — delineation over-reads on one lead — but the waveform is mine, on disk.

From Manual Syncs to a Daily Workflow

The transcript for the next month is dozens of two-word prompts: “sync”, “pull fresh data”, “sync the night and weather”. A ritual that regular is a workflow, so it became one — daily-health, a Swamp DAG of sync → derive → status alongside a weather forecast, ending in a notify step that sends a BLUF, no-emoji Telegram briefing at 11:00 and 22:00. The status method reads everything back from VM through the model’s own query helper — recovery, sleep stages, activity, energy balance, an illness early-warning line — and composes the message as a briefing resource.

Deployment to the Unraid swamp serve container surfaced one real bug: the serve scheduler cannot resolve a model resource’s vault-backed sensitive fields during workflow execution — sync fails with “No tokens” — while the same workflow run from a fresh in-container CLI process succeeds fully. Filed upstream; the workaround is an Unraid cron running docker exec swamp-serve swamp workflow run daily-health. Models execute, workflows orchestrate, and cron — as ever — outlives everything.

The API Grew Up Mid-Build

Built in June against a weeks-old API; by mid-July the ground had shifted. The Google Health API went GA (launched March 24), grew a real reference, a data-types index, a status dashboard, and split every scope into .readonly / .writeonly (ECG and irregular-rhythm remain read-only — telling). The legacy Fitbit Web API got its September 2026 sunset date. The side quest became the main road while I was standing on it.

The docs also explained the last probe mystery. Types answering 400 "List is not supported" — floors, total-calories, calories-in-heart-rate-zone — are rollup-only: fetched via

POST /v4/users/me/dataTypes/{type}/dataPoints:dailyRollUp
{ "range": { "start": {"date": {...}}, "end": {"date": {...}} }, "windowSizeDays": 1 }

with a civil-time range, one aggregated point per local calendar day — which sidesteps the timezone bucketing problem by construction. One semantic worth knowing: for presence-aware types a missing day means the watch wasn’t worn, not zero. The model gained a rollup fetch path plus three more list-able types the index surfaced (altitude, active-minutes, sedentary-period), and the serve deployment now collects all of it on the twice-daily cron.

What the Data Said

Once the pipeline was mine end-to-end — formulas visible, baselines mine — it stopped flattering me. The honest read was mundane: eight hours at a desk with my heart at 68, and a year of runs where I had quietly traded all intensity for easy volume. A graph I own tells me that. A dial I rent kept the number green.

Design Principles That Emerged