> **Disclaimer:** The following text is an AI-generated summary of the design
> decisions and evolution of the Google Health data pipeline built with Swamp
> and Claude. All work was done iteratively with `claude-code` handling
> implementation while I approved steps and provided direction.

## Choosing the Data Source

The starting prompt was simple: *"how to extract my health metrics from zepp
mobile app, I want to stream them into my victoria metrics."*

I wanted my wearable data in my own time-series database — VictoriaMetrics on
my Unraid box — instead of renting it back through an app. The first candidate
was an old Amazfit paired to the Zepp app. Every path into it failed:

| Path | Method | Result |
|---|---|---|
| Zepp cloud (huami-token) | Xiaomi OAuth | `Missing ssecurity or location in auth response` |
| App traffic capture | http-toolkit + adb MITM | app refuses the TLS cert |
| Local DB off the phone | adb pull | Android app sandbox |
| Legacy Fitbit Web API | new app registration | closed — funnels to Google Health |
| **Google Health API** | **standard Google OAuth + REST** | **works** |

The move that mattered was stopping: *"actually lets pause here for a moment,
how to extract data from pixel watches."* Switching the source beat grinding
the tactic. A `@magistr/fitbit` model was built for the legacy API and deleted
the same day — registration is closed, the API sunsets September 2026. The
Google Health API (`health.googleapis.com/v4`) is the only road forward, and in
June it was weeks old and thin on docs.

Everything became one Swamp extension model — `@magistr/google-health`, with
`authorize`, `exchange`, `probe`, `sync`, and `derive` methods — so every answer
the API ever gave landed as versioned, queryable data instead of scrollback.

## The OAuth Tax

All Google Health scopes are **Restricted**. The consent screen started in
"Testing" mode, and in that mode Google expires the refresh token after ~7
days. For weeks the pipeline demanded a manual re-auth ritual — open consent
URL, approve, paste a `4/0AdkVLP...` code into `exchange`.

The fix was one console click, found late: **publish the OAuth consent screen
to Production**. An unverified single-user app is fine; the refresh token then
persists until revoked. The weekly tax was self-inflicted.

**Lesson:** read the token-expiry rules of the publishing status before
accepting a re-auth ritual as the cost of doing business.

## The DataPoint Shape — Numbers Are Strings

There is no `value` field. Each DataPoint nests its reading under a camelCased
key *named after the data type*, and every number arrives as a string:

```json
{
  "dataSource": { "...": "..." },
  "heartRate": {
    "sampleTime": { "physicalTime": "2026-06-20T07:14:03Z" },
    "beatsPerMinute": "61"
  }
}
```

Where the timestamp lives depends on the type's cardinality:

| Cardinality | Types | Timestamp lives in |
|---|---|---|
| Sample | heart-rate, HRV, weight | `sampleTime.physicalTime` |
| Interval | steps, distance, active-energy | `interval.startTime` / `endTime` |
| Daily | resting-HR, daily-HRV, skin-temp | `date: {year, month, day}` |
| Session | sleep, exercise, ECG | interval + nested summary object |

Two more traps: type ids are kebab-case in URLs but snake_case in filter
params, and units are baked into field names — `distanceMillimeters`,
`weightGrams` — deliberately, to avoid losing precision.

Rather than guess, `sync` uses tolerant extractors (take the one object that
isn't `dataSource`, read the first scalar that isn't time or metadata,
`Number()` it) and stores the first DataPoint of every type as a `raw-<type>`
resource. `swamp data get google-health raw-exercise` shows the exact shape the
API really returns before the mapping gets trusted.

**Lesson:** verify shapes against live data, not docs. The `raw` resources are
the audit trail — I can still query the sample that proved the numbers arrive
as strings.

## Probing a Closed Enum

*"probe api what else there is exist and undocumented and could be usefull"*

At build time there was no index of data types and no list endpoint
(`GET users/me/dataTypes` 404s). The API is a closed enum, so a `probe` method
was added: give it candidate type ids, it fetches a small sample of each and
stores the raw shape. The status code is the answer:

- `200` with points — real, and I have data
- `200` empty — valid, but no device or log for it (blood-glucose, nutrition)
- `403` — real, needs a scope I haven't granted (that's how ECG was found)
- `400 "data type ID not supported"` — not a thing
- `400 "List is not supported"` — rollup-only type, different endpoint entirely

The probe sweep paid off beyond the obvious `heart-rate`/`steps`/`sleep`:
`daily-heart-rate-variability` (deep-sleep RMSSD — the readiness input),
`daily-heart-rate-zones` (my real zone cutoffs: 112/135/163),
`daily-sleep-temperature-derivations` (skin temp with a 30-day baseline already
computed), `active-zone-minutes`, `respiratory-rate-sleep-summary`,
`run-vo2-max`.

The most important probe result was negative: every derived score — readiness,
cardio load, stress, sleep score — returns `400`. Google computes them
client-side and does not expose them. Stress (cEDA "Body Response") has no data
type at all. If I wanted those numbers, I had to rebuild them from raw.

## Backfill and the VictoriaMetrics Gotchas

*"backfill data to nov 2025"*

Pixel Watch heart rate is high-frequency — ~26 samples/minute, about 8 million
HR points over 7 months. Backfill paginates newest-first via `nextPageToken`
until points predate the target date, flushing to VM every 200k lines to stay
under the 64MB ingest limit. A full run takes 20–40 minutes.

Three bugs came out of this phase, none of them where they first appeared:

**Bug 1 — the timezone day-shift.** Daily metrics were bucketed with
`new Date(y, m, d)` — local midnight, which is 22:00 UTC the *previous* day.
Every daily point landed in the wrong UTC day. Fix: `Date.UTC()`. The server
had already normalized the civil date; re-localizing it was the bug.

**Bug 2 — the re-derive no-op.** Recalibrated derivation formulas, re-ran
`derive`, nothing changed. Suspected stale bundles, suspected schema defaults —
both misattributions. A clean repro showed the real cause: **VictoriaMetrics
does not overwrite samples at identical timestamps**. Old midnight-stamped
values simply won. Fix: `derive` now deletes its own output series before
re-pushing. Idempotency required delete-then-write.

**Bug 3 — the query 422.** `query_range` rejects requests where
`(end − start) / step` exceeds 30k points per series. Per-minute HR over months
must be chunked — 14-day windows in `derive`.

One non-bug mattered as much: per-metric history depth genuinely differs.
Phone-sourced metrics reach Nov 2025; watch-only intraday HR starts ~Feb 2026;
intraday SpO2 turned to garbage after the March Pixel Watch feature drop — a
documented Google bug, not mine. The dashboard shows daily SpO2 only.

**Lesson:** don't "fix" missing data. First establish whether it ever existed.

## Reverse-Engineering the Readiness Score

*"now explain all magical constants in your formulas"*

The derived metrics were rebuilt from first principles: Cardio Load as Banister
TRIMP over per-minute HR (only counting HR above the 112 bpm fat-burn floor,
like the app), ACWR as the 7-day/28-day load ratio, Readiness as a composite of
deep-sleep HRV, resting HR, and sleep z-scores against a trailing baseline.

Then came calibration. I supplied 8 days of anchor values read off the app
screen, and the fit collapsed the mystery:

- **Cardio Load** was 3× under-scaled; `loadScale=1.35` matches the app within ±3.
- **Readiness is essentially linear in one input:**
  `readiness ≈ 2.85 × last-night-deep-sleep-RMSSD − 31`, r ≈ 0.97 against the
  app. Sleep duration and resting HR barely move it, despite what Fitbit's own
  docs imply. Six of eight anchor days matched within 1–3 points.

**Lesson:** the proprietary score they won't sell you back is a straight line
through a single measurement you already own.

## Sessions, Running Form, and ECG

*"each training session and run has additional metrics and metadata check and
pull them from api"*

The `exercise` session type carries a `metricsSummary` — duration, calories,
distance, pace, per-zone durations — and, on runs only, a nested
`mobilityMetrics` block: cadence, stride length, vertical oscillation, ground
contact time. Real biomechanics behind a summary screen. 178 sessions since
November came back carrying it, each pushed to VM tagged by exercise type and
stored as a queryable `session` resource.

The all-time trend was the story: Nov/Dec runs at 8 km/h @ HR 165, spring runs
at 5 km/h @ HR 130. I had dropped all high intensity and ramped easy volume —
which is exactly what suppressed HRV.

*"build the ecg pipeline, store files on locally for now"*

Probing `electrocardiogram` returned `403` until the `ecg` scope was added and
re-consented. The watch then hands over the raw waveform: single Lead I, 250
Hz, 7500 samples (30 seconds), with a `millivoltsScalingFactor` to convert. An
`ecg-export` method writes each reading as CSV + JSON metadata; a containerized
[NeuroKit2](https://neuropsychology.github.io/NeuroKit/) extractor delineates
the waveform and pushes QTc/PR/QRS/HRV features back into the same VM stack.
Single-lead is screening, not diagnosis — delineation over-reads on one lead —
but the waveform is mine, on disk.

## From Manual Syncs to a Daily Workflow

The transcript for the next month is dozens of two-word prompts: *"sync"*,
*"pull fresh data"*, *"sync the night and weather"*. A ritual that regular is a
workflow, so it became one — `daily-health`, a Swamp DAG of
`sync → derive → status` alongside a weather `forecast`, ending in a `notify`
step that sends a BLUF, no-emoji Telegram briefing at 11:00 and 22:00. The
`status` method reads everything back from VM through the model's own query
helper — recovery, sleep stages, activity, energy balance, an illness
early-warning line — and composes the message as a `briefing` resource.

Deployment to the Unraid `swamp serve` container surfaced one real bug: the
serve scheduler cannot resolve a model resource's vault-backed sensitive fields
during workflow execution — `sync` fails with "No tokens" — while the same
workflow run from a fresh in-container CLI process succeeds fully. Filed
upstream; the workaround is an Unraid cron running
`docker exec swamp-serve swamp workflow run daily-health`. Models execute,
workflows orchestrate, and cron — as ever — outlives everything.

## The API Grew Up Mid-Build

Built in June against a weeks-old API; by mid-July the ground had shifted. The
Google Health API went GA (launched March 24), grew a real reference, a
data-types index, a status dashboard, and split every scope into `.readonly` /
`.writeonly` (ECG and irregular-rhythm remain read-only — telling). The legacy
Fitbit Web API got its September 2026 sunset date. The side quest became the
main road while I was standing on it.

The docs also explained the last probe mystery. Types answering
`400 "List is not supported"` — `floors`, `total-calories`,
`calories-in-heart-rate-zone` — are **rollup-only**: fetched via

```
POST /v4/users/me/dataTypes/{type}/dataPoints:dailyRollUp
{ "range": { "start": {"date": {...}}, "end": {"date": {...}} }, "windowSizeDays": 1 }
```

with a civil-time range, one aggregated point per local calendar day — which
sidesteps the timezone bucketing problem by construction. One semantic worth
knowing: for presence-aware types a *missing* day means the watch wasn't worn,
not zero. The model gained a rollup fetch path plus three more list-able types
the index surfaced (`altitude`, `active-minutes`, `sedentary-period`), and the
serve deployment now collects all of it on the twice-daily cron.

## What the Data Said

Once the pipeline was mine end-to-end — formulas visible, baselines mine — it
stopped flattering me. The honest read was mundane: eight hours at a desk with
my heart at 68, and a year of runs where I had quietly traded all intensity for
easy volume. A graph I own tells me that. A dial I rent kept the number green.

## Design Principles That Emerged

| Principle | Origin |
|---|---|
| Switch the source, don't grind the tactic | Xiaomi login wall |
| Pull the raw signal first, tighten the mapping later | undocumented DataPoint shapes |
| Store a raw sample of every type as queryable data | `raw-<type>` resources |
| Probe the boundary; read status codes as a map | closed enum, no list endpoint |
| Match the extractor to the type's cardinality | timestamps in four shapes |
| Never re-localize a server-normalized date | UTC day-shift bug |
| Delete-then-push for derived series | VM same-timestamp no-overwrite |
| Chunk range queries | VM 422 at 30k points/series |
| Calibrate derived metrics against ground truth | 8 anchor days → readiness is linear |
| If they won't sell you the number, rebuild it from raw | derived scores all `400` |
| Don't "fix" missing data — check whether it existed | SpO2 feature-drop bug, per-metric depth |
| Publish the OAuth consent screen to Production | 7-day testing-token expiry |
| Models execute, workflows orchestrate | `daily-health` DAG + cron workaround |