Repository navigation
CI never runs the buzz-db replica-routing Postgres fixtures — enforce them before enabling BUZZ_REPLICA_READ_MAX_AGE_MS #3622
Description
Summary
None of the Postgres-backed replica-routing fixtures added in #3268 — nor most of the 34 #[ignore]d tests in crates/buzz-db/src/lib.rs — are ever executed by CI. They were verified exhaustively by hand at dd26caa9f (see the PR body), but a future regression on the routed-read path can merge green.
Found by Dawn during her #3268 sign-off amendment; independently confirmed by Max and Eva at the same SHA.
Evidence (at dd26caa9f)
Unit Testsrunsjust test-unit→cargo nextest run -p buzz-db --lib(Justfile:279-285). Ignored tests are excluded by design, and the Justfile comment says so.- The only
--run-ignored ignored-onlynextest invocations in.github/workflows/areci.yml:690and:702, both filtered to(package(buzz-db) and test(/relay_invite::tests/)) or (package(buzz-relay) and test(/api::invites::tests/)). - Therefore, of the 34
#[ignore]d buzz-db tests, CI runs only therelay_invite::testssubset. The one-budget acquire regression (routed_fallback_spends_one_acquire_budget_when_aurora_cache_is_cold), the seven-seam community-isolation fixture, and all fence/floor-guard/fallback fixtures have zero automated execution.
Why this matters
- The one-budget fixture exists precisely because a "fix is present" claim survived review without a measurement (feat(replica): portable heartbeat-token fence with snapshot-local reader routing #3268 thread, Dawn's Pass 11 postmortem). An
#[ignore]d test no job selects is a comment, not a gate. - Max's rollout condition: do not set
BUZZ_REPLICA_READ_MAX_AGE_MSon bb-block until this is fixed. Merge itself is safe (new seams are dark by default), but enabling the new seams in production without CI enforcement of their regressions is not.
Constraints on the fix
- Ordering dependency on test_usage_metrics_lock uses the production advisory-lock key on the shared test DB — fails whenever a local relay is running #3619: the moment any filter widening selects
test_usage_metrics_lock_has_single_owner_and_releases_on_drop, the shared-DB advisory-lock collision fires in CI. test_usage_metrics_lock uses the production advisory-lock key on the shared test DB — fails whenever a local relay is running #3619 (scratch-DB isolation for that test) must land first, or the selection must explicitly exclude it. - A naive regex is insufficient. Dawn's first-draft filter
test(/tests::(routed|replica|reader|fence|floor_guard)/)misses at least:query_events_routed_*,count_events_routed_*,lazy_reader_pool_*,head_fetch_routes_by_configured_budget,channel_window_routes_*,thread_replies_cursor_pages_*,read_session_degrades_*. A partially-covering filter recreates this bug one level down — green CI, unenforced invariants. - Several fixtures assume scratch DBs and serial execution; the selection should run serially against the existing
backend-integrationPostgres archive (that job already has the infra).
Suggested shape (per Max)
An explicit, auditable selection rather than a clever regex: either move the replica-routing fixtures into a dedicated test module (e.g. tests::replica_routing::) and filter on that module, or maintain an explicit nextest filter list in ci.yml with a comment stating it must cover every #[ignore]d routing fixture. Initially cover the exact routing matrix verified by hand in #3268. Keep test_usage_metrics_lock... excluded until #3619 lands.
Acceptance
- The one-budget fixture and the seven-seam isolation fixture demonstrably run in CI (visible in the job log at least once).
- A revert of only the feat(replica): portable heartbeat-token fence with snapshot-local reader routing #3268 production fix fails CI (the 320ms defang check, now automated).
- No selection of the usage-metrics lock test until test_usage_metrics_lock uses the production advisory-lock key on the shared test DB — fails whenever a local relay is running #3619 is closed.
Activity
tlongwell-block commentedon Jul 29, 2026
Decision update from the #3268 thread — the "suggested shape" above is now the required shape:
Dawn mechanically applied her original regex (
test(/tests::(routed|replica|reader|fence|floor_guard)/), unanchored, nextest semantics) against all 34 ignored tests and intersected with the 5 fixtures this PR adds. Result: 2/5 selected, 3/5 silently missed —One of the misses (
query_events_routed_defaults_dark...) pins the default-dark property the merge-safety argument rests on. A partial regex is worse than no gate — a green badge asserting coverage that isn't there.All five fixtures currently live in the flat
testsmodule (no submodule boundary to select on), so the fix is a small source change plus the filter, not a filter alone:mod replica_routing(insidetests).(package(buzz-db) and test(/replica_routing::/))to the backend-integration ignored-only selection, run serially.test_usage_metrics_lock...out ofreplica_routing(and out of the filter) until test_usage_metrics_lock uses the production advisory-lock key on the shared test DB — fails whenever a local relay is running #3619 lands.Module boundary over name-matching because names drift and the next test author won't know the regex exists; the module is visible at the point of writing.
Consensus in-thread (Dawn, Max, Eva): merge of #3268 stays recommended, but nonzero
BUZZ_REPLICA_READ_MAX_AGE_MSin bb-block is gated on this issue — it's the only step where a regression becomes user-visible staleness instead of a red test.feat(replica): portable heartbeat-token fence with snapshot-local rea…
feat(replica): portable heartbeat-token fence with snapshot-local rea…
feat(replica): portable heartbeat-token fence with snapshot-local rea…