Skip to content

CI never runs the buzz-db replica-routing Postgres fixtures — enforce them before enabling BUZZ_REPLICA_READ_MAX_AGE_MS #3622

Open

Description

@tlongwell-block

Summary

None of the Postgres-backed replica-routing fixtures added in #3268 — nor most of the 34 #[ignore]d tests in crates/buzz-db/src/lib.rs — are ever executed by CI. They were verified exhaustively by hand at dd26caa9f (see the PR body), but a future regression on the routed-read path can merge green.

Found by Dawn during her #3268 sign-off amendment; independently confirmed by Max and Eva at the same SHA.

Evidence (at dd26caa9f)

  • Unit Tests runs just test-unit → cargo nextest run -p buzz-db --lib (Justfile:279-285). Ignored tests are excluded by design, and the Justfile comment says so.
  • The only --run-ignored ignored-only nextest invocations in .github/workflows/ are ci.yml:690 and :702, both filtered to (package(buzz-db) and test(/relay_invite::tests/)) or (package(buzz-relay) and test(/api::invites::tests/)).
  • Therefore, of the 34 #[ignore]d buzz-db tests, CI runs only the relay_invite::tests subset. The one-budget acquire regression (routed_fallback_spends_one_acquire_budget_when_aurora_cache_is_cold), the seven-seam community-isolation fixture, and all fence/floor-guard/fallback fixtures have zero automated execution.

Why this matters

  • The one-budget fixture exists precisely because a "fix is present" claim survived review without a measurement (feat(replica): portable heartbeat-token fence with snapshot-local reader routing #3268 thread, Dawn's Pass 11 postmortem). An #[ignore]d test no job selects is a comment, not a gate.
  • Max's rollout condition: do not set BUZZ_REPLICA_READ_MAX_AGE_MS on bb-block until this is fixed. Merge itself is safe (new seams are dark by default), but enabling the new seams in production without CI enforcement of their regressions is not.

Constraints on the fix

  1. Ordering dependency on test_usage_metrics_lock uses the production advisory-lock key on the shared test DB — fails whenever a local relay is running #3619: the moment any filter widening selects test_usage_metrics_lock_has_single_owner_and_releases_on_drop, the shared-DB advisory-lock collision fires in CI. test_usage_metrics_lock uses the production advisory-lock key on the shared test DB — fails whenever a local relay is running #3619 (scratch-DB isolation for that test) must land first, or the selection must explicitly exclude it.
  2. A naive regex is insufficient. Dawn's first-draft filter test(/tests::(routed|replica|reader|fence|floor_guard)/) misses at least: query_events_routed_*, count_events_routed_*, lazy_reader_pool_*, head_fetch_routes_by_configured_budget, channel_window_routes_*, thread_replies_cursor_pages_*, read_session_degrades_*. A partially-covering filter recreates this bug one level down — green CI, unenforced invariants.
  3. Several fixtures assume scratch DBs and serial execution; the selection should run serially against the existing backend-integration Postgres archive (that job already has the infra).

Suggested shape (per Max)

An explicit, auditable selection rather than a clever regex: either move the replica-routing fixtures into a dedicated test module (e.g. tests::replica_routing::) and filter on that module, or maintain an explicit nextest filter list in ci.yml with a comment stating it must cover every #[ignore]d routing fixture. Initially cover the exact routing matrix verified by hand in #3268. Keep test_usage_metrics_lock... excluded until #3619 lands.

Acceptance

Refs: #3268, #3619.

Activity

  1. tlongwell-block commented on Jul 29, 2026

    @tlongwell-block
    CollaboratorAuthor

    Decision update from the #3268 thread — the "suggested shape" above is now the required shape:

    Dawn mechanically applied her original regex (test(/tests::(routed|replica|reader|fence|floor_guard)/), unanchored, nextest semantics) against all 34 ignored tests and intersected with the 5 fixtures this PR adds. Result: 2/5 selected, 3/5 silently missed —

    MISSED   count_events_routed_is_bounded_only              (lib.rs:7248)
    MISSED   lazy_reader_pool_still_spawns_fence_probe        (lib.rs:7687)
    MISSED   query_events_routed_defaults_dark_and_routes_covered_when_enabled  (lib.rs:7171)
    SELECTED routed_fallback_spends_one_acquire_budget_when_aurora_cache_is_cold
    SELECTED routed_reads_are_confined_to_the_requested_community
    

    One of the misses (query_events_routed_defaults_dark...) pins the default-dark property the merge-safety argument rests on. A partial regex is worse than no gate — a green badge asserting coverage that isn't there.

    All five fixtures currently live in the flat tests module (no submodule boundary to select on), so the fix is a small source change plus the filter, not a filter alone:

    1. Move the replica-routing fixtures into mod replica_routing (inside tests).
    2. Add (package(buzz-db) and test(/replica_routing::/)) to the backend-integration ignored-only selection, run serially.
    3. Keep test_usage_metrics_lock... out of replica_routing (and out of the filter) until test_usage_metrics_lock uses the production advisory-lock key on the shared test DB — fails whenever a local relay is running #3619 lands.

    Module boundary over name-matching because names drift and the next test author won't know the regex exists; the module is visible at the point of writing.

    Consensus in-thread (Dawn, Max, Eva): merge of #3268 stays recommended, but nonzero BUZZ_REPLICA_READ_MAX_AGE_MS in bb-block is gated on this issue — it's the only step where a regression becomes user-visible staleness instead of a red test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      Sponsor
      SponsoredKunjungi sekarang
      Promo