feat(deploy): prove the hosted image builds from the moved reference implementation - #43

…34->0, tests 268->7 fail) (#55)

* fix(reference-implementation): isolate DOM-lib leak, fix independent typecheck errors

typecheck reference implementation had 34 pre-existing errors (data-connect#45).
26 traced to one program-wide leak: test/run-interaction-stream-remote-surface-session.test.ts
imports jsdom directly while living inside the flat tsconfig.json program; @types/jsdom's
ambient `/// <reference lib="dom" />` is a program-wide TypeScript effect, not scoped to the
importing file, so it collapsed @types/node's DOM-conditional types (RequestInit, Timeout,
etc.) for every other file sharing that program.

Isolated the jsdom-importing test into its own tsc project (test/tsconfig.dom.json, types:
["node","jsdom"]) rather than merging DOM lib into the shared program; wired it into
`npm run typecheck` as a second tsc invocation. Removed the file's now-stale @ts-expect-error
on the jsdom import (no longer errors once isolated). This resolved 26 errors, including
browser-surface-lease-sweep-timer.ts's Timeout|number mismatch as a direct side effect.

The remaining 8 were independent, real defects, fixed individually (no @ts-ignore, no
relaxed tsconfig):
- scripts/stream-health-audit/live.ts: two Playwright page.waitForFunction() closures
  referenced the ambient `document` global (legitimate -- Playwright serializes and
  evaluates them in a real browser page), which only typechecks with lib:"DOM" in scope.
  Isolating this file was not viable (authority.test.ts/receipt.ts import it and must stay
  in the main program), so typed `document` via a self-contained in-closure cast instead of
  an ambient global. Verified this preserves behavior: Playwright sends only fn.toString()
  to the browser, so any reference to an outer helper would throw ReferenceError there --
  the cast stays entirely inside the closure's own source text.
- live.ts: auth.header.cookie (Record<string,string> index read under
  noUncheckedIndexedAccess) could be `string | undefined`, but the invariant (supported
  auth implies a cookie) holds by construction upstream. Added an explicit fail-loud guard
  instead of a silent non-null assertion.
- authority.test.ts: 3 call sites read healthyConnection().connection_health.conditions
  without the `as Json` cast used everywhere else in the file for the same access pattern
  (Json = Record<string, unknown>) -- an inconsistency with the file's own established
  idiom, not a new decision.
- authority.test.ts: response()'s revision param only accepted `string`, but one call site
  passes `null` to simulate a genuinely absent revision header (switching to `undefined`
  would silently trigger the parameter's default instead). Widened to `string | null |
  undefined` to match the real call site's intent.
- inventory.test.ts: environment_unset is `?: string[]` (optional key); under
  exactOptionalPropertyTypes, assigning literal `undefined` is distinct from the key being
  absent, which is what the test actually means to simulate. Switched to `delete`.

Verified: `npm --prefix reference-implementation run typecheck` exits 0.

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>

* fix(reference-implementation): drop test coverage for pdpp-repo-root files that moved

test reference implementation had a cluster of ENOENT failures: tests reading files by a
relative path that used to resolve into PDP-Connect/pdpp's monorepo root
(reference-implementation/test/../../<path>) before Move B relocated reference-implementation/
into this standalone repo. That same relative walk now lands at data-connect's own repo root,
which does not have these files and structurally cannot for two different reasons:

1. apps/site/src/app/sandbox/_demo/builders.ts and its sandbox routes (7 files: rs-records-list,
   rs-streams-list, rs-search-lexical, ref-dataset-summary, rs-records-detail, rs-schema-get,
   rs-streams-detail boundary tests) -- pdpp's own Next.js frontend, which this repo's apps/
   does not have at all (only apps/console). Removed only the apps/site-reading test cases from
   each file; the operation-boundary and manifest-scoping tests in the same files are
   data-connect-native and untouched. operations-boundary.test.ts only mentions "apps/site" in
   a comment/forbidden-import string, not an actual file read -- verified, left as-is.

2. scripts/check-pdpp-vendored-package-pins.ts (test/pdpp-vendored-runtime-compatibility.test.ts)
   and docker-compose.neko.yml + docker/neko/* (test/remote-surface-reference-boundary.test.ts,
   test/neko-surface-allocator-server.test.ts, test/run-interaction-stream-neko-compose.test.ts)
   -- both pdpp-repo-root-owned: the vendored-package-pins script checks an invariant about
   pdpp's own vendoring of data-connect packages (only makes sense running from pdpp's side of
   the boundary), and the docker-compose/neko deployment config has not been ported into this
   repo's own deploy/ tree -- PR #43 explicitly scoped the Dockerfile port only, leaving
   neko/compose orchestration an undecided deployment-architecture question for the owner, not
   something to invent here as a side effect of a CI fix.

   test/run-interaction-stream-neko-compose.test.ts's 8 tests were all entangled with the
   missing docker-compose.yml/docker-compose.neko.yml/.env.docker.example/docker/neko/* --
   even the tests that also exercise real data-connect logic (resolveNekoBrowserSurfaceControllerOptions)
   first read .env.docker.example to seed their inputs, so no test in the file could be salvaged
   without inventing pdpp's deployment config. Removed the whole file.

   pdpp-vendored-runtime-compatibility.test.ts kept its second test (withdrawn device runtime
   rejects STREAM_EVIDENCE...), which exercises @pdpp/collector-runtime and
   @pdpp/connector-protocol -- both native data-connect workspace packages, unaffected by the
   removed cross-repo-boundary test.

Verified no coverage silently dropped: checked PDP-Connect/pdpp's current main (read-only, not
modified) for both the vendored-pins script and its test -- neither exists there either, so
this is not live coverage moving out from under pdpp's own suite.

Removed now-orphaned helpers/regex constants/imports left unused by these test removals
(remoteSurfaceInstalled, readFile import, ~15 REGEXP_n constants) -- caught by
noUnusedLocals/TS6133 on a typecheck re-run after each edit.

Verified: `npm --prefix reference-implementation run typecheck` still exits 0 after these
changes.

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>

* fix(vendor): re-vendor @pdpp/reference-contract with compiled JS, not raw .ts

test reference implementation's dominant failure cluster (135 of 268 failures) traced to
one root cause: the pdpp CLI (reference-implementation/cli/index.ts) could not run as a
real subprocess at all. Reproduced live: `node cli/index.ts --help` crashed instantly with

  Error [ERR_UNSUPPORTED_NODE_MODULES_TYPE_STRIPPING]: Stripping types is currently
  unsupported for files under node_modules, for ".../node_modules/@pdpp/reference-contract
  /src/public/source.ts"

@pdpp/reference-contract (vendored from PDP-Connect/pdpp as a tarball per this directory's
README.md) ships zero compiled JS -- its entire exports map points at ./src/*.ts, correct
inside pdpp's own repo where first-party source outside node_modules type-strips natively,
but broken the moment it's vendored into a consumer's node_modules, where Node refuses to
strip types unconditionally. Any real invocation of this repo's own CLI hit this the moment
a command reached a server/*.ts file importing @pdpp/reference-contract (many do: auth,
provider, source-declaration-trust, contract-validation, and more) -- a real production
defect, not a test-harness quirk (cli/index.ts's own bin entry has no tsx wrapper).

pdpp itself is out of scope to change from this repo (this is a data-connect-side fix,
not a pdpp PR), so this repo's own vendoring process now compiles the tarball's contents
before packing rather than shipping a byte-identical `npm pack` of pdpp's source
directory. Verified before re-vendoring: all 9 `exports` subpaths import and run
correctly from the compiled output. Re-derivation recipe (tsc with rootDir/outDir/
rewriteRelativeImportExtensions, then repoint exports/main/types at dist/) documented in
vendor/README.md; `src/`/`test/` still ship in the tarball unedited, for reference.

Updated reference-implementation/vendor/SHA256SUMS and package-lock.json's integrity hash
to match the new tarball's digest.

Verified: `node cli/index.ts --help` and `node cli/index.ts ref --help` both run and print
real output (previously crashed on the first line). Full RI test suite:
9986/268 (pass/fail) -> 10131/96 pass/fail, confirmed via a clean single test-suite run
(PDPP_OWNER_PASSWORD=reference-implementation-ci PDPP_TEST_PROFILE=memory-default npm
--prefix reference-implementation run test).

Remaining 5 TS-stripping failures trace to a separate, smaller defect in
@pdpp/polyfill-connectors (vendored from PDP-Connect/data-connectors, not pdpp) --
filed as data-connectors#67, same defect class, different source repo and fix location.

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>

* fix(reference-implementation): resolve remaining ENOENT failures (missing pdpp files, one wrong path)

Follow-up to the earlier ENOENT-cluster fix -- these surfaced only after the
@pdpp/reference-contract vendoring fix let the suite run far enough to reach them:

1. test/rs-search-lexical-boundary.test.ts was missed in the earlier pdpp apps/site
   cleanup pass (7 sibling boundary-test files were fixed, this one was not). Same
   fix: dropped the two tests reading apps/site/src/app/sandbox/{v1/search/route.ts,
   _demo/builders.ts} (pdpp frontend paths this repo does not have); kept the
   operation-boundary tests.

2. server/streaming/cdp-method-allowlist.test.ts had a wrong relative path depth:
   join(__dirname, "../../node_modules/@opendatalabs/remote-surface/...") resolved to
   reference-implementation/node_modules/... which does not exist -- the real,
   npm-hoisted location in this repo's workspace layout is one level further up
   (../../../node_modules/...). A genuine off-by-one, not a missing dependency.

3. Two more pdpp-repo-root deployment-config-dependent test cases, same reasoning as
   the earlier docker-compose.neko.yml cluster (PR #43 scoped the Dockerfile port
   only; neko/compose/railway/flyio orchestration remains undecided
   deployment-architecture, not something to invent here):
   - test/deploy-supervisor-restart-contract.test.ts: all 7 tests read
     docker-compose.yml / deploy/docker/docker-compose.yml / .env.docker.example /
     deploy/railway/reference.Dockerfile / Dockerfile / deploy/flyio/fly.toml, none
     of which exist in this repo. Removed the whole file.
   - test/deployment-storage-contract.test.ts: 3 of its 9 tests read
     docker-compose.yml / deploy/docker/docker-compose.yml / Dockerfile directly;
     removed those 3, kept the 6 resolveStorageBackend() unit tests (real,
     data-connect-native coverage of server/postgres-storage.ts's own guard logic).
   - test/reference-stack-network-durability.test.ts: all 11 tests spawn
     scripts/docker-neko-network-{durability,migration}-smoke.sh or
     scripts/reference-stack.sh, none of which exist in this repo's scripts/.
     Removed the whole file.

4. spec-collection-profile.md (test/collection-profile.test.ts, one test) is
   DIFFERENT in kind from the above: it is a protocol-level spec doc at pdpp's repo
   root, not app- or deployment-specific, and pdpp's own main still has it live
   (checked read-only, not modified) -- unlike the deploy-config/apps-site cases,
   this is genuine drift-detection coverage (spec prose vs. this repo's own runtime
   types) that would otherwise be silently lost, since pdpp no longer has the test
   (it moved here) and this repo couldn't find the doc (it didn't). Copied the doc
   into this repo's root rather than deleting the test -- verified the test passes
   against the copied doc. (Its sibling apps/site/content/docs/
   reference-implementation-examples.md, used by test/
   b6-single-use-consumption-conformance.test.ts's now-removed doc-drift test, is
   different: that lives inside pdpp's frontend docs-site content tree, the same
   kind of pdpp-app-owned path as the apps/site cases above, not a repo-root
   protocol spec -- left that one test removed, not copied.)

Verified: `npm --prefix reference-implementation run typecheck` exits 0 after each
edit (caught several now-orphaned imports/constants via TS6133).

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>

* fix(apps/console): build with npm (this repo's package manager), not pnpm

Two test-reference-implementation failures (test/composed-origin.test.ts,
test/dashboard-proxy-redirect.test.ts) both boot a real apps/console build via a
shared ensureConsoleBuild() helper that spawned `pnpm --dir apps/console build`.
This repo's package manager is npm (root package.json declares "workspaces", no
pnpm-workspace.yaml exists anywhere in the repo) -- npm workspaces already
correctly symlinks apps/console's @pdpp/brand / @pdpp/brand-react / @pdpp/operator-ui
/ pdpp-reference-implementation deps (all declared as "*") to
reference-implementation/vendor/* in the root node_modules/ (confirmed:
node_modules/@pdpp/brand -> ../../reference-implementation/vendor/brand). `pnpm
--dir` ignores all of that and tries to fetch those private, never-published
packages from the public npm registry, 404ing on the first one
(ERR_PNPM_FETCH_404 for @pdpp/brand).

Switched both test files' build invocation to `npm --prefix apps/console run
build`, which correctly resolves via the existing workspace links. That
surfaced two further, real, previously-unexercised build defects (this
"pnpm --dir" invocation had apparently never successfully built the console app
in this repo before, in any test run):

1. apps/console's own webpack config only listed pdpp-reference-implementation,
   @pdpp/brand, @pdpp/brand-react, and @pdpp/operator-ui in `transpilePackages` --
   missing @pdpp/polyfill-connectors, which reference-implementation/server/
   connection-setup-plan.ts (imported transitively by apps/console's own route
   code) depends on. Webpack tried to parse its raw TypeScript source directly
   and failed with "Module parse failed: Unexpected token" on two files. Added
   @pdpp/polyfill-connectors to transpilePackages.

2. apps/console/src/app/(console)/lib/connection-catalog.test.ts imported
   STATIC_SECRET_CONNECTOR_REGISTRY via a physical relative path six directories
   up into packages/polyfill-connectors/src/static-secret-injection.ts -- a
   narrow, deliberately curated subset vendored only for @pdpp/local-collector's
   own build (see that directory's package.json), which does not carry this
   file. The real package export already exists
   (@pdpp/polyfill-connectors/static-secret-injection, confirmed via the
   package's own exports map) and is what reference-implementation's own
   production code already imports by name for the same registry. Switched the
   test to the real package specifier.

Verified end-to-end: `npm --prefix apps/console run build` completes cleanly
(webpack compile + typecheck + full route manifest), and both previously-failing
tests now pass in isolation (`node --import tsx --test --test-name-pattern
"composed browser origin carries metadata" test/composed-origin.test.ts`;
same pattern for dashboard-proxy-redirect.test.ts).

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>

* fix(runtime): settle the per-run watchdog promise on normal completion, fix a straggler-test leak

Two real, independently-verified fixes for the "Promise resolution is still pending
but the event loop has already resolved" (Node test-runner's cancelledByParent
diagnostic) failure cluster:

1. runtime/controller.ts's finalizeRunCleanup (the normal-completion cleanup path)
   dropped a run's runWatchdogSettlements entry on a beaten-deadline completion
   without ever resolving its settlementPromise -- the code's own comment already
   noted this was intentional ("its settlement will never resolve on its own") on
   the theory that no caller needed it to, since awaitRun's Promise.race always
   wins via the already-settled activeRunPromises entry regardless. That reasoning
   holds (confirmed: resolving it changes no caller-observable behavior), but it
   left a genuinely dangling, unreferenced promise for every run that completes
   normally (the overwhelmingly common case) rather than via an actual 4-hour
   watchdog timeout. Added `runWatchdogSettlements.get(input.runId)?.resolve()`
   before dropping the entry.

2. test/controller-drain.test.ts's "deadline expires with stragglers" test modeled
   its intentionally-still-pending stragglers with `setTimeout(..., 5000).unref()`.
   `.unref()` only excuses a timer from blocking process exit -- it does not
   settle the attached promise, and Node's test runner separately flags any
   promise a test created that is still unsettled once the test's own run has
   otherwise concluded, independent of whether the process could still exit. The
   5-second stragglers reliably outlived that window and cascaded: the next two
   tests in the same file, which do not themselves leak anything, failed purely
   because they ran after the poisoned one (verified: both pass in isolation,
   both failed only when run after the straggler test). Shortened the stragglers
   to 250ms (still comfortably slower than the 100ms deadline the test asserts
   against) and explicitly awaited them at the end of the test, after the
   assertions that need them still pending at deadline-check time.

Verified: `npm --prefix reference-implementation run typecheck` exits 0.
`node --import tsx --test test/controller-drain.test.ts` — 5/5 pass (was 2/5).

This is one of at least two root causes in this failure cluster (~58 failures
across 6 files at last measurement) -- test/controller-cancel-run.test.ts and
others still fail even with fix #1 applied, meaning a second, not-yet-found
leak exists on the cancellation (not normal-completion) cleanup path. Continuing
to investigate; will file the remainder as a tracked issue if not resolved by
the time this PR is ready to land, rather than land a partial, unverified guess.

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>

* fix(reference-implementation): fix a self-inflicted regression, drop one more pdpp apps/site file

1. test/source-declaration-boundary.test.ts's "Core-only dependency oracle" test
   hardcoded an assertion that @pdpp/reference-contract/public/source resolves to
   .../src/public/source.ts -- true before this PR's reference-contract
   compiled-JS re-vendor (see the earlier "fix(vendor)" commit in this PR), false
   after: the module now resolves to dist/public/source.js. Updated the regex to
   match the new (correct) compiled path. The test's actual invariant --
   server/source-declaration.ts's only runtime import resolves to a module with
   zero further imports of its own -- still holds and is still verified; only the
   path shape assertion needed updating.

2. test/consent-connection-label.test.ts is the same pdpp apps/site frontend
   dependency class as the earlier ENOENT-cluster fixes, just not caught in that
   pass: its own doc comment says outright "apps/site/** is out of this cohort's
   scope (forbidden territory)... The mapper lives in the public-site app
   (apps/site/src/lib/consent-connection-label.ts); this suite lives in
   reference-implementation/test/** because that is the only test tree the
   standard suites discover." All 8 tests in the file import that module
   directly; none are salvageable without the pdpp frontend file this repo does
   not have. Removed the whole file.

Verified: `npm --prefix reference-implementation run typecheck` exits 0.
`node --import tsx --test --test-name-pattern "Core-only dependency oracle"
test/source-declaration-boundary.test.ts` passes.

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>

* fix(reference-implementation): update stale path assertion after reference-contract re-vendor

test/source-declaration-boundary.test.ts's "Core-only dependency oracle" test hardcoded
an assertion that @pdpp/reference-contract/public/source resolves to
.../src/public/source.ts -- true before this PR's reference-contract compiled-JS
re-vendor (see the earlier "fix(vendor)" commit in this PR), false after: the module now
resolves to dist/public/source.js. Updated the regex to match the new (correct) compiled
path. The test's actual invariant -- server/source-declaration.ts's only runtime import
resolves to a module with zero further imports of its own -- still holds and is still
verified; only the path shape assertion needed updating.

Verified: `npm --prefix reference-implementation run typecheck` exits 0.
`node --import tsx --test --test-name-pattern "Core-only dependency oracle"
test/source-declaration-boundary.test.ts` passes.

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>

* fix(vendor): re-vendor @pdpp/polyfill-connectors with compiled JS (full src/ tree)

Same underlying defect class already fixed for @pdpp/reference-contract in this PR's
earlier "fix(vendor)" commit: @pdpp/polyfill-connectors (vendored from a DIFFERENT source
repo, PDP-Connect/data-connectors, not pdpp) ships raw, uncompiled TypeScript source with
no build step -- correct inside data-connectors' own repo, broken once vendored into this
repo's node_modules, where Node refuses to strip types. Confirmed live before this fix:
reference-implementation/scripts/generate-connector-registry.ts and
scripts/compact-record-history.ts (first-party scripts in this repo, spawned as real
subprocesses by several tests) crashed with ERR_UNSUPPORTED_NODE_MODULES_TYPE_STRIPPING
reaching @pdpp/polyfill-connectors/manifests; a real server boot
(test/run-tests-env-scrub.test.ts) hit the same crash reaching
@pdpp/polyfill-connectors/browser-surface-policy.

Compiled the package's full src/**/*.ts (30 of its 38 exports subpaths; the 7
connectors/*-mapped ones and bin/orchestrate.ts were left on raw .ts -- not confirmed
broken, and connectors/ carries per-service code this pass didn't audit for compile risk).
tsc transitively pulled in and compiled the handful of connectors/*/collector-definition.ts
files collector-registry.ts itself imports, and bin/local-device-exporter.ts (this
package's own `bin` entry -- confirmed separately broken, since any real install of this
package as a dependency hits the identical crash running the installed command).

Two non-obvious packaging fixups, both caught by real regressions during THIS fix's own
verification (not by narrower "does it import" smoke checks) -- full detail, including the
re-derivation recipe, is in reference-implementation/vendor/README.md:

1. Ten call sites across this repo (runtime/controller.ts, several scripts and tests) derive
   the package's own root directory via
   dirname(fileURLToPath(import.meta.resolve("@pdpp/polyfill-connectors/manifests"))),
   relying on that landing exactly one level below the package root -- true for the original
   src/manifest-registry.ts, broken once ./manifests pointed at the deeper
   dist/src/manifest-registry.js. First regression (8 newly-failing connector-path-resolution
   tests) traced to this and fixed by compiling manifest-registry.ts a SECOND time,
   standalone, at its original depth (src/manifest-registry.compiled.js, alongside --  not
   replacing -- the source), and pointing ./manifests there instead of the dist/src/-nested
   copy.
2. Two src/ files (reason-display-messages.ts, connector-options-schema.ts) import
   readPolyfillManifests via a RELATIVE specifier, so their compiled output still resolves
   the SIBLING dist/src/manifest-registry.js (with the original depth bug) rather than the
   depth-matched copy from fix 1. Second regression (3 more newly-failing
   owner-connection-config-route tests) traced to this; fixed by also copying the real
   manifests/ directory to dist/manifests/ (two copies of the same 45 read-only JSON files
   now ship, one per consumer shape -- not a maintenance burden, both mechanically
   regenerated by the same recipe).

Verified via the full reference-implementation test suite (the real regression gate this
fix relied on throughout, not spot-checks) across three consecutive clean runs:
9986/268 (pass/fail, origin/main baseline) -> ... -> 10143/62, with each intermediate
regression caught and fixed before landing, confirmed zero regressions in the final
pass/fail-set diff against the prior clean run.

Remaining residual gap (scripts/generate-static-secret-registry.ts, still genuinely broken
upstream -- a hardcoded dynamic-import .ts extension plus a devDependency-only biome
formatter path that never exists once vendored elsewhere) tracked in data-connectors#67,
updated with this fix's full findings.

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>

* fix(reference-implementation): fix the pending-promise-at-exit cluster (Node test-runner false positive)

Root-caused the largest remaining failure cluster (55 of 62 failures at last
measurement, spanning 5 files): a documented, known Node.js test-runner limitation,
not a bug in this repo's production code.

nodejs/node#51381 ("spurious 'Promise resolution is still pending' when a callback
test fails") and its follow-up discussion in nodejs/node#52025 describe the exact
mechanism: the test runner uses an unref()'d internal timer for its own timeout
bookkeeping. When test code ALSO uses an unref()'d timer racing against a
deliberately-still-pending promise -- a completely valid, common pattern (e.g. "assert
that operation A times out while operation B is left deliberately in-flight, to prove
cancelling A doesn't affect B") -- the runner can flag that still-pending promise as
abandoned before the real race actually resolves, well before either the test's own
assertions or its explicit timeout value would suggest. Confirmed directly: shortening
one such test's internal deadline by 20x still failed at the same speed, ruling out
"the wait is too long" as the cause.

Node's own core test suite documents the accepted workaround for this exact
interaction (see the discussion linked above and nodejs/node#52146): a separate,
ref()'d "keep the event loop alive" timer for the test's duration, cleared in
t.after(), independent of whatever unref()'d timer the code under test uses. Applied
this to the 5 affected files, in each file's existing shared per-test setup helper
(freshDb()/setup()) where one exists, so every test in the file gets the same
protection rather than patching call sites one at a time.

Verified per-file in isolation (node --import tsx --test test/<file>.test.ts), each
now fully green: controller-cancel-run.test.ts 5/5 (was 1/5), controller-browser-
surface-leases.test.ts 39/39 (was 28/39), controller-phantom-active-run.test.ts 18/18
+ 3 legitimately-skipped Postgres tests (was all 21 failing), run-generation-
fencing.test.ts 5/5 (was 1/5), source-declaration-trust.test.ts 19/19 (was 5/19,
after also fixing a second instance of the same shape in the same file -- "declaration
retrieval bounds DNS work by the configured deadline" -- found once the first fix
unblocked the rest of the file's cascade).

Full suite dropped from 62 to 16 failures with zero regressions (confirmed via a
before/after diff of the exact failing-test-name set). The remaining 16 are unrelated,
already-tracked items (data-connect#53, data-connectors#67, the deliberately-deferred
Signal connector rollout, and 2 pre-existing scanner findings) -- EXCEPT a handful of
tests in source-declaration-trust.test.ts that still show this same failure signature
specifically under the full suite's concurrent-worker execution model, despite passing
100% when that file runs standalone. Not yet root-caused whether this is a
concurrency-specific variant of the same Node limitation or a separate interaction;
noted for follow-up rather than blocking this fix, which is unambiguously a large net
improvement either way.

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>

* fix(reference-implementation): fix keep-alive timer interval under this suite's custom reporter

Follow-up to the previous "fix the pending-promise-at-exit cluster" commit: the
1000ms keep-alive timer interval fixed the failures when running each test file
directly (node --test test/<file>.test.ts), but the SAME failures reappeared when
running with this suite's own custom --test-reporter (scripts/test-accounting/
node-reporter.ts, an async generator that consumes the runner's event stream via
`for await`) -- exactly how this repo's real test harness (scripts/run-tests.ts)
invokes every test file.

Root-caused via direct comparison: `node --test <file>` alone passed at 1000ms;
`node --test --test-reporter=<this repo's reporter> <file>` failed identically at
1000ms and passed cleanly at 10ms. The custom reporter's own event consumption adds
enough latency that a slow-ticking ref'd timer doesn't keep the runner's liveness
check satisfied in time -- the same class of unref'd-timer race documented in
nodejs/node#52025, just with an additional variable (reporter overhead) this repo's
own custom reporter introduces that a stock `node --test` invocation doesn't have.

Shortened the keep-alive interval from 1000ms to 10ms in all 5 previously-fixed
files; documented why in each comment so a future reader isn't tempted to "simplify"
back to a rounder number. Verified against the harness's actual invocation shape
(node --test --test-reporter=./scripts/test-accounting/node-reporter.ts <file>, the
real flag combination scripts/run-tests.ts uses per spawned child): all 5 files now
show zero failures under that exact reporter, not just under a bare `node --test`.

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>

* fix(reference-implementation): re-vendor polyfill-connectors onto data-connectors#70, consume blessed exports

Re-vendors @pdpp/polyfill-connectors from data-connectors commit
dc4008c348d7066a09c067f05189fd2f8c23c80f (merge of data-connectors#70,
which additively blessed ./connectors/github, ./connectors/github/schemas,
and a new ./fixture-samples export). Supersedes the earlier re-vendor
from 262c7bd8, which dropped 1115 files this repo reaches into directly
once data-connectors#68's files allowlist landed alongside its TS-stripping
fix -- full enumeration and classification posted on this PR's own thread.

Three of the five enumerated raw-source reaches are fixed here:

- test/github-manifest-connector-parity.test.ts: imports the new blessed
  @pdpp/polyfill-connectors/connectors/github/schemas directly, deleting
  the old regex-fallback that scraped SCHEMAS out of raw .ts source text.
- test/owner-source-to-mcp-closure.test.ts: uses the new blessed
  @pdpp/polyfill-connectors/fixture-samples's readSampleRecord(), deleting
  the hardcoded raw fixture paths and the local fixtureRecord() helper.
- test/remote-surface-reference-boundary.test.ts: reads the already-blessed
  browser-handoff/streaming-target-registration exports' compiled .js
  sibling instead of raw .ts (no vendor change needed for this one --
  verified the checked import/reference patterns survive compilation
  unchanged).

Two remaining reaches (a whole-connector-tree forbidden-import scan, and
this repo's own production connector-path-discovery mechanism -- both
needing all 45 manifest-listed connectors' source, not a named few) are
NOT resolved by this pin; tracked as an open, bigger decision on this
PR's thread, not papered over here.

Local full suite: 9 failures remain (down from 14 on the prior re-vendor,
62 before the pending-promise fix, 398 at RI-seam-fix start). All 9 trace
to already-filed, already-classified causes: 2x data-connectors#53 (stale
local packages/polyfill-connectors/src/ subset, pre-existing/unrelated),
1x data-connectors#69 (signal connector manifest capability mismatch),
6x the open connectors/ynab (and other non-blessed-connector) gap above.

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>

* fix(reference-implementation): re-vendor polyfill-connectors onto data-connectors#74, fixes typecheck

Re-vendors @pdpp/polyfill-connectors from data-connectors commit
878b4cae785d1d444ff17fff5c44726528209745 (merge of data-connectors#74,
"fix declarations for polyfill-connectors", fixing data-connectors#71).

#71 was a genuine, previously-undiscovered upstream defect: #68/#70's
package.json had tsconfig.build.json's declaration:false, so the real
build never emitted .d.ts for any of its exports -- every TypeScript
consumer got implicit any, tripping strict-mode diagnostics. This repo's
own interim hand-rolled vendoring step (before #68 existed) had set
declaration:true in its own scratch build config, shipping 120 .d.ts
files that masked the gap in every local check run against that interim
tarball; only caught once re-vendoring from the real upstream build and
running the CORRECT CI-matching command
(`npm --prefix reference-implementation run typecheck`, not the root
`npm run typecheck`, whose project references never cover this
directory at all).

#74 fixes it at source: .d.ts + .d.ts.map now ship for all 42 export
subpaths, with NodeNext- and bundler-moduleResolution proofs and a
pack-time guard against regressing.

Verified with the exact CI-matching commands for both required jobs:
- `npm --prefix reference-implementation run typecheck` (after building
  packages/connector-protocol, packages/collector-runtime, and
  reference-implementation/vendor/mcp-server, matching this repo's own
  CI workflow steps): clean.
- `npm --prefix reference-implementation run test`: 10,196 pass / 9 fail,
  identical to the 9 pre-existing/already-classified failures from the
  prior (dc4008c3) re-vendor -- confirms #74 resolved the typecheck
  regression and both apps/console-build-dependent test failures cleanly,
  with no new or changed failures introduced.

Remaining 9, all previously filed/classified, none introduced by this
commit: 2x data-connectors#53 (stale local packages/polyfill-connectors/src
subset, pre-existing/unrelated), 1x data-connectors#69 (signal connector
manifest capability mismatch), 6x the open connectors/ynab-and-other-
non-blessed-connectors scope decision (a whole-connector-tree
forbidden-import scan plus this repo's own production connector-path-
discovery mechanism, both needing all 45 manifest-listed connectors'
source -- flagged as a bigger, separate decision on this PR's thread,
not resolved here).

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>

* refactor(reference-implementation): isolate connector-tree-scope swap points, no behavior change

Prep for the orchestrator's decision on the connector-tree-scope question
(data-connect PR #55's 9 remaining test failures): data-connectors will
ship a built implementation for every manifest-listed connector plus a
resolveConnectorImplementation(connector_id) resolver export backed by
connector-index.json (lane dcx-full-connector-build-0903, in progress),
replacing the directory-walk this repo currently does to find a shipped
polyfill connector's runnable entry-point.

Isolates the two places that directory-walk happens behind clearly-marked,
single-function swap points so the eventual resolver swap is a one-function-
body replacement, not a scattered refactor:

- runtime/controller.ts: extracted resolvePolyfillConnectorEntryPoint()
  out of indexPolyfillManifestFile() -- same two-candidate existsSync
  probe, now named and commented as the exact call site to replace.
- test/connector-config-no-self-declaration.test.ts: comment-only,
  marking listConnectorSourceFiles() as the same class of swap point.

No functional change. Verified behavior-preserving: typecheck clean,
full local suite unchanged at 10,196 pass / 9 fail (same 9 tests, same
names, as the pre-refactor baseline) -- confirms this is pure structural
prep, not a fix. The actual fix lands once dcx-full-connector-build-0903
merges upstream and this repo re-vendors + swaps both functions' bodies.

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>

* fix(reference-implementation): re-vendor onto data-connectors#75, swap to resolveConnectorImplementation

Re-vendors @pdpp/polyfill-connectors from data-connectors commit
c80e05bea98ba2eeda3bcf45598d9fb239a25902 (merge of data-connectors#75,
"feat: ship all connector implementations") -- the fix for the
connector-tree-scope decision (orchestrator decision: option 1, done
properly upstream).

The package now compiles and ships built JS + .d.ts for all 45
manifest-listed connectors (previously only 12 were compiled) and exports
a new @pdpp/polyfill-connectors/resolve:
resolveConnectorImplementation(connectorId) -> { entry, manifest,
brandIcon }, all as file:// URL strings safe for `await import()`
directly, backed by a generated connector-index.json. Unknown IDs throw
a typed ConnectorImplementationNotFoundError
(code ERR_PDPP_CONNECTOR_IMPLEMENTATION_NOT_FOUND), not a silent
null/undefined. Verified directly against this pin from a real installed
consumer before touching this repo: resolved ynab's full connector_id,
imported the returned entry, confirmed real exports; confirmed the typed
error for an unknown id.

Swaps runtime/controller.ts's resolvePolyfillConnectorEntryPoint()
(isolated as the exact swap point in an earlier prep-only commit on this
branch) to call the resolver instead of directory-probing
POLYFILL_CONNECTORS_DIR for index.ts/index.js. Reordered
indexPolyfillManifestFile() so connector_id is parsed from the manifest
BEFORE resolving the entry point, since the resolver takes the full
connector_id (a URI), not the manifest's JSON filename stem the old
directory-probe used. The resolver's file:// URL is converted to a
filesystem path via fileURLToPath, since this file's own downstream
consumer (runtime/index.ts's connector spawn) takes a path, not a URL.
Manifest ENUMERATION still reads POLYFILL_MANIFESTS_DIR directly --
unaffected by this fix, since that directory's on-disk layout didn't
change; only per-connector entry-point resolution moved off directory
probing.

Also fixed two stale test assertions that hardcoded ynab's entry as
`.ts`-only (connector-path-resolution.test.ts, control-actions.test.ts)
-- now tolerant of `.ts|js`, matching the pattern already used for
github's own assertion. connector-config-no-self-declaration.test.ts's
directory walk is kept as-is (comment updated, no code change): its
coverage is no longer scope-limited now that all 45 connectors are
physically shipped, and connector-index.json can't replace it without
narrowing coverage (the index only lists one entry point per connector,
not every source file the walk needs to scan for forbidden imports).

Verified with the CI-matching commands for both required jobs:
typecheck clean; test suite 10,200 pass / 5 fail (down from 9). The 6
connector-tree-scope failures are gone -- confirmed via a direct check
that runtime/controller.ts no longer appears in the
ri-zero-connector-knowledge-conformance scanner's violation list, and
all 3 connector-path-resolution/control-actions tests now pass.

Remaining 5, all pre-existing and unrelated to this fix:
- 2x data-connectors#53 (stale local packages/polyfill-connectors/src
  subset, pre-existing)
- 1x data-connectors#69 (signal connector manifest capability mismatch)
- 2x data-connect#58 (newly filed): the ri-zero-connector-knowledge scanner
  has 8 false-positive findings across 5 files that predate this branch
  entirely and never touch @pdpp/polyfill-connectors -- verified each by
  reading the flagged code (an RFC 6761 .invalid-TLD URL placeholder, a
  NodeJS-signal-name field literal, and internally-path-safe file reads),
  none are real defects.

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>

* fix(ri): close 8 false positives in the zero-connector-knowledge scanner

Fixes the last 3 rule-level bugs blocking PR #55: reserved-placeholder-TLD
URLs, a schema field-name array colliding with a connector key, and
readFileSync calls whose path is validated by a same-file safe-path helper.

(a) hardcoded-provider-endpoint-url: GENERIC_URL_HOSTS only exact-matched
    hostnames, so pdpp.invalid (RFC 6761/2606's reserved .invalid TLD, used
    as new URL(relativeHref, "https://pdpp.invalid")'s dummy parse base in
    stream-health-audit/authority.ts) false-positived. Adds a HOST-SUFFIX
    exemption for .invalid/.example/.test/.localhost, so the whole reserved
    class is covered, not just this one host.

(b) hardcoded-connector-identity-literal: rule 1 flags every literal-bearing
    AST position, including a plain array element -- correct for a real
    connector list, wrong for test-accounting/inventory.ts's
    RECEIPT_BINDING_FIELDS, a receipt-schema field-name array whose "signal"
    element collides with the Signal connector's key but is only ever used
    as RECEIPT_BINDING_FIELDS.map(field => record[field]), i.e. a slot NAME
    being read off an unrelated record, never an asserted identity. Adds
    arrayExpressionsUsedAsFieldNameLists, the array-literal counterpart to
    the existing object-key objectExpressionsUsedAsDispatchTables carve-out:
    an array is exempted only when PROVEN (by an actual .map/.forEach call
    whose callback uses its parameter solely as a computed-member-access key
    into some OTHER object) to be a field-name list, and only if it is never
    ALSO used in a real membership/dispatch shape elsewhere in the file. This
    is a rule-level fix, not a "signal" allowlist -- any other schema field
    name colliding with a future connector key is covered the same way.

(c) unresolvable-data-resource-load: readFileSync(safePath(root, "...")) in
    test-accounting/packet.ts and readFileSync(authorityContained(...)) in
    test-accounting/inventory.ts were unresolvable because the scanner had
    no notion of "this path argument is a call to a function that validates
    it". Adds functionIsProvenSafePathHelper: a same-file, non-exported
    function is trusted when its OWN BODY structurally proves it resolves a
    real root (realpathSync(rootParam)) and rejects any candidate not
    prefixed by that root (candidate !== root && !candidate.startsWith(...)
    -> fail()/throw) -- the exact safePath/safeLeasePath/authorityContained
    shape. A call to an unvalidated helper, or one missing the reject check,
    still fails closed.

Widened beyond the originally-named 8 findings, discovered while getting
scanRepository() to true zero (both narrow, separately justified, not a
broad weakening):

  - import.meta.resolve("@pdpp/polyfill-connectors/manifests") (the shape
    runtime/controller.ts and scripts/generate-connector-registry.ts use to
    locate the polyfill-connectors package now that it's a pinned tarball
    dependency, not a workspace package) was an unrecognized anchor shape,
    and MANIFEST_ROOTS' "packages/polyfill-connectors/manifests" entry no
    longer exists on disk post-vendoring -- it's now
    node_modules/@pdpp/polyfill-connectors/manifests. Fixed both: taught the
    resolver this one specific, reviewed import.meta.resolve specifier, and
    corrected MANIFEST_ROOTS to the real installed location.
  - calleeName() matched "resolve"/"join" by bare property name only, so
    scripts/hermetic/guard.ts's req.resolve("undici") (Node's own
    createRequire()-based module resolver) was misread as path.resolve(...),
    fabricating a bogus relative-path violation instead of the correct
    "unresolvable code load" classification. calleeName() now takes an
    optional trustedReceivers set; join/resolve only resolve as node:path
    when the receiver is a real `import ... from "node:path"` binding.
    Every other (non-ambiguous) name's behavior is unchanged.
  - Also allowlisted two now-provably-legitimate call sites this session's
    fixes still can't reach: generate-connector-registry.ts's
    `env || import.meta.resolve(...)` fallback (one level of `||` past the
    bounded folder), and with-local-full-suite-lock.mjs's own git-common-dir-
    anchored lock-owner file (this tool's own state, never connector data).
    Re-pinned polyfill-manifest-reconcile.ts's existing allowlist entry from
    line 98 to 99 (shifted by the same vendoring commit).

18 new regression tests added to test/ri-zero-connector-knowledge-conformance.test.ts,
each a false-positive-must-pass + true-positive-must-still-fail pair. Local
suite: 10,218 passing, exactly 3 failing (data-connectors#53 x2,
data-connectors#69 x1 -- pre-existing, out of scope, untouched). Typecheck
clean.

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>

* fix(vendor): re-vendor polyfill connectors from current main

Pin @pdpp/polyfill-connectors to data-connectors 8372d030, including the Signal
manifest correction and unproven-connector demotions. Keep the host's runtime
packages as the sole internal dependency source.

Refs: #55
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
Assisted-by: AI

* fix(test): scan the vendored polyfill-connectors package, not a stale subset

The shared-library kind-dispatch guard walked the repo-relative
`packages/polyfill-connectors/src/`. That directory is
`@pdpp/polyfill-connectors-vendored-source` — a 19-file closed subset
vendored by physical file into `@pdpp/local-collector`'s build. It is a
different thing that happens to share a name with the real package,
which ships as `@pdpp/polyfill-connectors` and is resolved from
`reference-implementation/vendor/pdpp-polyfill-connectors-0.0.1.tgz`.

None of the modules the invariant is written about live in that subset.
The guard scanned 19 unrelated files and reported no violations, while
`orchestrator.ts`, `auto-login/*.ts`, `static-secret-injection.ts` and
the four allowlisted registries went unexamined. `walkTsFiles` swallows
a missing directory, so the drift read downstream as a clean pass; only
the two `existsSync` fixture assertions caught it, as the 2 failures
this fixes.

Resolve the root through the package's own `./collectors` export, the
same posture this file already uses for `readPolyfillManifests()`.
Violations are still reported under the stable
`packages/polyfill-connectors/src/...` names, so the allowlist and every
`Violation.file` are unchanged. Scanned coverage goes from 19 files to
86, still with zero violations.

Two guards stop the silence recurring: an empty resolved root now throws
instead of passing vacuously, and the falsifiability injection tests
write their synthetic file into the root the scanner really walks, so
they would fail if it drifted again.

Fixes #53

Assisted-by: AI
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>

* fix(ri): regenerate connector registry

Regenerate the manifest-derived registry after updating the vendored connector package.

Refs: #55
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
Assisted-by: AI

* fix(test): resolve compiled vendored scanner sources

Keep PR #60 scanner coverage stable when the vendored package ships compiled JavaScript rather than TypeScript source.

Refs: #55, #60
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
Assisted-by: AI

* fix(test): align legacy aliases with connector registry

Remove Signal from the hand-maintained alias expectation after the manifest-derived registry demotion.

Refs: #55
Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
Assisted-by: AI

---------

Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>