test(hosted): production e2e for scan --mode hosted across every ecosystem and package manager

[Mikola Lysenko (mikolalysenko)]

What

The first test suite that exercises hosted mode (scan --mode hosted) against the real Socket production service, with no mocking anywhere, using patches that are actually published on production today.

Every other hosted-mode capstone in this repo (e2e_redirect_*_build.rs) serves the patched artifact from a local wiremock. That proves the CLI's rewrite grammar, but it cannot notice production drifting away from the CLI. This suite can.

Each leg:

1. installs a pinned, known-vulnerable dependency from its real upstream registry with the real package manager;
2. asserts the installed bytes are pristine (anti-vacuity);
3. runs socket-patch scan --mode hosted --json --yes against patches-api.socket.dev;
4. asserts the lockfile / registry config now pins patch.socket.dev + the expected patch UUID;
5. wipes the install tree and reinstalls from the rewritten lock alone — the package manager itself fetches from Socket and verifies the integrity pin it was handed;
6. asserts the reinstalled bytes carry the patch.

Step 5 is the point. It is the only place in the repo where a third-party package manager — not socket-patch — downloads a Socket-hosted artifact and independently verifies its checksum.

The suite runs against the unauthenticated public proxy on purpose: no API token is used or needed, and SOCKET_API_TOKEN is scrubbed from the child environment. That is the surface every user without a token gets.

Coverage — 14 legs, all green

Ecosystem	Package managers	Status
npm	package-lock, npm-shrinkwrap, pnpm, yarn classic, yarn berry, bun	✅ full install proof
PyPI	requirements.txt, uv.lock	✅ full install proof
Cargo	per-patch sparse registry	✅ full install proof
RubyGems	bundler	⚠️ redirect asserted; install blocked by a server defect (below)
Maven / NuGet / Composer	—	canary: production publishes zero free patches for these
Go	—	negative assertion (refused by design)
Deno	—	negative assertion (unsupported)

test result: ok. 14 passed; 0 failed  (SOCKET_PATCH_HOSTED_E2E_STRICT=1)

Maven, NuGet and Composer all implement hosted mode, but production has no free-tier patches for them, so there is nothing real to redirect to. Rather than skipping silently, canary_unpublished_ecosystems probes production every run and reports the moment that changes.

Required production patches

The suite is pinned to these. preflight_required_patches_are_published verifies all four every run, so a withdrawn patch produces one clear failure naming the PURL instead of N confusing ones that look like CLI regressions.

Ecosystem	PURL	Patch UUID	Advisory
npm	pkg:npm/minimist@1.2.2	80630680-4da6-45f9-bba8-b888e0ffd58c	GHSA-xvch-5gv4-984h / CVE-2021-44906
PyPI	pkg:pypi/urllib3@1.26.18	any of de58c8b8…, 26242e35…, e828efa5…	GHSA-38jv-5279-wg99 &co
Cargo	pkg:cargo/traitobject@0.1.1	cf2e6f58-d9fa-4096-9151-c34afa717f89	GHSA-pp8r-vv2j-9j5v
gem	pkg:gem/activestorage@7.0.2.2	2535d43d-67ce-4944-be27-c19e113997fb	GHSA-w749-p3v6-hccq

docs/testing/hosted-production-e2e.md documents how these were chosen and how to re-pick one if withdrawn. urllib3 carries three patches (one per advisory) and the suite accepts any of them, so a server-side reorder does not go red.

Cargo is the thin spot: traitobject is the only crate with a free published patch today (4 versions of it).

🔴 Three real issues this found

None are test bugs. Full evidence in docs/testing/hosted-production-e2e.md.

1. gem hosted mode is unusable for gems with dependencies — server fix

Socket's gem patch-registry serves a compact index that declares no runtime dependencies, while the .gem it serves declares six. Bundler fails closed:

Bundler::APIResponseMismatchError: Downloading activestorage-7.0.2.2 revealed
dependencies not in the API (activesupport (= 7.0.2.2), actionpack (= 7.0.2.2), …)

# rubygems.org
curl -s https://index.rubygems.org/info/activestorage | grep '^7\.0\.2\.2 '
# 7.0.2.2 actionpack:= 7.0.2.2,activejob:= 7.0.2.2,…|checksum:7997042a…

# Socket patch-registry
curl -s "https://patch.socket.dev/patch-registry/gem/<grant>/<uuid>/info/activestorage"
# 7.0.2.2 |checksum:89b47c6d…

The compact-index generator must emit the gemspec's runtime dependencies. The test asserts the redirect (correct) and tolerates the install failure, failing loudly if it fails for any other reason. SOCKET_PATCH_HOSTED_E2E_GEM_STRICT=1 promotes it to a hard failure — flip that on as the regression guard once the server is fixed.

2. pnpm 11 rejects hosted lockfiles by default — CLI UX fix

pnpm 11 added a lockfile supply-chain policy comparing each entry's tarball URL against the registry's published metadata. Hosted mode deliberately rewrites that URL:

[ERR_PNPM_TARBALL_URL_MISMATCH] minimist@1.2.2 has a tarball URL
(https://patch.socket.dev/…) that does not match the registry's published
metadata (https://registry.npmjs.org/minimist/-/minimist-1.2.2.tgz)

pnpm install --trust-lockfile is pnpm's documented opt-out and works (verified — the patched artifact installs cleanly). Neither --trust-policy-exclude nor --no-verify-store-integrity helps.

scan --mode hosted should emit a redirect_pnpm_* warning naming the flag, the way it already warns for redirect_gem_no_checksums_section and redirect_rush_repo_state_stale.

3. uv.lock sdist entry rewritten to a wheel URL — CLI, minor

The rewriter points sdist at the patched wheel and keeps the original sdist's size, so URL, hash and size disagree. uv tolerates it because it prefers the wheel; it would bite on --no-binary.

CI: the hosted-e2e job

Gated off by default — the suite is #[ignore]-gated, so the existing test and e2e jobs are untouched. It runs only in the new dedicated hosted-e2e job, which is intended to be a required status check.

Deliberately no job-level if:, no needs:, no matrix, no continue-on-error — a skipped required check is ambiguous to branch protection and can wedge a PR at "Expected — waiting for status". The job always runs and always reaches success or failure; the kill switch gates the steps.

SOCKET_PATCH_HOSTED_E2E_STRICT=1 in CI turns the suite's local "toolchain missing" soft-skips into hard failures, so a required check can never report green on an unexercised leg. The job retries the suite 3× with backoff, because the public proxy intermittently 503s — the documented reason the older live-API suites were pulled from the PR matrix.

🔌 Escape hatch

Set a repository variable (Settings → Secrets and variables → Actions → Variables):

HOSTED_E2E_DISABLED = true

then hit Re-run failed jobs on any blocked PR. vars is read at job-run time, so no commit and no push are needed — the job goes green with a loud ::warning:: and a BYPASSED banner in the job summary, and every open PR clears on its next re-run. Delete the variable to re-arm.

Any value other than exactly true leaves the suite armed — a typo must not silently disable production coverage. Verified across all five gate states (default / disabled / typo'd / dispatch-force / dispatch-skip).

For a single run: Actions → CI → Run workflow → hosted_e2e = force | skip.

⚠️ One-time setup after merge

The job runs as soon as this lands, but making it required is a repo setting, not something this PR can do:

Settings → Branches → protection rule for main → Require status checks to pass → add hosted-e2e

Do that after the first green run on main.

Verification

• cargo test … --test e2e_hosted_production -- --ignored → 14 passed, 0 failed
• Same under SOCKET_PATCH_HOSTED_E2E_STRICT=1 → 14 passed, 0 failed
• Default run (no --ignored) → 0 passed, 14 ignored — existing jobs unaffected
• cargo clippy --workspace --all-features -- -D warnings → clean
• cargo fmt --check → clean for the new file
• ci.yml parses; hosted-e2e has no if:/needs:/continue-on-error

🤖 Generated with Claude Code

---

Note

Medium Risk
Introduces a branch-protection-oriented CI job that depends on live production and external registries; merge friction is mitigated by kill-switch and retries, but operational coupling to Socket production is new.

Overview
Adds the first unmocked hosted-mode capstone: e2e_hosted_production.rs runs scan --mode hosted against real patches-api.socket.dev / patch.socket.dev and real upstream registries. Each leg installs a pinned package, proves pristine bytes, runs the CLI redirect, then reinstalls from the rewritten lock so the package manager fetches and verifies Socket-hosted artifacts—coverage wiremock redirect tests cannot provide.

Legs span npm-family managers (lock, shrinkwrap, pnpm, yarn classic/berry, bun), PyPI (requirements.txt, uv.lock), Cargo sparse registry, gem redirect (install tolerated for a known server index bug), plus negative/canary probes for Go, Deno, and maven/nuget/composer with no free production patches. Tests are #[ignore]-gated with SOCKET_PATCH_HOSTED_E2E_STRICT for CI. docs/testing/hosted-production-e2e.md documents patch pins, gaps, and issues the suite found (gem compact-index deps, pnpm 11 tarball policy, uv.lock sdist rewrite).

CI adds a dedicated hosted-e2e job (intended required check): installs toolchains, runs the suite with retries for transient 503s, and supports bypass via HOSTED_E2E_DISABLED or workflow_dispatch hosted_e2e force/skip without skipping the job itself.

^{Reviewed by Cursor Bugbot for commit 13e5442. Configure here.}

[socket-security-staging]

No dependency changes detected. Learn more about Socket for GitHub.

👍 No dependency changes detected in pull request

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 3 potential issues.

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Retry loop never retries
  • Added set +e before cargo test and set -e after capturing status to disable errexit during test execution, allowing the retry loop to function correctly.

Or push these changes by commenting:

@cursor push 5f1965cb50

Preview (5f1965cb50)

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -916,9 +916,11 @@
           # transient 503 does not block merges through a required check.
           for attempt in 1 2 3; do
             echo "::group::hosted-e2e attempt $attempt"
+            set +e
             cargo test -p socket-patch-cli --test e2e_hosted_production -- \
               --ignored --nocapture --test-threads=4
             status=$?
+            set -e
             echo "::endgroup::"
             if [ "$status" -eq 0 ]; then
               exit 0

_{You can send follow-ups to the cloud agent here.}

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 13e5442. Configure here.}

[cursor]

Retry loop never retries

High Severity

The hosted-e2e retry loop captures status=$? after cargo test, but GitHub Actions runs steps with bash -e. A failing cargo test aborts the step immediately, so attempts 2 and 3 never run. Transient production 503s — the reason this retry exists for a required check — will still fail the job on the first try.

 

^{Reviewed by Cursor Bugbot for commit 13e5442. Configure here.}

[cursor]

Incomplete API token scrub

Medium Severity

run removes SOCKET_API_TOKEN but not the peer alias SOCKET_CLI_API_TOKEN. The binary promotes that alias onto SOCKET_API_TOKEN at startup, so an ambient socket-cli token still authenticates the child and moves the suite off the free public proxy this test is meant to cover.

 

^{Reviewed by Cursor Bugbot for commit 13e5442. Configure here.}

[cursor]

Orphaned workflow YAML fragment

Low Severity

if-no-files-found: warn from the setup-matrix upload-artifact step was left at the end of the hosted-e2e run: | script after exit 1. It is unreachable shell text there, and the upload step no longer carries that option.

Additional Locations (1)

• .github/workflows/ci.yml#L760-L766

 

^{Reviewed by Cursor Bugbot for commit 13e5442. Configure here.}