# STRESS-TESTING.md — stress-testing a Thunderpolt deployment > **Retired-tooling note (2026-09-13).** The `just sp6-*` wrappers, the > `loadgen ramp`/component bands, `just demo-scenarios` and the isolated > component-benchmark matrix referenced later in this document were removed in > the codebase-simplification program. Treat those mentions as historical; the > surviving local-stack entry point is `demo/mvp/sp7-stack.sh` over > `demo/lib/stack.sh`, and the retired-command history is preserved in > [`docs/history/retired-stress-tooling.md`](../history/retired-stress-tooling.md). > The authoritative live command list is the current `justfile` (`just --list`). You are an agent asked to stress-test a Thunderpolt deployment. The only thing you truly need is the deployment's public **site URL** (the reference deployment lives at `https://tp.mudit.blog` — if you are reading this doc from there, that is your `SITE_URL`). Everything else — facilitator, explorer, dispenser, the deployed contract addresses — is discovered from the site's `/meta`, and the load generator (`loadgen`, built from source you fetch straight from the deployment, §0) does that discovery for you. **You do NOT need engine URLs**: production engines are private-network-only by design, and the whole recipe below works against an engines-private deployment from any laptop with outbound HTTPS. Don't assume any tool is preinstalled, and don't assume you can reach the project's git hosting — §0 bootstraps everything this doc uses from a bare macOS or Linux machine using only OS-default tooling plus what it installs itself, and fetches the source from the deployment itself. Read `PAYING.md` (served next to this doc) if you want the underlying vocabulary — schemes, `/meta`, the payer/session-key split, decline reasons — but this doc is self-contained for load testing. **Testnet only.** MockUSDC has no real-world value. POL on Polygon Amoy is testnet gas, not real money — but the dispenser's POL treasury is still **finite**, which is the whole reason this doc opens with etiquette rather than commands. This is an MVP proof of concept: channel funds trust the hub operator within the challenge window (Level-1, operator-trusted settlement — see the repo README's "Epoch-root settlement (M8)" section for the full trust-model disclosure). Do not model custody on this system, and do not model the dispenser as an infinite faucet. **Never print private keys.** Wallet-pool files (`.wallets/*.json`) hold many private keys at once (as one derivation seed) — treat the whole file like PAYING.md's `chmod 600` key files: never `cat`/echo it, never paste its contents anywhere. --- ## 0. From zero on a fresh machine (macOS or Linux) A bare box is missing three things this recipe needs — and none of them requires Homebrew, an OS package beyond the C compiler, or any access to this project's (private) git host: 1. a **C linker** — Rust links the built binary against `cc`; 2. the **Rust toolchain** — `cargo`; 3. the **source** — fetched as a tarball straight from the deployment, so you never touch our git hosting or need `git` installed at all. The build needs nothing else: TLS is pure-Rust (rustls — no OpenSSL, `libssl-dev`, or `pkg-config`), and no dependency is a git source (only crates.io). Every command below is idempotent — safe to re-run, skips what is already present. ```bash SITE_URL="https://tp.mudit.blog" # or your operator's handout # 1. C toolchain (provides `cc`; skip if `cc --version` already works). if ! cc --version >/dev/null 2>&1; then case "$(uname -s)" in Darwin) xcode-select --install ;; # accept the GUI prompt, then re-run this block Linux) sudo apt-get update -y && sudo apt-get install -y build-essential \ || sudo dnf install -y gcc || sudo yum install -y gcc ;; # sudo may prompt esac fi # 2. Rust toolchain (skip if `cargo --version` already works). command -v cargo >/dev/null 2>&1 || \ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y . "$HOME/.cargo/env" # 3. Source tarball FROM THE DEPLOYMENT — no git, no GitHub. `curl` + `tar` # ship out of the box on macOS and virtually every Linux. curl -fsSL "$SITE_URL/src.tar.gz" -o thunderpolt-src.tar.gz tar xzf thunderpolt-src.tar.gz # -> Thunderpolt/ cd Thunderpolt # 4. Build the load generator. The `mvp` feature gate is REQUIRED — the # amoy/merchant/report/component/chain-bench/ramp subcommands don't # exist without it. cargo build --release -p loadgen --features mvp ./target/release/loadgen amoy --help # authoritative if this doc ever drifts ``` The extracted tree already carries the live deployment's address book at `deployments/amoy.json` (byte-identical to what the site serves at `$SITE_URL/meta/deployments`). That file plus the public URLs — all discoverable from `/meta` — are the whole handout: ```bash curl -s "$SITE_URL/meta" | python3 -m json.tool # urls.{facilitator,explorer,dispenser,source}, chainId, hubs[] DISPENSER_URL="$SITE_URL/dispenser" # = urls.dispenser RPC_URL="https://polygon-amoy-bor-rpc.publicnode.com" # chainId 80002 = Polygon Amoy; see the RPC note below ``` `python3` ships on macOS and essentially every Linux; it is the only JSON tool the run itself needs. (If you *do* have access to the source repo, `git clone` still works — the tarball is simply the path that does not require it.) ### Pick an RPC that won't rate-limit your bootstrap — this bites first-timers `RPC_URL` is used for **exactly one thing**: the on-chain bootstrap on a first/cold run — floor-checking wallet balances and landing each new payer's `approve`+`deposit` (§2, §3). Payments themselves are off-chain and reconcile over HTTPS to the facilitator, so **a warm run that reuses already-deposited wallets barely touches the RPC at all.** But on a cold run the deposit step polls for transaction receipts in a tight loop, and that is where a throttling RPC kills you before a single payment is sent. **Do NOT use `https://rpc-amoy.polygon.technology` for this.** It is behind Cloudflare and rate-limits the receipt-poll loop hard — you will see `chainio: submit gave up after 5 broadcast attempt(s) ... HTTP error 429 with body: error code: 1015` (1015 = Cloudflare rate limit) or a bare `approve ... tx not included after 5 attempts`, and bootstrap aborts before the burst. Tested working alternatives (both return chainId 80002, neither Cloudflare-throttled the deposit loop in practice): - `https://polygon-amoy-bor-rpc.publicnode.com` ← the default above - `https://polygon-amoy.drpc.org` A dedicated key (Alchemy/Infura/QuickNode Amoy) is best of all if you have one. If bootstrap still can't land a deposit, it is an RPC problem, not a Thunderpolt one — switch endpoints and re-run (a dropped tx clears the wallet's nonce, so a clean re-run is always safe; you will NOT double-spend or double-fund). See §3 "First run vs warm run" for how to confirm bootstrap actually completed. The tarball is scoped to **building `loadgen` and driving a remote deployment** (the §3 recipe) — it carries the Rust workspace, web assets, and docs, but not the Solidity contracts, prod-deploy scripts, or the `just sp6-*` local-stack helpers (§4/§7). Those local-stack conveniences need a full source checkout; the primary showcase recipe below needs only what the tarball ships. ### Optional tooling for reading reports (§5–§6) — not for the run The burst itself needs nothing beyond the four steps above. Two more tools appear later; install them only when you reach those sections: - **`jq`** — the report queries in §5/§6. Any of them can be done with `python3` instead if you'd rather not install it. To install: `sudo apt-get install -y jq` (Debian/Ubuntu) · `sudo dnf install -y jq` (Fedora/RHEL) · `brew install jq` (macOS, if you have Homebrew). - **`cast`** (Foundry) — only for manual on-chain balance/tx inspection you do OUTSIDE `loadgen` (loadgen does all its own chain IO). Install with the official one-liner, then make `cast` usable IN THIS SAME SHELL (`foundryup` only wires up FUTURE shells via your profile): ```bash curl -L https://foundry.paradigm.xyz | bash export PATH="$HOME/.foundry/bin:$PATH" [ -f "$HOME/.foundry/env" ] && . "$HOME/.foundry/env" foundryup ``` See PAYING.md §0 for the full per-tool rationale if anything above fails. --- ## 1. Rules of engagement Read this section before running anything. It is the point of this doc. - **The Amoy dispenser treasury is finite** and every `/fund` call costs 0.1 POL. Budget your run before starting it. Every chain-touching harness in this repo prints its worst-case POL budget up front and **refuses to start** rather than silently truncating the workload if the estimate exceeds a cap — if a harness refuses, lower your scale, don't work around the refusal. - **Dispenser limits:** 5 funds per address, 1000 total, per 24h window. `429` means you hit one of these — back off. **Never rotate to a fresh address to evade a rate limit or a floor check** — that multiplies real treasury spend for no benefit; the whole point of the wallet pool (§2) is reusing the same funded wallets run after run. - **Exact-scheme payments are real on-chain transactions.** Keep `--exact-per-min` a trickle (the flag's default, 6/min, is already tuned for this) — channel payments are the load vehicle. For a high-rate burst set `--exact-per-min 0`. - **Run bounded bursts, not open-ended floods.** 60 seconds at an explicit `--target-rate` is the polite unit of load. **Never run uncapped** (`--target-rate 0`) **against a shared or live Amoy stack** — uncapped/non-retrying runs are for the isolated ramp probe (§7) against your OWN local stack only. Raise the target stepwise between bursts, only while success stays ~100%. - **Watch the explorer scoreboard while you drive.** The explorer (`urls.explorer` from `/meta`; UI at `/web/explorer.html`) shows live applied-payment throughput — your independent, server-side view of whether the deployment is absorbing your load or you're just generating declines. - **Back off on decline storms.** A brief `duplicate-payment` / `unauthorized-cumulative` blip that self-heals is normal (the driver quarantines and resyncs those payers itself); a *sustained* stream of declines, `429`s, or `503`s means STOP, wait a minute, and come back at a lower `--target-rate`/`--in-flight`. Hammering a declining stack produces garbage numbers and real operator pain. - **One stress run at a time per stack.** `just sp6-ramp` refuses to start while a soak is running against the same stack; hold yourself to the same rule for any run you launch by hand — two concurrent load generators fighting over the same wallet pool and floors produce confusing, useless results, not double the throughput. - **macOS runner prereq:** stay on AC power with the lid open for anything longer than a couple of minutes. Maintenance Sleep can stretch a run's wall-clock time by roughly 5× (reported timings stay honest — they're `Instant`-based — but the run just crawls). --- ## 2. Wallet fleets and deposit sizing `loadgen amoy` keeps a **wallet-pool file**, one per network, created on first run and **REUSED forever after** — never regenerated per run. Pool size only ever grows: `--payers N` against an existing pool of size ≥ N re-derives the exact same wallets (same addresses, same on-chain deposits, nothing re-funded); a bigger N extends the pool, funding and depositing only the shortfall. A pool never shrinks and never gets a fresh seed on its own. - **Local drill stacks:** point `--wallets` at a throwaway path — there's no real money behind a local Anvil, so a fresh pool per run costs nothing. - **The shared Amoy pool** lives at `.wallets/amoy-pool.json` in the repo root — literally the same file across every run against live Amoy. **Losing it orphans every existing wallet's on-chain channel deposit** — a fresh pool derives different addresses from a different seed. Treat it like a low-value-but-annoying-to-replace credential: gitignored, never deleted casually, never committed. - The pool file also persists each wallet's **local payment counters** (`(cumulative, epoch)` per merchant, schema `thunderpolt-walletpool/2`, additive — old `/1` pools load fine). They're the engine-less fallback baseline; drift self-heals via the facilitator's personalized quote, and a payer that can't be disambiguated simply parks for the run. **Deposit sizing is arithmetic, not vibes.** A burst spends `target_rate × duration × price` of MockUSDC across the pool: - The cheapest paid route (`/api/paid/time`) costs **$0.0001** (100 base units of 6-decimal MockUSDC). A 2,000/s × 60s burst on it costs 2,000 × 60 × $0.0001 = **$12 total** — ~$0.38 per payer across 32 payers (§3's recommended shape). - The dispenser mints **1000 MockUSDC per `/fund` call**, and the default channel deposit (`--deposit-units`, 500000000 = 500 USDC) fits inside a single fund with headroom to spare. One default deposit covers ~330 of those $12 bursts. - **Deposits are the ONLY per-payer on-chain step** (one `approve` + one `deposit` transaction, submitted by loadgen automatically), and the only per-payer wait: ~18s from deposit-tx to indexer-confirmed visibility on prod. They run in parallel (`--deposit-concurrency`, default 4). - Therefore: **a few dozen payers with deep deposits beats hundreds of shallow ones — but "a few dozen", not a handful.** Payers spread both concurrency AND the per-payer throughput cap (each payer's cumulative chain tops out ~8–20 payments/sec against real prod, §3), so you need *enough* payers to hit a target rate: ~32 payers reach ~570 TPS from a laptop where 8 payers plateau at ~65 (§3's sizing table). What you do NOT need is *thousands* — 32 payers × a modest `--in-flight` delivers the same concurrency as thousands of single-shot payers at ~1% of the on-chain setup cost, funding spend, and dispenser-limit pressure. Deposit those few dozen deep; don't chase throughput by adding payers into the hundreds. --- ## 3. The showcase: from a laptop to thousands of TPS, engine-less This is the primary recipe. It drives the FULL public path (you → site → facilitator → hub) against an engines-private deployment — no `--engine-url` at all — and uses the **`--in-flight` ceiling-voucher knob** to put hundreds of requests in flight from one process. **Why `--in-flight` exists:** the classic channel discipline signs one voucher at a time per payer, so each payer's throughput is capped at `1 / round-trip-time` (~2.5/s at 400ms). The hub's ceiling-voucher accept rule lifts that: a voucher is an authorization *ceiling* — loadgen signs ONE voucher per refill window at `C = known_cumulative + K × price` and fires K concurrent requests with DISTINCT payment-ids under it, advancing the window when it resolves. Drift resyncs itself: an `unauthorized-cumulative` (or `duplicate-payment`) decline parks the payer for ONE personalized quote to the facilitator, which returns the hub-recorded ceiling, and the payer resumes. If the deployment's quote does not expose the ceiling (an older hub), loadgen **refuses `--in-flight > 1` at startup** with a clear message — drop to `--in-flight 1` there. Two Phase-0 knobs ride along with `--in-flight K > 1`: - **`--window-advance-frac` (default `0.5`)** — the next window (a fresh signature at `C' = C + K × price`) opens once `ceil(K × frac)` of the current window's requests have resolved with at least one success, instead of waiting for all K. The old full-barrier behaviour is `1.0`. Overlap removes the per-window tail-latency stall (measured on a jittered local stub: **1.51× the per-payer rate** at 0.5 vs 1.0; a WAN tail makes the gap bigger, a constant RTT makes it vanish). Money-safety is unchanged — the early-signed headroom is deposit-*reserved*, the hub never serves past `servedSum ≤ recorded ceiling`, and unused headroom is refunded on exit (see `crates/loadgen/src/mvp/ceiling.rs`'s module docs). - **`--verification {ecdsa-per-payment|verified-by-cache}` — REQUIRED with `--in-flight > 1`.** One signature now covers K payments, so a hub with its signature cache on (the engine default since Phase 0, `--verify-cache-entries 262144`) verifies requests 2..K of every window from the cache — that run is a **`verified-by-cache`** run, a *separate tier* under CLAUDE.md's fifth benchmark-disclosure rule. Loadgen cannot introspect the hub's config, so you assert it, exactly like `--ack-durability`: `verified-by-cache` when the target hubs run the cache, `ecdsa-per-payment` only if the operator disabled it (`--verify-cache-entries 0`). Either value makes the run a `/2` document carrying `verification`, and a `verified-by-cache` run's `label` carries the marker (§6). `--in-flight 1` runs may omit the flag (absent = the documented `ecdsa-per-payment` default) and may NOT claim `verified-by-cache` — one signature per payment has nothing to cache. **Sizing math (do this, don't guess) — payer count is the real lever, NOT in-flight.** `total in-flight ÷ p50` is only an *upper* bound. The *binding* limit against a real deployment is per-payer: each payer is one monotone cumulative chain, and in public-only mode (the prod shape, no `--engine-url`) its refill window advances through a facilitator round-trip, so a single payer tops out around **8–20 payments/sec no matter how high you set `--in-flight`.** You scale past that by adding *payers*, not by cranking in-flight on a few. Measured, one laptop over the open internet to `tp.mudit.blog`, `/api/paid/time`, 60s bursts: | `--payers` × `--in-flight` | total slots | sustained (peak) TPS | notes | | --- | --- | --- | --- | | 8 × 100 | 800 | **65** (steady) | in-flight-rich but payer-starved — plateaus at ~8 TPS/payer | | 32 × 32 | 1024 | **570** (~755 peak) | 100% success, ramps then holds — the sweet spot here | | 32 × 128 | 4096 | ~15, **decaying 27→12→5** | over-subscribed: throughput COLLAPSES, run hangs | Read that table before you pick knobs: 4× the payers at the *same* total slots bought ~9× the throughput, while 4× the *in-flight* at fixed payers **destroyed** it. Rule of thumb from a laptop: **`--payers 32 --in-flight 32`** (≈1024 slots) is a good high-rate default; raise payers to climb, keep `--in-flight` in the **16–32** band, and keep **total slots ≲ ~1000**. **Why over-subscription collapses it:** each in-flight slot is one OS thread mostly blocked on HTTP, sharing a keep-alive pool sized to payers × in-flight. Past ~1–2k slots on a laptop the threads and connections thrash, per-request latency climbs without bound, the refill windows stall, and throughput decays segment over segment (the `27→12→5` row above) — often without a single decline, because nothing is being *rejected*, just starved. If you see per-segment `tpsSucceeded` falling instead of rising-then-flat, you are over-subscribed: cut `--in-flight`, don't raise it. (The driver itself is not the bottleneck: measured on a laptop, 8 payers × 100 in-flight sustains ~7,000 req/s against a 100ms-latency local stub — a harness measurement, NOT payment TPS. Against real prod the ceiling is per-payer latency, above, not driver CPU.) A driver box **close to the stack** (low RTT) raises the per-payer cap and lets the same payer count go far higher — see §4's showcase note. **The run** (60s bounded burst, time-endpoint only, no exact trickle): ```bash ./target/release/loadgen amoy \ --site-url "$SITE_URL" \ --dispenser-url "$DISPENSER_URL" \ --deployments deployments/amoy.json \ --rpc-url "$RPC_URL" --chain-id 80002 \ --wallets .wallets/amoy-pool.json \ --payers 32 --in-flight 32 \ --verification verified-by-cache \ --target-rate 2000 --exact-per-min 0 \ --mix "/api/paid/time=1" \ --duration-secs 60 --report-every-secs 15 \ --out-dir bench-results ``` (`--verification verified-by-cache` asserts the target hubs run their signature cache — the shipped default; see the knob notes above. The resulting `final.json` is a `thunderpolt-soak/2` document labelled `verified-by-cache`, never to be compared with an `ecdsa-per-payment` run.) Note what is NOT in that command: any engine URL. Bootstrap discovers the facilitator/explorer from `$SITE_URL/meta`, learns registration and deposits from the facilitator's `GET /payer/{addr}/channels` (indexer-backed, ~18s deposit finality), and baselines every payer's `(epoch, cumulative)` with one personalized `POST /quote` each. This is exactly how the production deployment is meant to be driven from any box with only the public URL. **First run vs warm run:** the first run funds 8 wallets (8 × 0.1 POL from the treasury) and lands 8 deposits (~18s, parallel) before the burst starts. Every later run reuses all of it and goes straight to load. **Confirm bootstrap actually finished — don't trust the exit code.** If the on-chain bootstrap aborts (a stuck/throttled deposit, §0's RPC note), the process can still exit `0` while having sent **zero payments** and written **no** `soak-/` directory. A successful run prints, in order, `[amoy] deposits: N wallet(s) ...` → `[soak] segment 1 written: ...` → a final `[amoy] sustained TPS (fully-verified, full-path): ` line. If you don't see the `sustained TPS` line and a fresh `bench-results/soak-/ final.json`, the burst did NOT run — check the tail of the output for `not included`, `gave up`, or `429 / error code: 1015`, switch RPC per §0, and re-run. Never quote a number from a run whose `final.json` you haven't confirmed exists. A quick post-run check: ```bash latest=$(ls -td bench-results/soak-*/ | head -1) test -f "$latest/final.json" && echo "OK: $latest" || echo "NO BURST RAN — bootstrap aborted, see §0 RPC note" ``` **Etiquette ladder, not a light switch:** don't open at full scale — climb by **adding payers**, holding `--in-flight` in the 16–32 band (per the sizing table above; more in-flight does not buy more TPS here, it buys collapse). Keep `--target-rate` high enough to not be the limiter (it only throttles *down* from the concurrency cap) and let payer count set the pace. Run 60s at each rung, checking ~100% success, a *rising-then-flat* (never falling) per-segment `tpsSucceeded`, and the explorer scoreboard before the next rung, watching for decline storms (§1): | `--target-rate` | `--payers` × `--in-flight` | total slots | observed sustained (laptop→prod) | | --- | --- | --- | --- | | 2000 | 8 × 24 | 192 | ~65/s (anchor: payer-starved) | | 2000 | 16 × 32 | 512 | ~250–350/s (interpolated) | | 2000 | 24 × 32 | 768 | ~450–550/s (interpolated) | | 2000 | 32 × 32 | 1024 | ~570/s sustained, ~755/s peak (anchor) | Only the 8× and 32× rows are measured anchors (this doc's reference runs); the middle rungs interpolate — treat them as "raise payers, re-measure", not as guarantees. Past ~1024 total slots on a laptop, throughput *falls* (the `27→12→5` decay row in the sizing table) — a low-RTT driver box near the stack is what pushes the ceiling higher, not more slots (§4 showcase note). Segments print every `--report-every-secs`; watch `tpsSucceeded` and the `declined`/`errors` maps live rather than waiting for the final report. The first segment is warm-up and reads low (169/s in the 32×32 anchor before it climbed to ~755) — judge the run by the steady-state segments, not segment 1. **Reading the result:** the run writes `bench-results/soak-/` segments plus `final.json` — `sustainedTpsSucceeded` is the ONLY number you may quote as payment TPS, and it is a **full-path, HTTP-bound** number: NEVER comparable to the repo's engine-direct T1–T4 tiered-benchmark numbers (those measure an in-process verify/apply pipeline with no HTTP and no chain). The report's `corpus` block disclosures include `inFlight`, `windowAdvanceFrac`, `signaturesPerPayment` (= K: one signature per K payments; 1 = classic), the top-level `verification` tier, and the per-hub `reconcile` source (`engine` vs `public`) so the run shape is auditable. A `verified-by-cache` number is a different tier from an `ecdsa-per-payment` number — never compare the two (§6). See §6 for the merged report. --- ## 4. Classic bounded burst (and what `--engine-url` actually does) The gentler, mixed-workload shape — same binary, `--in-flight 1` (the default), modest rates. Every flag below is real (`./target/release/loadgen amoy --help` is authoritative if this doc and the binary ever disagree): ```bash ./target/release/loadgen amoy \ --site-url "$SITE_URL" --dispenser-url "$DISPENSER_URL" \ --deployments deployments/amoy.json \ --rpc-url "$RPC_URL" --chain-id 80002 \ --wallets .wallets/amoy-pool.json --payers 4 --workers 2 \ --target-rate 5 --exact-per-min 2 \ --mix "/api/paid/time=90,/api/paid/mirror=9,/api/paid/report=1" \ --duration-secs 120 --report-every-secs 30 \ --out-dir bench-results ``` Notes: - **`--engine-url` is an OPTIONAL per-hub enhancement, never a requirement.** It is repeatable and **positional against the address book's `hubs[]` array**: pass one per hub in order (count must equal `hubs[]`'s length when any are passed), with `""` for a hub whose engine you can't reach. A hub with an engine URL bootstraps and reconciles from the engine's `/debug/ledger` directly; a hub without one (`""`, or omitting the flags entirely — the normal case against prod) is driven **PUBLIC-ONLY**: bootstrap via the facilitator's `/payer/{addr}/channels` + personalized `/quote`, mid-run drift reconcile via the same quote call. **A hub is never excluded for lacking an engine URL.** Only a payer whose drift genuinely can't be disambiguated (e.g. an old hub that exposes no quote ceiling) parks for the rest of the run. - `--wallets`, `--chain-id`, and `--deployments` are the three things that change between a real-Amoy run and a local drill stack — everything else is the same command. - `--workers` partitions payers across driver threads in `--in-flight 1` mode; in ceiling mode (`--in-flight > 1`) the thread count is payers × in-flight and `--workers` is not used. **Local-drill variant** — same command pointed at a local stack's handout (e.g. `just sp7-stack`'s), with a throwaway pool and `--chain-id 31337`; engine URLs become genuinely useful there since the drill stack publishes them: ```bash ./target/release/loadgen amoy \ --site-url "$SITE_URL" --dispenser-url "$DISPENSER_URL" \ --deployments "$DEPLOYMENTS_FILE" \ --engine-url "$ENGINE1_URL" --engine-url "$ENGINE2_URL" \ --rpc-url "$RPC_URL" --chain-id 31337 \ --wallets /tmp/stress-drill-pool.json --payers 4 --workers 2 \ --target-rate 5 --exact-per-min 2 \ --mix "/api/paid/time=90,/api/paid/mirror=9,/api/paid/report=1" \ --duration-secs 60 --report-every-secs 20 \ --out-dir bench-results ``` ### Showcase mode — operator max-throughput demo Everything above is calibrated for hitting **someone else's** stack: coordinate, stay capped, don't hammer. Against **your own deployment** — the one you operate end to end, pods and driver box alike — the goal flips: showing off the real ceiling IS the point of a showcase run, not a courtesy violation. This is the same `loadgen amoy` full path as above, uncapped — the tool that already proved **39.44 TPS sustained fully-verified full-path, 100.00% success, through a real LB against real Amoy** (`docs/perf-notes.md`'s "Production fleet stress" section, its M4 leg) at a 40/s cap; showcase mode is that same command with the cap removed. **Drive it from a box close to the stack** — a dedicated driver host on the same network as your pods is strongly PREFERRED FOR THROUGHPUT (low round-trip latency + room for high worker concurrency is exactly what an uncapped run needs); a laptop over the open internet will find a far lower ceiling, latency-bound, not because the tool won't run there. If you want the pod-direct ceiling instead of the LB-fronted full path — that section's M1/M2 legs — that's already `loadgen merchant`'s job (see "Merchant-boundary mode" just above, and its uncapped ramp-probe twin in §6): ```bash ./target/release/loadgen amoy \ --site-url "$SITE_URL" --dispenser-url "$DISPENSER_URL" \ --deployments "$DEPLOYMENTS_FILE" \ --engine-url "$ENGINE1_URL" --engine-url "$ENGINE2_URL" \ --rpc-url "$RPC_URL" --chain-id 80002 \ --wallets .wallets/amoy-pool.json --payers 1024 --workers 256 \ --target-rate 0 --exact-per-min 2 \ --mix "/api/paid/time=100" \ --deposit-units 5000000000 \ --duration-secs 3600 --report-every-secs 30 \ --out-dir bench-results ``` Three changes from the polite-default shape above: - **`--target-rate 0`** — uncapped, find-and-sustain-peak. This is still the real `loadgen amoy` full path, not the ramp probe (§6): payments retry per the normal decline contract (PAYING.md §8), they just aren't rate-limited going in. "Your own deployment" means the WHOLE path, not just the pods — a quick check is still worth it if anything downstream of your driver box (a shared LB, a chain RPC other traffic depends on) genuinely isn't yours alone. - **`--mix "/api/paid/time=100"`** — all weight on the cheapest real route. - **A big jump in `--payers`/`--workers`** — see the parallelism rule right below; throughput comes from concurrency ACROSS payers, never from pushing one payer harder. **Parallelism: scale ACROSS payers, never WITHIN one.** Two levels, and only one of them is a tunable: - **Across payers/workers — the entire throughput lever, fully concurrent.** `--workers` is real concurrency: each worker is one independent thread cycling its own disjoint slice of payers, and payers are partitioned round-robin so a given payer's whole cumulative chain is owned by exactly one worker for the life of the run (`partition_round_robin` in `crates/loadgen/src/mvp/soak.rs` — "once a group is handed to a worker thread, no other thread ever touches those `PayerState`s again"). Scale throughput by adding workers (bounded by cores/sockets) backed by enough payers that no worker ever idles waiting on a single payer's epoch/ambiguous state — `--payers` well above `--workers` (§2: "keep `--payers` ≥ `--workers`"), not equal to it. The production ceiling numbers in `docs/perf-notes.md` scale the same way: the M1/M2 legs' "128+128 sockets" IS 128 concurrent workers per pod (`loadgen merchant`'s blaster, one persistent connection per worker) driving 61,356/s aggregate — the lever there is worker/socket count too, not a single hot payer. - **Within one payer — strictly ≤ 1 in-flight, sequential, a CORRECTNESS invariant, not a knob.** Channel vouchers are monotone cumulative signatures (PAYING.md §6): two concurrent payments from the SAME payer race, and the loser lands `duplicate-payment` or worse, out-of-order. `loadgen`'s per-payer worker exclusivity enforces this structurally so you can't get it wrong by accident — but if you ever hand-roll parallelism on top (e.g. multiple scripts against the same wallet-pool payer, or PAYING.md's single-payer recipe run twice at once), you'll manufacture declines, not throughput. PAYING.md's hand-driven recipe (one shell session, one payer, ~2 payments/sec) is a **correctness** demo of exactly this invariant, never a throughput tool — that ceiling is by design, not a bug to work around. **Amount: why smaller is cheaper here, precisely.** Channel-scheme payments cost **zero gas at any amount whatsoever** — the amount never touches gas cost, full stop. What it DOES affect is how long one USDC deposit lasts: a smaller per-payment delta lets the same deposit fund far more payments before `insufficient-channel-balance`, so fewer re-deposits — and re-deposits, not per-payment fees, are the entire POL cost of a long run (§2). Two tiers: - **Polite default, no config changes needed:** `/api/paid/time` is already the reference site's cheapest route — **100 base units**, which happens to exactly equal the engine's default `--min-delta-floor` (the hub abuse guard that rejects any positive delta below it: hub reason `delta below hub floor ()`, mapped to wire `errorReason: "insufficient-payment"`). You cannot go lower against the reference site's existing routes this way — `--mix` only WEIGHTS which of a merchant's already-priced routes you hit, it does not override their price. - **Operator showcase, your own deployment only:** going below 100 needs a merchant route priced under the floor (your own merchant/site config — not a `loadgen` flag) AND a lowered or disabled floor on your OWN engines to accept it: `engine serve --min-delta-floor 1` (or `0` to disable the guard entirely). Only loosen this abuse guard on infrastructure you operate — never on a stack you don't control. Size `--deposit-units` so the pool outlasts the whole run — bigger means fewer re-deposits. At 100 base units/payment, a 5,000 USDC deposit (`--deposit-units 5000000000`) funds 50 million payments before topping up. **Disclosure is unconditional — whose stack you ran against changes nothing here.** §5's rule stands: **only `headline.sustainedTpsSucceeded` at ~100% success may ever be called "payment TPS."** An uncapped run WILL push past that success band as it finds the knee (declines, sheds, `429`s) — that's the point of running it uncapped — so report the labeled ceiling it finds (exactly like `docs/perf-notes.md`'s prod-fleet numbers, each tagged with its mode/boundary/network) separately from the sustained headline, never blended into it. ### Merchant-boundary mode — driving the facilitator directly `loadgen amoy` (above) always measures the FULL site→facilitator→hub path. `loadgen merchant` measures a different, narrower thing: the facilitator's own `/settle` boundary, driven DIRECTLY with a pre-built, pre-signed voucher corpus — no 402 fetch, no `/quote` call per payment, no site process at all. It is shaped like a fleet of merchant servers hitting one pod's facilitator, which is exactly the production question ("how many payments/s can merchant servers collectively push through our facilitator boundary") the 100–150k/pod target is about: ```bash ./target/release/loadgen merchant \ --facilitator-url "$FACILITATOR_URL" \ --hub "$HUB1_ADDRESS" --asset "$USDC_ADDRESS" --chain-id 80002 \ --payers 64 --merchants 8 --vouchers 200000 \ --workers 8 --target-rate 20 --duration-secs 120 \ --ack-durability async ``` Every flag is real — `./target/release/loadgen merchant --help` is authoritative. Notes: - The run WRITES `thunderpolt-soak-segment/2` + `thunderpolt-soak/2` artifacts under `<--out-dir>/soak-/` (boundary `"facilitator"`, `sustainedTpsSucceeded` = FRESH applies only), which `loadgen report` accepts as a `--soak` source. `--ack-durability` is REQUIRED and operator-asserted — pass the mode the target engines actually run; `--batching-*` flags disclose the facilitator's batcher config (defaults = the shipped posture: on, 256/4). - The corpus is partitioned BY PAYER across `--workers` (a payer's per-merchant cumulative chains never leave their worker), so any worker count is ordering-valid — `duplicate-payment` declines in a run's summary indicate a real server-side collision, never a driver artifact. - `--sigs-per-payment K` (default 1) builds the **K-payments-per-signature corpus** (Phase 0 P0-1c): `--vouchers` stays the PAYMENT count, the corpus signs `ceil(vouchers / K)` vouchers at chain ceilings `C_n = n × K × price` and emits K `/settle` bodies per signed voucher — same voucher + signature, distinct payment-ids, each priced at `price` — the same ceiling-window shape `loadgen amoy --in-flight K` signs on the site path. A hub with its signature cache on ecrecovers once per window and serves the other K−1 from the cache, so **`--verification {ecdsa-per-payment|verified-by-cache}` is REQUIRED when K > 1** (operator-asserted, like `--ack-durability`; `verified-by-cache` at K = 1 is refused). The artifact's `corpus` block discloses `signaturesPerPayment` (K) and `uniqueSignatures`, and a `verified-by-cache` run's `label` carries that marker; `loadgen report` refuses to merge the two tiers (§6). Reusing the same `--seed` identities after a K = 1 run? Add `--resume-from-ledger ` so windows at or below each chain's served prefix are dropped before signing. - `--driver prebuilt-replay` swaps the pooled-`ureq` loop for the minimal-CPU replay blaster (~10–20 µs/req driver CPU vs ~200 µs): complete request bytes are precomputed untimed, the timed loop only writes bytes on keep-alive sockets, and every `--sample-every`-th response is fully parsed as an honesty cross-check (recorded in the artifact together with the driver's own rusage CPU). Use it whenever the driver shares the box with the stack under test. - The corpus is **synthetic** — `--payers`/`--merchants` derive their own keypairs, unrelated to any real wallet pool (§2). Pointed at a facilitator whose hub has never registered those specific addresses (deposited real funds), every settle declines `unknown-payer` — that's expected on a bare/local stack; a payer-funded facilitator-boundary run is an evidence-run concern, not something this harness sets up for you. - Retries reuse the IDENTICAL body (same payment-id, same voucher) exactly like `loadgen amoy`'s discipline — never mint a fresh one on retry. - `overloaded` in a run's error summary means the facilitator's settle queue was full and shed the request with a fast `429` (`Retry-After: 1`) — nothing applied, nothing enqueued, safe to retry. It is counted SEPARATELY from other errors on purpose (a target that sheds cleanly past its ceiling is healthy; a target that times out or decays is not). - `just sp6-ramp LAYER=merchant-boundary` (§7) is the uncapped ceiling-probe equivalent of this mode — same corpus/driver, stepped worker counts, knee detection. `just` wraps the two most common shapes so you rarely need to hand-build the command above: - `SOAK_SECS=60 just sp6-soak` — boots a fully self-contained local stack, runs the burst, and generates the report, all in one call. Good for a quick end-to-end sanity check without needing any handout at all. - `just sp6-amoy-smoke` — a 2–3 minute smoke test against real Amoy (needs operator credentials in `.env.amoy`), tiny and cheap (≈0.2 POL on a fresh pool, ≈gas-only on a reused one). --- ## 5. Longer soaks A soak is the exact same `loadgen amoy` command as §3/§4 with a much larger `--duration-secs` (the SP6 gate runs 3600 — a full hour; soaks can run for hours or days). Output lands under `bench-results/soak-/`: - `segment-NNNNN.json` — one per `--report-every-secs` interval, schema `thunderpolt-soak-segment/1`. - `final.json` — written once the run completes, schema `thunderpolt-soak/1`. Both schemas are **FROZEN** (SP8's endurance evidence depends on them) — never hand-edit either file. New run-shape disclosures (`inFlight`, the per-hub `reconcile` source map) ride inside the free-form `corpus` block, additively — the frozen top-level shape is untouched. **`/2` — additive, never a mutation of `/1`.** `loadgen amoy --ack-durability {fsync|async} --boundary {site|facilitator}` (plus `--batching-enabled [--batching-max-size N --batching-pipeline N]`) is a DISCLOSURE, not a behavior switch — loadgen cannot introspect what mode the target actually runs, so these flags let the operator LABEL a run correctly. Leave every one at its default and a run's `segment-*.json`/`final.json` stay byte-for-byte `thunderpolt-soak{,-segment}/1` — set ANY of them away from default and the schema bumps to `/2`, with three fields appearing together (`ackDurability`, `boundary`, `batching: {enabled, maxSize, pipeline}`; see `docs/runbook.md`'s "Reading a soak report" section for the full field semantics). `loadgen report` REFUSES to merge/compare two `/2` sources whose `ackDurability` or `boundary` disagree — an async number is never comparable to an fsync number, and a facilitator-boundary number is never comparable to a site-boundary one, exactly like never conflating optimistic with fully-verified. A fourth, OPTIONAL `/2` field, **`verification`** (`ecdsa-per-payment` | `verified-by-cache`; absent = `ecdsa-per-payment`), is refused across tiers the same way — see §6. While a long soak is running, watch the latest segment rather than waiting for `final.json`. Both schemas share the same payment-counter and per-stage-histogram fields (`attempted`, `succeeded`, `declined`, `errors`, `schemes`, `endpoints`, `stagesMs`), but nesting differs: a segment carries them at its TOP level plus its own timing fields (`segment`, `elapsedSecs`, `startedUnixMs`/`endedUnixMs`, `tpsSucceeded`); `final.json` wraps the same counter fields one level down, under `totals`, alongside run-level metadata (`runId`, `hardware`, `corpus`, `polBurn`, top-level `sustainedTpsSucceeded`, …) that a segment doesn't carry: ```bash jq '{tpsSucceeded, attempted, succeeded, declined, total: .stagesMs.total}' \ bench-results/soak-/segment-.json ``` A success-ratio dip correlated with `duplicate-payment` (or, on ceiling-capable hubs, `unauthorized-cumulative`) declines means a payer's local counter drifted out of sync with the hub's — the worker quarantines and reconciles these cases itself (engine ledger where an engine URL exists, personalized quote otherwise), so a brief dip that recovers is expected background noise, not a problem to chase. A **sustained** dip that doesn't recover is a different story — look at the engine/facilitator/RPC logs, not the wallet pool. `declined`/`errors` are plain reason→count maps (e.g. `{"duplicate-payment": 12}`) — there's no separate breakdown field, the map keys ARE the breakdown. Mind the `totals` wrapper difference from the paragraph above when you point this at `final.json` instead of a segment: ```bash jq '.declined | to_entries | sort_by(-.value)' bench-results/soak-/segment-.json # segment: top-level jq '.totals.declined | to_entries | sort_by(-.value)' bench-results/soak-/final.json # final: under totals ``` --- ## 6. Read the report This section is condensed from `docs/runbook.md`'s "Reading a soak report" — read that for the full internals (the disclosure-validation machinery, the tri-state `/fund` retry policy that feeds `polBurn`, wallet-pool lifecycle detail). A soak's own `final.json` (§5) carries raw counters and a per-stage histogram (`totals.stagesMs`, keyed by stage name, each with `count`/`p50`/`p95`/`p99`/`max`/`meanMs`) plus a top-level `sustainedTpsSucceeded` — but the frozen **`headline`**/**`stageAttribution`** shape (what the rest of this section describes, and what disclosure rule R5 enforces) only exists in the MERGED report, produced by pointing `loadgen report` at that `final.json`: ```bash ./target/release/loadgen report --soak bench-results/soak-/final.json --out-dir bench-results # -> bench-results/mvp-report-.json (schema thunderpolt-mvp-report/1) + a rendered .md twin ``` (`--component`/`--chain`/`--engine-bench` are optional extra sources — see §7 — omit them for a bare soak report; `just sp6-report` is the same command via `just`.) - **`stageAttribution.stages`** is an array of exactly 7 frozen rows — `total`, `quote`, `verify`, `settle`, `apply`, `site`, `handler` — each carrying `p50`/`p95`/`p99`/`max`/`count` (ms) plus `shareOfTotalP50` (`jq '.stageAttribution.stages'` on the merged report). Two of these rows are NOT independent siblings of the other five: - **`settle` CONTAINS `apply`.** `apply` is the facilitator→hub inner span (the engine's own verify→apply pipeline) — a decomposition of `settle`, not an additional hop. Never add `apply`'s time on top of `settle`'s, and never rank `apply` against the other stages when picking "the bottleneck" — when `settle` IS the bottleneck, the report separately calls out whether it's apply-dominated (≥80% of settle's time inside apply) or facilitator-hop-dominated. - **`handler` contains the report-endpoint's indexer sidecar time** — `/api/paid/report`'s own indexer query was never broken out as its own timing bucket, so it's invisible inside `handler` by construction. Not a bug, just where that cost lives. - **`headline.sustainedTpsSucceeded` is the ONLY number that may ever be called "payment TPS."** Never `attempted`, never any per-stage count, never a component-bench or ramp number (§7) — those get their own labels, and mixing them up is exactly the mistake these disclosure rules exist to prevent. - **`headline.verification` is the payment's verification TIER — read it before you quote the number.** `ecdsa-per-payment` (the default when a source omits the field) means every payment carried a freshly-recovered signature. `verified-by-cache` means ONE signature authorised K payments (a `--in-flight K` site run or a `--sigs-per-payment K` merchant run) and the hub verified requests 2..K of every window from its `(digest, sig) → signer` cache instead of an ecrecover each. That is a *separate tier* under CLAUDE.md's fifth benchmark-disclosure rule: its `label` MUST carry the `verified-by-cache` marker, its `corpus` block MUST disclose `signaturesPerPayment` (K) and `uniqueSignatures`, and it is **never comparable to an `ecdsa-per-payment` number** — a cache-verified TPS measures a hub doing K times less crypto per payment, not a faster hub. `loadgen report` enforces all of that: a source declaring the tier without the marker (or wearing the marker without the field) is refused, and a `--soak` set that mixes the two tiers is refused with nothing written, exactly like the `ackDurability`/`boundary` refusals. The repo's headline stays `ecdsa-per-payment` fully-verified; a `verified-by-cache` report may exist, but only under its own label. The engine-direct tiered bench has the same split: `just bench`'s `t3c_verified_by_cache` (K = 8, `--verified-by-cache-k`) is printed under its own "NOT comparable to T3" heading, runs on a loadgen-hosted mirror harness (its `harness_note` says so), and never touches the thesis gate. - **`polBurn.perSource`** has one entry per merged source (`"soak"` always; `"chain"` only if you passed `--chain`, §7). A chain-bench entry carries `includedInTotal` (`true` only if its network matches the headline's) — a mismatched-network leg is still shown (for visibility) but excluded from `totalTreasuryDeltaWeiApprox`, never silently blended in. **Reference points** (each labeled — quote the label, not just the number, if you cite these elsewhere): - The SP6 gate's 1-hour local soak: **19.16 TPS sustained fully-verified full-path** (68,969/72,271 = 95.43% success), measured pre-rearch with `--in-flight 1` at a 20/s target — a pacing/config number, not a ceiling. Its decline-artifact window was **segments 27–60** (a machine-sleep-induced payer cumulative desync, ~4.5% of attempts) — since fixed by a decline→quarantine policy in the worker. **27–60 is the corrected, authoritative window** — an earlier internal note cited a different range before the correction; always use 27–60. - SP6.5 ramp ceilings (§7) — **diagnostic, NOT headline, uncapped and non-retrying**: full path ~8,257 TPS @ 512 workers, facilitator-layer ~8,309, engine-HTTP ~13–14k (a later fix raising a tokio blocking-thread cap pushed engine-HTTP to ~27.7k @ 2048 workers — see `docs/perf-notes.md`'s "Gap decomposition (SP6.5b)" section; still diagnostic, still never the headline). - Driver capability (harness measurement, NOT payment TPS): 8 payers × 100 in-flight sustains ~7,000 req/s from one laptop process against a 100ms-latency local stub, zero declines, exact client/server count match. - **Laptop → prod (`tp.mudit.blog`) full-path bursts, `/api/paid/time`, public-only reconcile, Apple M5 Max, 60s** — the concurrency-sizing anchors behind §3's table: **32 payers × 32 in-flight = 570 TPS sustained (~755 peak), 100% success (35,980/35,980)**; the same box at **8 payers × 100 in-flight plateaued at 65 TPS** (payer-starved), and at **32 × 128 (4,096 slots) collapsed** (27→12→5 decay). Lesson: scale payers, cap total slots ≲1,024. These are laptop-over-open-internet, latency-bound numbers — a driver box near the stack goes higher. - Amoy smoke: 300/303 payments, ≈0.21 POL burned. - macOS endurance prereq: AC power + lid open (see §1). Your own numbers on a live testnet are **full-path and HTTP-bound** — the full path (site→facilitator→hub over real HTTP + real chain calls) is **never comparable** to engine-direct T1–T4 tiered-benchmark numbers (those measure the verify/apply pipeline in-process, with no HTTP or chain involved at all). Every number above was `ecdsa-per-payment`; a run you label `verified-by-cache` (any `--in-flight K > 1` against a cache-enabled hub, or `loadgen merchant --sigs-per-payment K`) is a third, separately labelled tier and is **never comparable** to either — see the `headline.verification` bullet above. A `hub-ingress` run (`loadgen hubstream`, §7 — the hub's gRPC ingress driven directly with pre-encoded binary items, no facilitator, no HTTP) is a fourth tier under CLAUDE.md's sixth rule: its own label, its own verify-thread ladder, **never compared to** any of the other three, and never the headline. --- ## 7. Isolated benches and the ramp probe Four isolated component benches each swap ONE real service against a stub neighbor, isolating that service's own overhead from the rest of the pipeline — none of these produce a payment-TPS number: | `just` target | Real component | Stubbed neighbor(s) | Needs a chain? | Measures | | --- | --- | --- | --- | --- | | `sp6-bench-facilitator` | facilitator | stub engine | Anvil (facilitator boots against it; no real hub) | channel-scheme routing/serde/HTTP overhead | | `sp6-bench-site` | site | stub facilitator | no | site middleware routing/serde/HTTP overhead | | `sp6-bench-indexer` | indexer | stub engine (synthetic journal) | yes | ingest rate + query latency under a large synthetic payment journal | | `sp6-bench-chain` | dispenser + facilitator (exact scheme, no hub) | — (real chain: local Anvil, or `NETWORK=amoy`) | yes, always | exact-settle latency, confirmation-depth distribution, dispenser funding rate — **NOT payment TPS** | `just sp6-ramp LAYER=engine|facilitator|site|merchant-boundary [--steps 8,16,32 --step-duration-secs 10]` steps worker counts against exactly one layer until it hits a knee (latency spike or success-rate drop), on fresh ports (8580–8583) distinct from every other harness, and refuses to start while a soak is running against the same stack. It is **uncapped and non-retrying by design** — a diagnostic ceiling for finding where a layer breaks, never a number you'd call payment TPS or compare to the headline. `LAYER=merchant-boundary` is the ceiling-probe twin of `loadgen merchant` (§4): it drives the facilitator's `/settle` DIRECTLY with a pre-built corpus, no shim and no site process at all (every other layer either hits its target raw or fronts it with `facshim.rs`'s in-process resource shim). `LAYER=facilitator`'s shim is **settle-only** since wave 3 (2026-09-08), mirroring the real site's hot path: it quotes the facilitator ONCE at startup (re-quoting only on an `unsupported-requirements` settle) and a paid probe request costs the facilitator exactly one `/settle`. Earlier facilitator-layer ramp numbers (e.g. the ~8.3k @ 512 workers in §6's reference points, and the profile's 14.9k/s @ 32 workers) included one `/quote` AND one `/verify` per paid request — three facilitator handlings per payment — and are **not comparable** to a post-wave-3 run. Every facilitator-layer artifact now carries `harnessCorrection` (that sentence) and `requestsPerPayment: {"facilitator": 1}`; quote them together. Past a target's shed ceiling, expect fast `429`s, not decaying success or timeouts — the report's per-step `errors.errors.overloaded` count is the signal to watch; it stays SEPARATE from generic errors on purpose (a target that sheds cleanly is healthy, one that stalls is not). ### Hub-ingress tier — `loadgen hubstream` (a fourth tier, never compared) `loadgen hubstream` drives a hub's **gRPC ingress directly** (wave 3): it pre-encodes a corpus into the binary `ChannelItemV1` items the facilitator would have produced and streams them in `ApplyFrame`s over N long-lived bidi streams — no facilitator, no JSON, no HTTP anywhere on the measured path. That makes it the most engine-flattering shape short of the in-process engine-direct bench, and precisely why CLAUDE.md's **sixth** benchmark-disclosure rule makes it its own tier: a hub-ingress number is labelled `hub-ingress`, discloses `streams`, `frameItems`, `verifyThreads` and a verify-thread scaling ladder, and is **never merged with or compared to** facilitator-boundary, full-path, or engine-direct in-process numbers. Its goals are ABSOLUTE (items applied/s and hub µs/payment at a given verify-thread count on stated hardware), never a ratio to another tier. ```bash ./target/release/loadgen hubstream \ --hub-grpc http://127.0.0.1:9402 \ --hub "$HUB1_ADDRESS" --asset "$USDC_ADDRESS" --chain-id 31337 \ --payers 512 --merchants 8 --vouchers 2000000 \ --streams 8 --inflight 256 --frame-items 64 --duration-secs 30 \ --ack-durability async \ --engine-metrics-url http://127.0.0.1:8402 \ --run-tag hubladder-vt12 ``` Notes (`./target/release/loadgen hubstream --help` is authoritative): - The hub's gRPC ingress is **private-network only** (`engine serve --grpc-listen`, default `127.0.0.1:9402`, allow-listed CIDRs) — this tier runs on the stack's own box or private net, never from a laptop over the internet, and never against the hosted demo/prod hubs without the operator's say-so (§9). The synthetic corpus needs its payers registered on the hub exactly like `loadgen merchant`'s (§4) — on a bare engine use `--demo-payers`/`--demo-seed` matching `--payers`/`--seed`. - `--ack-durability` is REQUIRED and operator-asserted, AND cross-checked: the hub's handshake reports its own mode, and a disagreement refuses the run instead of writing a mislabeled artifact. `--hub`/`--asset`/ `--chain-id` are cross-checked the same way. - **`verifyThreads` is mandatory (rule 6).** Pass `--engine-metrics-url` (the driver scrapes `tp_engine_verify_threads` and the `tp_engine_ingress_batch_size` median bucket after the run) or assert `--verify-threads N`; without either the run refuses to start. The driver restarts nothing — the **ladder** is produced by restarting the engine with `--verify-threads {1,2,4,8,12}` between runs (`demo/mvp/profile/run-matrix.sh` does this) and handing every run's `final.json` to `loadgen report --soak --hub-ingress a.json --hub-ingress b.json …`, which renders them under their OWN "Hub-ingress tier" heading as a table sorted by verify threads (with the step ratio between rungs of the same shape). A hub-ingress doc passed as `--soak`, or any other tier's doc passed as `--hub-ingress`, is refused. - `--run-tag` is stamped INTO every binary item (run identity rides per item, never per frame) and becomes the artifact's `runId` + the `pay__…` payment-id prefix, so the indexer attributes the run. - The artifact is `thunderpolt-soak/2` with `boundary: "hub-ingress"`, `driver: "hubstream-grpc"`, a verbatim `boundaryDefinition` (`applied` acks per second, ECDSA-per-payment verified on the hub; replays/declines/unavailable never count), `ackCodes` (applied / replay / conflict / declined / bad-request / unavailable / in-flight — the hub's seven ack codes), `ackReasons`, `ackLatencyUs` (per-FRAME send→last-ack, microseconds — there are no HTTP stages at this boundary, `stagesMs` is empty by construction), `inflightPerStream`, `hubBatchSizeP50Bucket` (the hub's own coalescing) and the handshake (`shardCount`, `laneCapacity`). `unavailable` acks with reason `ingress lane full` mean `--inflight` exceeded the hub's `lane_capacity` — the handshake tells the driver, which warns; size in-flight at or below it. - **Gate modes** (what the gate scripts use instead of their old curl-to-the-hub legs): `--one-shot-x-payment --pay-to --min-delta 10000 [--resource /r/gate --run-tag t]` decodes the header exactly as the facilitator would and prints `{"code":"applied","seq":1,"delta":"10000","replay":false,"reason":null}` (exit 0 once ANY ack arrived — read `.code`); `--items-file items.json` (a JSON array of `{xPayment, payTo, minDelta, resource, method, runTag}`) sends ONE frame and prints the positional acks array (`idx` = position in the file), so a fresh/replay/decline mix is asserted in one call. `just sp6-report` merges one soak `final.json` with any component-bench and chain-bench JSONs (and optionally a read-only engine-direct tiered-bench JSON, which it only ever lifts two numbers out of and relabels — never re-validated as a live source, never eligible for the headline; and optionally hub-ingress documents via `--hub-ingress`, rendered under their own heading — see above) into one disclosure-validated report. If it refuses your merge, the fix is your labels, not the validator — it aborts the whole write rather than land a silently-mislabeled number. --- ## 8. Results hygiene `bench-results/` is gitignored — every soak/component/chain/report JSON (and any rendered markdown) it holds is ephemeral local-run output, not committed evidence. Anything worth keeping past your next run — a report you'll want to reference later, a soak you're treating as a baseline — gets copied **outside** `bench-results/` before the next run overwrites or reuses it. When you report numbers to anyone — a PR description, a chat message, a doc — carry the disclosure labels with them: the headline/diagnostic distinction (§6), hardware, and voucher-corpus cardinality (payers × merchants × unique vouchers — plus `inFlight` and the per-hub `reconcile` source for an amoy run). The JSON already has all of this; quote it rather than stripping it down to a bare number. --- ## 9. Demo stack (Hetzner private-net devnet) — a demo-only amendment to §1 The live demo deployment (`deploy/demo/`, plan `docs/superpowers/plans/2026-09-08-demo-pipeline.md`) deliberately runs an **always-on reference load plus several visitors at once**. That is the point of the demo, and it is safe there for four reasons — none of which hold on a shared or production stack, so **§1's "one stress run at a time per stack" rule is unchanged everywhere else**: 1. **Every lane has its own wallet pool.** The reference merchant load uses pre-deposited corpora (one per hub, seeds `REF_POOL_SEED_BASE+i`), the reference site load uses `/var/lib/thunderpolt/pools/ref-site.json`, the pay-my-site driver uses `pools/byo-site.json`, and you bring your own (`--wallets`). Nobody shares floors or counters, so concurrent runs cannot corrupt each other's cumulatives. 2. **The reference floor is sized by a rule the deploy enforces** (`deploy/demo/README.md` "Reference-load sizing rule": `REF_MERCHANT_RATE ≤ min(40 % of the measured knee per hub × hubs, 50 % of the indexer's measured ingest, the journal retention window over a 30-min outage, the funded balance per restart period, the pre-signed corpus per restart)` — 24,000/s total = 4,000/s per hub with the wave-4 defaults; `REF_SITE_RATE` 200/s). The journal rotates past a snapshot (bounded disk), so the floor is continuous. The 50–100k/s figure is a **scheduled 60 s burst**, announced on the explorer ("reference burst in progress") — expect your p95 to widen during one; measurements (knee probe, cross-box, thread grid) PAUSE the reference load and announce that in the topology `notice`. 3. **Pay-my-site runs are serialized and bounded** (`runctl`: queue depth 4, ≤ 300 s, ≤ 64 payers, ≤ 16 in-flight, ≤ 2000/s; refused during a burst). 4. **The devnet has no finite treasury**: the dispenser is open and unlimited, MockUSDC mints are free, the chain resets on redeploy. There is nothing to exhaust — the §1 budgeting rules exist for Amoy POL, which the demo never touches. What you do differently on the demo stack: - Claim a run id first — `curl -X POST $EXPLORER/api/runs -H 'content-type: application/json' -d '{"kind":"byo-driver"}'` — and pass `--run-id --report-url $EXPLORER` to `loadgen amoy` (or send `x-tp-run: ` on hand-made requests). Your run then gets its own waterfall at `$SITE_URL/benchmark#/run/` with per-stage p50/p95, the bottleneck component and who owns it (ours / yours / network), your client RTT (from the segments you post) and your decline taxonomy. Untagged traffic is still counted, but only as `_untagged` in the whole-system view. - The tier badge is derived server-side from how your run was driven (`full-path · site boundary` for `loadgen amoy` and agents; `full-path · BYO site` for pay-my-site; `facilitator-boundary` for `loadgen merchant`). The page never puts two tiers on one axis and never shows a server-observed applied/s as a client headline — the same disclosure rules as §6. - Bursts you launch yourself still follow §1's *shape*: bounded duration, an explicit `--target-rate`, step up only while success stays ~100 %. Uncapped runs belong to the ramp probe against your own local stack (§7) — on the demo they only produce shed `429`s that pollute everyone's waterfalls. - `just demo-local` runs the whole shape on your laptop (ports 8700–8799) if you want to rehearse before touching the shared demo. - **Scenarios (wave 5) move the facilitator / site / driver tiers between boxes while the demo stays up.** A `just demo-scenario ` (or the multi-hour `just demo-scenarios` study) re-renders which box runs which facilitators and sites and where the drivers live; it never stops a hub, never restarts the anvil and never resets the chain — your channels, your deposits and your run ids survive every scenario. Visitors keep working through nginx the whole time: the edge re-points its `tp_fac` / `tp_sites` upstreams to the new placement and a request that lands during the hand-over just sees a facilitator answer from another box. The topology `notice` (a banner on `/benchmark`) announces each re-render — `scenario applying …` — and clears when the placement has verified; the reference load is paused (and confirmed paused) during a study cell, so your own run's numbers during a study reflect a QUIETER fleet than usual, not a representative one. Read scenario rows as what they are: `reported` placement studies with an experiment tier badge (`facilitator-boundary` or `full-path-site`) and per-component CPU slices that are explicitly non-additive — never a new headline.