SWE-Marathon · October 2026 · 4 tasks
SWE-Marathon Task Audit
All 4 audited tasks have a defect that can flip the pass/fail reward.
In BioFabric, the agent writes the code that compares its own output to the answer key, so a pass doesn't prove parity. In Next.js, test results overwrite each other by title and skipped tests count as passes. In the Slack and S3 clones, the verifier installs the agent's own packages into the Python that writes the score, and the agent has open internet to a public copy of the solution. Each finding says how an honest solution loses or how an agent could game the score, and cites the file and line it rests on.
Sample: 4 of SWE-Marathon's 20 tasks: Next.js on Vite (nextjs-vite-rewrite), the BioFabric port (biofabric-rust-rewrite), the Slack clone (slack-clone) and the S3 clone (s3-clone) (abundant-ai/swe-marathon@dd4d2bb)
Each task's files were reviewed by an independent reviewer: the instruction, task.toml, environment, verifier, CUA rubric and reference solution, against the benchmark's own CONTRIBUTING.md and task-implementation rubric. Every finding cites the file and line it comes from, so it can be checked by reading. Tasks were not executed: trials need Modal and up to 90-minute verifiers, and the two clones also run an Anthropic-powered computer-use grader. Findings whose effect depends on runtime behaviour are kept as leads ('needs a run'). An analyst then re-read every finding against the code and rewrote it in plain words: what's wrong, how an agent could game the score or how an honest solution loses, the effect on the score, the evidence and a suggested fix. Where the analyst checked a specific claim, changed a severity or merged or dropped a finding, an 'Analyst note' says so.
Pilot: rubric findings come from one reviewer, not a multi-reviewer vote.
- 4/4 tasks have a critical or major finding
- 15 critical findings
- 0 have no blocking findings
Flagged (critical or major) 4 tasks Critical findings 15 Major findings 17
4 tasks, one square each
- Critical or major finding
- No blocking findings
- Not fully audited
Across the benchmark
The agent runs as root in the container that is later graded, and nothing in these tasks switches users. Anything it leaves behind (a patched interpreter, a startup hook, a background process) is present when tests/test.sh runs. This holds for every task and is reported once here. Per-task findings under 'grading state' are kept only where the verifier itself makes it worse, for example by installing the agent's own requirements.txt into the Python environment that then computes the score (slack-clone, s3-clone) or running the agent's npm install and build during grading (nextjs-vite-rewrite).
Every task's tests/ and reference solution/ are public in the GitHub repository, and several tasks open their encrypted hidden suites with a passphrase written in the same tests/ (biofabric-rust-rewrite among the four here). Where a task gives the agent open internet during its run (slack-clone, s3-clone), the agent can fetch both; that is reported under the task.
Defect types
How many tasks each blocking problem appears in. Select a row to filter the task list.
| Defect and check | Tasks |
|---|---|
Tasks
biofabric-rust-rewrite The agent writes the code that compares its output to the answer key 2 critical · 2 major flagged
- Staticpass13 checks, 2 with findings (0 blocking)
- Rubricfail34 criteria, 9 failed by a single reviewer: instruction_scoring_disclosure, environment_reproducibility, grading_state_isolation, verifier_hang_handling, correctness_gate_coverage, false_pass_risk, partial_score_contract, test_coverage_gaps, hidden_material_public_exposure (2 minor)
The agent writes the code that compares its output to the answer key
rubric · correctness_gate_coverage · correctness
What's wrong
The test cases are fixed, but they don't do the checking themselves. Each of the 407 parity tests is a one-line call into crates/core/tests/parity_tests/runners.rs, and that file holds the real step: run the library, load the golden file, assert the two are equal. In the starter those functions are todo!() stubs, the instruction tells the agent to implement them, and tests/test.sh keeps the agent's copy when it restores every other test file. So the agent writes both the implementation and the check on it. Ten subsystems are exercised only through these runners: hierarchical-DAG, control-top, node-cluster, set, world-bank, node-similarity, link-group and fixed-order layouts, BIF roundtrip and display options (about 176 of the ~560 tests). analysis_tests.rs, cli_tests.rs and hidden_tests.rs never call them.
How an agent could reward-hack it
An agent implements the default layout, alignment, analysis, I/O and the CLI, but can't get the hierarchical-DAG or cluster layouts to match Java. Its runners.rs returns early for those configurations, or compares the golden file with itself, instead of asserting parity. Every test still passes. With runners that do nothing at all, the 407 parity tests pass with no library code.
Effect on score: Reward 1 and partial_score 1.0 with about a third of the required subsystems missing. Runners that skip every comparison give partial_score of about 0.73 with no implementation (reward 0).
Analyst note: Analyst checked: counted the macro-generated tests in the starter: 407 go through runners.rs; of those, about 176 cover layouts no other test file calls (hierdag 18, control-top 48, cluster 36, set 6, similarity 18, link-group 12, fixed-order 9, roundtrip 12, display 8, extract 9). grep of analysis_tests.rs, cli_tests.rs and hidden_tests.rs finds none of those layouts. Merges the reviewer's separate false_pass_risk finding, which described the same mechanism.
Evidence
tests/test.sh:99-105 saves the agent's runners.rs, restores crates/core/tests from /private, then copies the agent's runners.rs back environment/starter/crates/core/tests/parity_tests/macros.rs:18 every parity macro calls crate::runners::run_noa_test / run_eda_test / run_bif_test environment/starter/crates/core/tests/parity_tests/runners.rs:327, 333, 339 run_noa_test, run_eda_test, produce_bif_string are todo!() stubs instruction.md: "You must implement ... All test runners in crates/core/tests/parity_tests/runners.rs" crates/cli/tests/cli_tests.rs:5 says algorithm tests live in crates/core/tests; it never passes --algorithm
Suggested fix: Ship runners.rs with the task and restore it like the other test files; have the agent implement only library functions the runners call.
A one-line change to Cargo.toml stops the real tests from running
rubric · grading_state_isolation · verifier
What's wrong
Before grading, tests/test.sh deletes the test folders and restores clean copies from its private backup, so the agent can't edit the tests. It doesn't restore the Cargo manifests (Cargo.toml, crates/core/Cargo.toml, crates/cli/Cargo.toml) or any build script, and those files decide which test files Cargo compiles. The pass check then asks only that cargo exits 0, nothing failed and at least one test passed. It never checks which test suites ran or how many tests there should be.
How an agent could reward-hack it
An agent sets autotests = false in a crate manifest, which tells Cargo not to discover the files under tests/. The restored parity, analysis, CLI and hidden tests are on disk but never compiled. The only tests left are unit tests in crates/*/src, which the agent writes; one trivial passing test satisfies the check.
Effect on score: Reward 1 and partial_score 1.0 with no implementation.
Analyst note: Analyst checked: test.sh:99-114 restores only test sources and goldens. autotests is standard Cargo manifest behaviour; not run.
Evidence
tests/test.sh:99-114 restores crates/core/tests, crates/cli/tests, the goldens and hidden_tests.rs; no Cargo.toml or build.rs is restored tests/test.sh:119 cargo test --workspace --no-fail-fast tests/test.sh:134 reward 1 if cargo exit 0, failed == 0 and passed > 0
Suggested fix: Restore the manifests and build scripts too, and fail the run unless each expected suite (parity_tests, analysis_tests, cli_tests, hidden_tests) reports its known test count.
Padding with trivial unit tests inflates the partial score
rubric · partial_score_contract · scoring
What's wrong
partial_score is passed / (passed + failed) over every test cargo runs, including unit tests in crates/*/src. The agent is allowed to edit those files and test.sh doesn't restore them, so the agent controls part of the denominator.
How an agent could reward-hack it
An agent's port passes 10% of the ~560 real tests. It adds 2000 one-line #[test] functions to crates/core/src that always pass. partial_score becomes about 0.80.
Effect on score: partial_score can be pushed toward 1.0 for any solution. Reward stays 0, so only leaderboards that read partial_score are misled.
Evidence
tests/test.sh:72-73 partial_score = passed / (passed + failed) tests/test.sh:123-124 passed and failed are summed over every 'test result:' line in the cargo output
Suggested fix: Compute partial_score only over the named suites (parity_tests, analysis_tests, cli_tests, hidden_tests), with their expected test counts as the denominator.
One hanging test wipes out all partial credit
rubric · verifier_hang_handling · verifier
What's wrong
test.sh runs cargo test with no timeout of its own and no per-test limit. If any test hangs, the 1200 s verifier timeout kills the whole script before it reaches the step that writes the final metrics, so metrics.json keeps the placeholder written at the start (partial_score 0).
How an honest solution loses
A good port passes ~550 of ~560 tests but has an infinite loop in one layout algorithm on one input. Instead of partial_score ~0.98 it records 0, the same as an empty submission.
Effect on score: partial_score drops to 0 for a near-complete solution. Reward is 0 either way.
Evidence
tests/test.sh:119 cargo test with no timeout task.toml:60 [verifier] timeout_sec = 1200 tests/test.sh:82 metrics.json is first written as 'initialized' with partial_score 0; the final write is after cargo returns
Suggested fix: Wrap cargo test in timeout(1) well inside the verifier budget and add a per-test timeout, so a hang fails one test instead of the run.
The image stops building once either Java reference repo gets a new commit
rubric · environment_reproducibility · environment · severity set by analyst (reviewer said major)
What's wrong
The Dockerfile clones BioFabric and AlignmentPlugin with --depth 1 (only the newest commit) and then checks out a fixed older commit. That works only while the fixed commit is still the newest one. As soon as either repo moves on, the checkout fails and the image can't be built. Separately, there is no Cargo.lock in the starter; dependency versions are resolved when the image is built, so two builds on different dates can compile against different crate versions with the same pinned rustc.
How it plays out
Upstream pushes one commit to abundant-ai/BioFabric. The next image build fails at git checkout, and no trial of this task can run until the Dockerfile is fixed.
Effect on score: No score at all for anyone while the build is broken. Not broken today.
Analyst note: Analyst checked on 2026-10-29: both pinned commits are currently the HEAD of their repos (git ls-remote), so the image builds today. Re-graded from major to minor because nothing is wrong until upstream moves.
Evidence
environment/Dockerfile:24-27 git clone --depth 1 ... && git checkout <sha> environment/Dockerfile:113-116 cargo fetch at build time; no Cargo.lock under environment/starter
Suggested fix: Fetch the pinned commit directly (git fetch --depth 1 origin
The 'held-out' tests, their passphrase and the full Rust oracle are public
rubric · hidden_material_public_exposure · contamination
What's wrong
task.toml calls the hidden suite held-out and not provided to the agent, and the image ships it encrypted. But the passphrase is written in plain text in tests/test.sh, the Dockerfile and solve.sh; the hidden tests and their input networks are also in plain text under environment/hidden/; and the full oracle Rust port is a public GitHub repository at a pinned commit.
How an agent could reward-hack it
A model trained on GitHub after this repository was published, or an agent that can reach GitHub, has the oracle implementation and every hidden test, and can reproduce the oracle instead of porting the Java.
Effect on score: Reward 1 and partial_score 1.0 without doing the port, for a contaminated model. The agent's own network is an allowlist during the run, so live fetching depends on what the allowlist permits.
Analyst note: Folds in the static check that found the passphrase at tests/test.sh:85. Public tests/ and passphrases apply to every task and are also noted once at the top of the report.
Evidence
task.toml:49-51 'held-out ... not provided to the agent' tests/test.sh:85, environment/Dockerfile:119, solution/solve.sh:13 passphrase in plain text environment/hidden/ hidden_tests.rs and the .gw/.align inputs in plain text environment/Dockerfile:18-19 clones the oracle from github.com/abundant-ai/BioFabric-rs
Suggested fix: Keep the held-out suite and oracle out of the public repository, or stop describing them as held-out.
task.toml promises per-suite metrics that are never written
rubric · instruction_scoring_disclosure · instruction · severity set by analyst (reviewer said major)
task.toml says a per-suite breakdown of passed/failed/ignored counts is written to metrics.json. test.sh writes only workspace-wide totals. task.toml also never explains partial_score, so a reader can't tell that each of the 407 parity cases weighs the same as each of the 20 held-out cases.
Effect on score: No effect on reward or partial_score; readers of metrics.json and the leaderboard are misinformed.
Analyst note: Re-graded from major to minor: it misdescribes metadata but doesn't change any score.
Evidence
task.toml:51-53 'a breakdown of per-suite passed/failed/ignored counts is written to metrics.json' tests/test.sh:63-74 writes only passed, failed, ignored, reward and partial_score totals
Suggested fix
Write per-suite counts as promised and describe partial_score in task.toml.
The hidden 'no-shadow' tests are copies of the default tests
rubric · test_coverage_gaps · verifier
What's wrong
The three hidden_mmus_noshadow tests make exactly the same library calls as hidden_mmus_default, so no-shadow behaviour on the held-out network is never tested, although task.toml lists it as part of the held-out suite. Separately, nothing checks the instruction's rule that the public API must stay as defined, beyond the restored tests compiling.
How a broken solution still passes
A solution whose no-shadow BIF output is wrong on large networks passes all three hidden no-shadow tests.
Effect on score: Small: three tests that can't fail independently of their default twins.
Evidence
environment/hidden/hidden_tests.rs:702-731 hidden_mmus_noshadow_* call run_default_layout and produce_bif_default instruction.md:30 'Do not modify ... the public API surface'
Suggested fix: Make the no-shadow hidden tests pass the no-shadow display option, or drop them from the held-out list.
nextjs-vite-rewrite Anti-cheat zeroes any solution whose code isn't in a src/ folder 4 critical · 3 major flagged
- Staticpass13 checks, 0 with findings (0 blocking)
- Rubricfail34 criteria, 8 failed by a single reviewer: instruction_test_alignment, grading_state_isolation, verifier_hang_handling, anticheat_false_zero, forbidden_approach_detection, correctness_gate_coverage, partial_score_contract, test_coverage_gaps (1 minor)
Anti-cheat zeroes any solution whose code isn't in a src/ folder
rubric · anticheat_false_zero · anti-cheat
What's wrong
One of the three anti-cheat checks requires /app/vinext/src to exist and to contain at least two .ts/.tsx/.js/.jsx files that use Vite plugin hooks. The instruction asks only for package.json and the source files in /app/vinext, never a src/ folder, and the visible copy of run_tests.py has no such check. Files are also counted by basename, so several hook files all named index.ts count as one.
How an honest solution loses
An honest plugin keeps its TypeScript in lib/ or at the package root and passes every Playwright test. The anti-cheat check fails, and reward is forced to 0.
Effect on score: Reward 1 → 0 for a working solution (partial_score 371/372 ≈ 0.997).
Analyst note: Analyst checked: tests/run_tests.py:188-192 returns failure when /app/vinext/src does not exist; instruction.md does not name a src/ directory.
Evidence
tests/run_tests.py:188-192 returns failure when /app/vinext/src does not exist tests/run_tests.py:201, :207 only .ts/.tsx/.js/.jsx, counted by basename, need 2 or more tests/run_tests.py:695-696 any regression (anti-cheat) failure sets reward 0 instruction.md:20 names /app/vinext, package.json and the bin field; no src/
Suggested fix: Search the whole of /app/vinext (excluding node_modules) for hook usage, count files by full path, and state the layout in the instruction if it matters.
Test results overwrite each other by title, and skipped tests count as passes
rubric · correctness_gate_coverage · correctness
What's wrong
Two separate bugs in how tests/run_tests.py collects Playwright results: 1. Results are stored by test title alone, and the three test projects (App Router, Pages Router, Pages Router production) are merged into one dictionary. 15 titles exist in both an App Router spec and a Pages Router spec, for example "onRequestError() is called when a route handler throws" and "dynamic route renders with params". For each of them, the App Router result is thrown away and the Pages Router result is counted twice. 2. Seven tests in the App Router specs are test.skip, but they are in the graded list. Playwright reports a skipped test as ok, so they count as passes whenever the server starts. They cover intercepting-route modals, ISR cache HIT/STALE, and revalidateTag/revalidatePath.
How a broken solution still passes
A solution whose App Router onRequestError, catch-all routes and ISR revalidation don't work, but whose Pages Router equivalents do, passes every graded item.
Effect on score: Reward 1 and partial_score 1.0 with 15 App Router behaviours broken and 7 never run.
Analyst note: Analyst checked: counted 15 titles graded in both projects from tests/filter.json, and 7 test.skip specs whose titles are in filter.json (the reviewer said 4).
Evidence
tests/run_tests.py:425 results[spec title] = ok tests/run_tests.py:629 all_pw_results.update(...) per project, in order app-router, pages-router, pages-router-prod tests/filter.json: 372 graded items, 357 unique titles; 15 shared between App Router and Pages Router projects tests/e2e/app-router/isr.spec.ts:22, :39, :59, :216, :272, :306 and advanced.spec.ts:62 are test.skip and all listed in filter.json
Suggested fix: Key results by project and file as well as title, and treat skipped tests as failures (or remove them from filter.json).
The agent's build script runs during grading and can rewrite the scorer
rubric · grading_state_isolation · verifier
What's wrong
During verification, tests/test.sh runs the agent's npm install and npm run build as root, after the grader's own files have been uploaded. The hardening that follows restores and hash-checks only Playwright and the global node install (/usr/local), and kills processes whose working directory is under /app. It doesn't protect or re-check the Python scorer that then decides the reward: tests/run_tests.py, filter.json, utils.py, /usr/bin/python3 and its packages.
How an agent could reward-hack it
The agent's build step changes the graded test list or the scoring script on disk, then exits normally. test.sh trusts the modified files when it computes the reward.
Effect on score: Reward 1 and partial_score 1.0 with no working framework.
Evidence
tests/test.sh:178-180 npm install --offline && npm run build, as root, during grading tests/test.sh:211-213 restore_pristine / hash_runtime / verify_pristine cover /usr/local only tests/test.sh:217 python3 $TESTS_DIR/run_tests.py $TESTS_DIR/filter.json ... computes the reward
Suggested fix: Build the agent's code before uploading tests/, or hash-check run_tests.py, filter.json and the Python runtime right before scoring.
The hidden `use cache` tests require caching to be off in dev mode
rubric · instruction_test_alignment · instruction
What's wrong
The instruction asks for "`use cache` for component caching". The hidden use-cache tests, which have no counterpart in the agent's visible e2e folder, fetch the page twice 50 ms apart in dev mode and require a newer timestamp the second time, meaning the directive must not cache in dev. Only the reference solution's own choice to bypass the cache when NODE_ENV=development makes them pass; the instruction never says so. The same tests never check that `use cache` caches anything, so the stated requirement itself is not graded: a no-op directive passes all five.
How an honest solution loses
An honest solution implements `use cache` with the 1-second profile the fixture declares, in dev and production. The second request within 50 ms returns the cached timestamp, and both dev-mode tests fail.
Effect on score: Reward 1 → 0 for a correct caching implementation (partial_score loses 2 of 372).
Evidence
tests/e2e/app-router/use-cache.spec.ts:18-35 and :81-95 two requests 50 ms apart, expect the second timestamp to be greater instruction.md:13 "`use cache` for component caching" solution/vinext/src/shims/cache-runtime.ts:312-370 skips the shared cache when NODE_ENV=development
Suggested fix: State the dev-mode behaviour in the instruction, and add a production test that checks `use cache` actually caches.
Reusing Next.js's own code from /opt/reference isn't detected
rubric · forbidden_approach_detection · anti-cheat
What's wrong
The instruction forbids using the next package. The only check looks for next in package.json and in /app/vinext/node_modules. The full next@16.0.11 dist tree that the instruction provides 'for reference only' sits at /opt/reference/node_modules/next and is still there when the tests run. Nothing scans for copied Next.js code or for imports from outside /app/vinext.
How an agent could reward-hack it
An agent copies Next.js's compiled modules into its own tree, or loads them at runtime by absolute path from /opt/reference, instead of implementing routing and rendering. The anti-cheat checks all pass.
Effect on score: Up to reward 1 and partial_score 1.0 for the forbidden approach, as far as the reused code makes tests pass.
Evidence
instruction.md:18, :30 forbid the next package; :24 provides /opt/reference/node_modules/next tests/run_tests.py:151-154 checks package.json dependency keys and /app/vinext/node_modules/next only
Suggested fix: Remove /opt/reference before grading, and fail the run if the built vinext resolves any module from /opt/reference or contains Next.js source fingerprints.
A solution caught by anti-cheat keeps a 0.997 partial score
rubric · partial_score_contract · scoring
What's wrong
When an anti-cheat check fails, reward is set to 0, but partial_score has already been computed from the full Playwright run and is kept. The anti-cheat checks are just 3 of 372 items in the partial score.
How an agent could reward-hack it
A submission that simply depends on the real next package (the exact approach the anti-cheat exists to catch) passes every test. It gets reward 0 but partial_score (369 + 2)/372 = 0.997.
Effect on score: Leaderboards that read partial_score rank a detected forbidden solution as near-perfect.
Analyst note: Analyst checked: tests/run_tests.py:689-691 computes partial_score over all items before the regression (anti-cheat) result sets reward to 0, so a caught submission keeps its pass fraction.
Evidence
tests/run_tests.py:689-691 partial_score computed over all new + regression items tests/run_tests.py:695-696 anti-cheat failure sets reward 0 afterwards
Suggested fix: Set partial_score to 0 whenever an anti-cheat check fails.
A slow App Router run loses every App Router result, including the passes
rubric · verifier_hang_handling · verifier
What's wrong
Each Playwright project runs under a 1200 s limit. If it runs out of time, the output is discarded and every test in that project is recorded as failed, including tests that had already passed. The code's own comment says even the reference solution's App Router run can exceed 600 s.
How an honest solution loses
A solution passes ~240 of 266 App Router tests, but ~14 broken tests each wait out the 45 s per-test timeout and are retried once. The project goes past 1200 s and all 266 App Router results become failures.
Effect on score: partial_score falls from ~0.93 to ~0.3, below a weaker solution whose failing tests fail fast. Reward is 0 either way.
Evidence
tests/run_tests.py:404-412 playwright_timeout_sec = 1200; on TimeoutExpired stdout is set to '' tests/run_tests.py:420-430 results come only from the JSON report, which is never written on timeout
Suggested fix: Use Playwright's own global timeout so the JSON report is still written, or keep partial results from a streaming reporter.
App Router production builds are never graded
rubric · test_coverage_gaps · verifier
What's wrong
The instruction requires vinext build / vinext start and the App Router, but only the Pages Router fixture is ever production-built. App Router production builds, RSC bundling and serving are never tested. SSG with getStaticPaths is checked by one production test that only looks for the slug in the body.
How a broken solution still passes
A solution whose App Router works only under vinext dev loses nothing.
Effect on score: No score effect for missing App Router production support.
Evidence
instruction.md:6, :8 require vinext build/start and the App Router tests/run_tests.py:573-588 production build of pages-basic only
Suggested fix: Add a production-mode App Router project.
s3-clone Any amazonaws.com URL in a comment or test zeroes the run 4 critical · 7 major flagged
- Staticpass13 checks, 1 with findings (0 blocking)
- Rubricfail34 criteria, 13 failed by a single reviewer: instruction_test_alignment, instruction_clarity, instruction_scoring_disclosure, network_policy_enforcement, grading_state_isolation, verifier_hang_handling, anticheat_false_zero, correctness_gate_coverage, partial_score_contract, cua_rubric_matches_instruction, cua_judge_reliability, cua_rubric_benchmark_standard, test_coverage_gaps (2 minor)
Any amazonaws.com URL in a comment or test zeroes the run
rubric · anticheat_false_zero · anti-cheat
What's wrong
tests/anti_cheat.py rejects any http(s) URL on an amazonaws.com host (except the XML namespace) in every .py/.md/.txt/.js file under /app, comments included. An S3 clone is likely to cite AWS: the task's own seed credentials are AWS's documentation example keys, and AWS's SigV4 examples use URLs like https://examplebucket.s3.amazonaws.com/test.txt. A test double named FakeS3 also trips the fake-s3 rule.
How an honest solution loses
An honest agent adds a SigV4 known-answer test using AWS's documented presigned URL, or a comment noting what real S3 returns in its Location header. Anti-cheat fails, and the run exits before UX grading.
Effect on score: Reward 1 → 0 and partial_score → 0 for a working solution.
Analyst note: Analyst checked: tests/anti_cheat.py:83 matches any http(s) URL on an amazonaws.com host except the s3.amazonaws.com/doc/ namespace. The reviewer's second point, that the moto rule zeroes solutions built on moto, is left out: using moto as the storage engine is reusing an S3 implementation, which the anti-cheat reasonably blocks.
Evidence
tests/anti_cheat.py:83 https?://(?!s3\.amazonaws\.com/doc/)[a-z0-9.-]*amazonaws\.com, comments included tests/anti_cheat.py:59 \bfake[-_]?s3\b, case-insensitive tests/test.sh:172 any hit exits with reward 0
Suggested fix: Flag only network calls to AWS endpoints in code, not URLs in comments, docs or test vectors.
The verifier installs the agent's requirements into the Python that writes the score
rubric · grading_state_isolation · verifier
What's wrong
test.sh puts the agent's virtual environment (/opt/venv) first on PATH, then pip-installs the agent's requirements.txt into it during grading. That same python3 runs the anti-cheat scan, the reward/metrics writer and the final blend. Only the pytest gates use the isolated /opt/verifier-venv. The agent already runs as root in this container; the difference here is that the verifier itself installs agent-chosen packages into the interpreter that decides the score.
How an agent could reward-hack it
A package in the agent's requirements.txt installs a Python start-up hook into the venv. Every later python3 run loads it, so the anti-cheat and reward writer can be made to report a pass.
Effect on score: Reward 1 and partial_score 1.0 regardless of the work.
Analyst note: Folds in the static check that flagged pip install at test.sh:107.
Evidence
tests/test.sh:19 PATH starts with /opt/venv/bin tests/test.sh:107 pip install -r requirements.txt during grading (needs network at grading time too) tests/test.sh:148 anti-cheat, tests/test.sh:257 reward writer, tests/cua_blend.sh:38 blend all run on that python3
Suggested fix: Run the anti-cheat, reward writer and blend with /opt/verifier-venv/bin/python -I, and install the agent's dependencies before tests/ is uploaded.
Hidden response formats the instruction never states
rubric · instruction_test_alignment · instruction
What's wrong
Each of these fails a whole test gate, and the correctness half is all-or-nothing: - /_admin/stats must return top-level keys stats and per_tenant, with rows keyed tenant (the instruction says only 'global and per-tenant counts') - the audit endpoint must return its list under the key events (no key is named) - a wildcard CORS AllowedOrigin must echo the request Origin rather than return *, which is what real S3 does - version ids must be 8+ characters and Suspended buckets must use VersionId 'null' (neither is documented, and the instruction says only documented behaviour is required) - notification records must use AWS's nested s3.bucket.name / s3.object.key shape (the instruction says only 'bucket name, and object key') - minted access keys must start with AKIA and be 20 characters, with a 40-character secret
How an honest solution loses
An honest solution returns /_admin/stats as {"global": ..., "tenants": [...]}, which matches the instruction's wording. The admin gate fails.
Effect on score: Each item alone takes reward 1 → 0 and caps partial_score at 0.5 × ux.
Analyst note: Analyst checked: tests/test_admin.py:70-71 asserts the `stats` and `per_tenant` keys and a `tenant` field per row; instruction.md describes the endpoint only as global and per-tenant counts.
Evidence
tests/test_admin.py:70-71, :47-49; tests/test_audit.py:43; tests/test_cors.py:94 tests/test_versioning.py:43, :149; tests/test_notifications.py:126 instruction.md:11, :93, :123, :129-130
Suggested fix: Document each of these formats in the instruction.
The agent has open internet, and the full solution is public on GitHub
rubric · network_policy_enforcement · environment
What's wrong
task.toml gives the environment public network access and sets no stricter agent setting, so the agent has open internet for the whole run. The stated justification covers pip installs only. The public swe-marathon repository holds this task's complete reference solution (about 5,300 lines of Python), which the anti-cheat doesn't fingerprint, and the public tests/ reveal every unstated format listed above. task.toml records the oracle scoring ux 1.000 in 5 of 5 trials.
How an agent could reward-hack it
The agent downloads solution/ from the public repository and submits it.
Effect on score: Reward 1 and partial_score 1.0 for a copied oracle.
Evidence
task.toml:113 [environment] network_mode = "public"; no [agent] network_mode task.toml:14 open_internet_justification covers pip only task.toml:84-85 oracle ux 1.000 in 5/5 trials tests/anti_cheat.py scans only for MinIO, moto, AWS URLs and similar
Suggested fix: Set [agent] network_mode to an allowlist of package registries, and add the solution's own fingerprints to the anti-cheat.
Several S3 features the instruction requires are never tested
rubric · correctness_gate_coverage · correctness · severity set by analyst (reviewer said critical)
What's wrong
These stated requirements have no test, so a solution that skips them loses nothing: - ListObjectsV2 start-after, and CopyObject x-amz-tagging-directive - the 15-minute clock-skew window (403 RequestTimeTooSkewed) - notifications for CompleteMultipartUpload and CopyObject, and the eventSource/eventVersion fields - quotas on CompleteMultipartUpload and CopyObject, and the cross-tenant CopyObject source check (only PutObject quota is tested) - lifecycle delete markers on versioned buckets
How a broken solution still passes
A solution that never enforces clock skew, ignores quotas on multipart uploads and copies, and lets one tenant copy another tenant's objects passes every gate.
Effect on score: Reward 1 and partial_score 1.0 with these features missing, including a cross-tenant access check.
Analyst note: Re-graded from critical to major: these are edge features rather than the core of the task, apart from the cross-tenant copy check.
Evidence
instruction.md:35, :37, :81, :93, :138 no test passes StartAfter or TaggingDirective, sends a skewed date, or copies across tenants
Suggested fix: Add a test per listed behaviour; the cross-tenant copy check matters most.
Reward 1 needs 7 of 7 PASS verdicts from a single judge run
rubric · cua_judge_reliability · cua-rubric
What's wrong
Reward is 1 only if all 7 UX verdicts are PASS, in one computer-use session with one judge sample each. The judge runs at its default temperature rather than 0, and several items ask for pixel-level measurements (such as a hover colour change of at least 0x08) that the agent has to estimate from screenshots, because reading the CSS doesn't count.
How an honest solution loses
A good console is judged PARTIAL on the hover-delta item in one run. Reward is 0 for that trial; a re-run might give 1.
Effect on score: Reward flips between 0 and 1 across runs of the same solution.
Evidence
tests/cua_blend.sh:62 reward 1 only if ux >= 1.0 tests/cua_verifier.py:34 no temperature for opus-4-7 tests/cua_config.json:8 150 turns, fully allocated with a 5-turn buffer (lead: may run out; needs a run)
Suggested fix: Take several judge samples per item, and drop pixel thresholds the agent can only guess from screenshots.
The UX rubric demands styling the instruction never asks for
rubric · cua_rubric_matches_instruction · cua-rubric
What's wrong
Reward 1 needs every rubric item to PASS, and several depend on standards the instruction never states (it lists only testids and behaviours): - bucket_list needs 2+ cells per row (name plus created-at) and empty-state copy matching specific patterns - auth and create_bucket need error text in the red family with a red border or icon; the instruction asks only for a visible error - realism needs striped or bordered rows, header styling and a non-grey call-to-action at least 80×28 px
How an honest solution loses
An honest console lists bucket names only. bucket_list is judged PARTIAL, ux is 0.923, and reward is 0 (partial_score 0.96).
Effect on score: Reward 1 → 0 for a solution that meets the instruction.
Evidence
tests/rubric.json:9, :15, :45 instruction.md:101-111
Suggested fix: Put every rubric cue in the instruction, or drop the cues it doesn't state.
Quota fields look required, but the tests leave them out
rubric · instruction_clarity · instruction
What's wrong
The instruction gives the tenant-create body as {name, quota_bytes: int|null, quota_buckets: int|null} and never says the quota fields may be omitted. The tests and fixtures post only {"name": ...}. A server that treats the listed fields as required rejects those requests.
How an honest solution loses
An honest solution validates the body with a model where quota_bytes is int | None with no default. It returns 422 for {"name": ...}, and the tenants, policy and admin gates all fail.
Effect on score: Reward 1 → 0 and partial_score at most 0.5 × ux, for one of two valid readings.
Evidence
instruction.md:123 tenant-create body
tests/test_tenants.py:48, tests/test_admin.py:77 POST {'name': name} onlySuggested fix: Say that missing quota fields mean null.
task.toml calls the 50/50 blend 'the trial reward', but the reward is pass/fail
rubric · instruction_scoring_disclosure · instruction
task.toml says the trial reward is 0.5 × correctness + 0.5 × ux. In the code, reward.txt is 1 only if every gate passes and ux is 1.0; the blend is only partial_score, and its correctness term is the all-gates result, not a fraction. A stage note kept in the final metrics.json refers to a 'weighted_sum aggregator' that doesn't exist.
Effect on score: No score changes, but readers and leaderboards are told the published number means something it doesn't.
Evidence
task.toml:41, :91 'The trial reward is 0.5 * correctness + 0.5 * ux' tests/cua_blend.sh:61-62 tests/test.sh:330-336 stage_note
Suggested fix
Describe reward and partial_score separately and correctly in task.toml and the stage note.
partial_score gives no correctness credit unless every gate passes
rubric · partial_score_contract · scoring
What's wrong
The blend takes its correctness half from the binary reward.txt and throws away the gate fraction test.sh computed, so passing 21 of 22 gates earns the same correctness credit (0) as passing none.
How an honest solution loses
A solution passing 21/22 gates with ux 0.6 scores 0.30. One passing 0/22 gates with a polished console at ux 1.0 scores 0.50: the much weaker solution ranks higher.
Effect on score: partial_score misranks solutions on leaderboards.
Evidence
tests/cua_blend.sh:23 correctness = reward.txt tests/cua_blend.sh:61 partial_score = 0.5*correctness + 0.5*ux tests/test.sh:327 gate fraction, overwritten
Suggested fix: Use the gate or test pass fraction as the correctness term of partial_score.
One unanswered request can hang grading and lose the UX half
rubric · verifier_hang_handling · verifier
What's wrong
The pytest gates run with no overall time limit and no pytest-timeout, and many test HTTP calls have no timeout. If the agent's server accepts a connection on one route but never answers, the whole verifier blocks until its 5400 s limit. The gate metrics are never written and the UX stage never runs.
How an honest solution loses
A server with a working console deadlocks on one admin route (the instruction itself warns about deadlocks). Grading hangs, and the console's UX half is never scored.
Effect on score: Up to 0.5 of partial_score lost for one stuck request. Reward is 0 either way.
Evidence
tests/test.sh:192 pytest per gate with no wall-clock wrapper environment/Dockerfile:100-105 no pytest-timeout in the verifier venv tests/test_admin.py:18, tests/test_audit.py:37 requests without a timeout
Suggested fix: Give every HTTP call a timeout and wrap each gate in timeout(1).
The UX rubric breaks the benchmark's own CUA grading rules
rubric · cua_rubric_benchmark_standard · cua-rubric
The benchmark's task-implementation standard asks for binary PASS/FAIL items with equal weights and per-item results in metrics.json. This task scores PARTIAL as 0.5, weights one item 0.5, and keeps per-item verdicts only in a side file.
Effect on score: No direct score change; results aren't comparable with tasks that follow the standard.
Evidence
tests/cua_verifier.py:434 PARTIAL = 0.5 tests/rubric.json:26 tenant_visibility weight 0.5 tests/cua_blend.sh:70-80 only scalar fields added to metrics.json
Suggested fix
Bring the rubric and metrics output in line with the benchmark's standard.
Process respawn, key deletion and audit logging are never tested
rubric · test_coverage_gaps · verifier
Further stated requirements nothing checks: start.sh respawning dead children; deleting console access keys (with 409 CannotDeleteCurrentAccessKey) and console logout; 404 NoSuchTenant; audit lines for mutating admin requests and the audit.log file; and the notification eventSource and eventTime fields.
Effect on score: No score effect for these missing requirements.
Evidence
instruction.md:5, :93, :99, :124, :134
Suggested fix
Add a test for each, or drop them from the instruction.
slack-clone Mentioning Slack's API docs, Mattermost or slack-sdk anywhere in /app zeroes the run 5 critical · 5 major flagged
- Staticpass13 checks, 1 with findings (0 blocking)
- Rubricfail34 criteria, 13 failed by a single reviewer: instruction_test_alignment, instruction_clarity, instruction_scoring_disclosure, network_policy_enforcement, grading_state_isolation, reward_channel_integrity, anticheat_false_zero, false_fail_risk, partial_score_contract, cua_rubric_matches_instruction, cua_judge_reliability, cua_rubric_benchmark_standard, test_coverage_gaps (2 minor)
Mentioning Slack's API docs, Mattermost or slack-sdk anywhere in /app zeroes the run
rubric · anticheat_false_zero · anti-cheat
What's wrong
tests/anti_cheat.py searches every text file under /app, including comments, READMEs and docs, with no exemption. Any match sets reward 0 and partial_score 0 and skips the UX grading. Several of its patterns are things an honest Slack clone is likely to write: any URL on slack.com (such as an api.slack.com docs link), the words rocketchat or mattermost anywhere, and the string slack-sdk.
How an honest solution loses
An honest agent writes a comment linking Slack's conversations.history docs while modelling the message API, or a README line saying 'does not use slack-sdk or Bolt', which is exactly what the instruction asks for. Anti-cheat fails and the run scores 0.
Effect on score: Reward 1 → 0 and partial_score → 0 for a working solution.
Evidence
tests/anti_cheat.py:49 https?://[a-z0-9.-]*slack\.com tests/anti_cheat.py:37-38 rocketchat / mattermost, case-insensitive, anywhere tests/anti_cheat.py:41 slack-sdk tests/anti_cheat.py:58-63 TEXT_EXTS include .md, .txt, .json, .html; comments are not skipped tests/test.sh:178-184 any hit zeroes the reward
Suggested fix: Scan only code (not comments or docs), and match imports and network calls rather than names and documentation URLs.
The crash test can kill the wrong process
rubric · false_fail_risk · correctness
What's wrong
To simulate a node crash, tests/test_crash.py picks the process to kill by searching every process's command line for port 8001 (forms like --port 8001, :8001, PORT=8001) and taking the highest PID among the matches. A cluster whose nodes get a static peer list on the command line, such as 127.0.0.1:8000,127.0.0.1:8001,127.0.0.1:8002, matches on every node, and the highest PID is usually node 2 or the IRC gateway, not node 1.
How an honest solution loses
An honest cluster passes its peer list as command-line arguments. The test kills node 2, waits for port 8001 to go down, times out, and all three crash tests fail.
Effect on score: Reward 1 → 0 and partial_score at most 0.5 for a working cluster, depending on how it is launched.
Evidence
tests/test_crash.py:114 regex (?:--port[= ]|:|-p\s+|PORT=)8001 over every /proc cmdline tests/test_crash.py:119-120 candidates sorted by PID, highest taken tests/test_crash.py:196-197 fixture fails if port 8001 stays up
Suggested fix: Find the process by its listening socket (the one bound to :8001), not by command-line text.
The verifier installs the agent's requirements into the Python that writes the score
rubric · grading_state_isolation · verifier
What's wrong
test.sh puts the agent's virtual environment (/opt/venv, which the Dockerfile calls the agent-owned venv) first on PATH, then pip-installs the agent's requirements.txt into it during grading. That same python3 then runs the anti-cheat scan, the script that writes reward.txt and metrics.json, and the final blend. Only the pytest gates use the separate test venv. The agent already runs as root in this container; the difference here is that the verifier itself installs agent-chosen packages into the interpreter that decides the score.
How an agent could reward-hack it
A package in the agent's requirements.txt installs a Python start-up hook into the venv. Every later python3 run loads it, including the anti-cheat and the reward writer, which can then report a pass whatever the tests did.
Effect on score: Reward 1 and partial_score 1.0 regardless of the work.
Analyst note: Folds in the static checks that flagged pip install (test.sh:93) and npm install (test.sh:108) at grading time.
Evidence
tests/test.sh:23 PATH starts with /opt/venv/bin environment/Dockerfile:118-119 /opt/venv is the 'agent-owned venv' tests/test.sh:93 pip install -r requirements.txt from /app, during grading (needs network at grading time too) tests/test.sh:181 anti-cheat, tests/test.sh:322-412 reward writer and tests/cua_blend.sh:38 blend all run on that python3
Suggested fix: Run the anti-cheat, reward writer and blend with an isolated interpreter (/opt/test-venv/bin/python -I), and install the agent's dependencies before tests/ is uploaded.
Hidden UI selectors and event names the instruction never states
rubric · instruction_test_alignment · instruction
What's wrong
The correctness half of the score needs every test gate to pass, and the tests depend on several details the instruction never gives:
- member rows must carry a data-username attribute (only the testid is stated)
- the sign-up form must have a display_name field (the instruction calls display name optional)
- current-user must show the lowercase username, not the display name
- the auth form must open in login mode, so the toggle switches to register
- the emoji picker and reaction chips must be inside the message element, not a page-level popover
- channel-title must read exactly '#' + name
- WebSocket events must be named reaction.added, message.reply, member.joined and so on (the instruction says only {"type":
How an honest solution loses
An honest clone follows the instruction exactly, opens its auth form in register mode, and names its reaction event 'reaction'. The onboarding journey and the API test fail, so the frontend and API gates fail.
Effect on score: Each item alone takes reward 1 → 0 and caps partial_score at 0.5.
Evidence
tests/journey_onboarding.py:175 data-username; :98 input[name="display_name"]; :394 lowercase username; :94 auth-toggle direction; :218 '#' + name
tests/test_e2e.py:290, :300 emoji-picker / reaction-chip looked up inside the message
tests/test_api.py:506 type == 'reaction.added'
instruction.md:104, :115-116 list only the testids and {"type": <kind>}Suggested fix: List every selector, attribute, text format and event name the tests rely on in the instruction.
The agent has open internet, and the full solution is public on GitHub
rubric · network_policy_enforcement · environment
What's wrong
task.toml gives the environment public network access and sets no stricter agent setting, so the agent has open internet for its whole 4-hour run. The stated justification covers package installs only. The public swe-marathon repository holds this task's complete reference solution in plain text, and none of it trips the anti-cheat patterns. The public tests/ also reveal every hidden selector and assertion.
How an agent could reward-hack it
The agent downloads solution/ from the public repository and submits it.
Effect on score: Reward 1 and partial_score 1.0 for a copied oracle, as far as the oracle passes.
Evidence
task.toml:97 [environment] network_mode = "public"; no [agent] network_mode solution/ app.py, irc_gateway.py, static/*, start.sh in the public repo grep of solution/ for the anti-cheat patterns (rocketchat, mattermost, slack.com, slack-sdk, slack_bolt) finds nothing
Suggested fix: Set [agent] network_mode to an allowlist of package registries, and add the solution's own fingerprints to the anti-cheat.
Reward 1 needs 9 of 9 PASS verdicts from a single judge run
rubric · cua_judge_reliability · cua-rubric
What's wrong
Reward is 1 only if every one of the 9 UX rubric items is judged PASS, in a single computer-use session with one judge sample per item. The judge model runs at its default temperature rather than 0, and items cascade: if sign-up fails, every later item fails too.
How an honest solution loses
A polished clone is judged PARTIAL on one item in one run because of a borderline screenshot. Reward is 0 for that trial; a re-run might give 1.
Effect on score: Reward flips between 0 and 1 across runs of the same solution.
Evidence
tests/cua_blend.sh:62 reward 1 only if correctness = 1 and ux >= 1.0 tests/cua_verifier.py:34, :403-404 temperature not set for opus-4-7 tests/cua_config.json:7 max_turns = 80 for all 9 items (lead: may be too few; needs a run)
Suggested fix: Take several judge samples (or majority over sessions) per item, and base the binary reward on correctness alone or on a UX threshold below 1.0.
The UX rubric demands polish the instruction never asks for
rubric · cua_rubric_matches_instruction · cua-rubric
What's wrong
Reward 1 needs every UX rubric item to PASS, and some items need things the instruction never mentions: - polish (12.5% of UX) needs 3 of 5 cues; the instruction asks for only 2 of them (hover toolbar, empty-state copy). Sidebar hover states, loading indicators and an auto-growing composer are never requested. - realism (12.5%) needs a bottom composer with placeholder text, a bold name with a secondary-coloured timestamp, and visual fidelity to real Slack. - validation_feedback needs visible focus styling on auth inputs.
How an honest solution loses
An honest clone built exactly to the instruction shows at most 2 of the 5 polish cues. The polish item is not PASS, so reward is 0.
Effect on score: Reward 1 → 0 for a solution that meets the instruction.
Evidence
tests/rubric.json:45 polish, 3 of 5 cues tests/rubric.json:51 realism tests/rubric.json:39 validation_feedback item (3) focus styling instruction.md:5 the only UI quality requirements
Suggested fix: Put every rubric cue in the instruction, or drop the cues it doesn't state.
The shape of read_state is ambiguous, and only one reading passes
rubric · instruction_clarity · instruction
What's wrong
The instruction says workspace detail returns read_state with 'the same shape' as {channel_id, last_read_seq, unread_count, mention_count}, without saying whether it is a list of those objects or a map keyed by channel id. The test iterates it as a list.
How an honest solution loses
An honest solution returns read_state as a map keyed by channel id. The test indexes a string, raises TypeError, and the API gate fails.
Effect on score: Reward 1 → 0 and partial_score at most 0.5, for one of two valid readings.
Evidence
instruction.md:42, :90 '{workspace, channels, read_state}' and 'embeds the same shape'
tests/test_read_state.py:192-193 for rs in payload['read_state']: rs['channel_id']Suggested fix: State that read_state is a list.
task.toml calls the 50/50 blend 'the trial reward', but the reward is pass/fail
rubric · instruction_scoring_disclosure · instruction
task.toml says the trial reward is 0.5 × correctness + 0.5 × ux. In the code, reward.txt is 1 only if correctness is 1 and ux is 1.0; the 50/50 blend is only metrics.json's partial_score. A stage note left in the final metrics.json repeats the wrong formula. Nothing says the correctness half is all-or-nothing across about 150 tests and 5 gates, or that reward 1 needs every UX item PASS.
Effect on score: No score changes, but readers and leaderboards are told the published number means something it doesn't.
Analyst note: Analyst checked: cua_blend.sh:62 sets reward 1 only when correctness is 1 and ux is 1.0; task.toml's verification_explanation calls the 50/50 blend the trial reward.
Evidence
task.toml:61, :73 'The trial reward is 0.5 * correctness + 0.5 * ux' tests/cua_blend.sh:61 partial_score = 0.5*correctness + 0.5*ux; :62 reward = correctness >= 1 and ux >= 1 tests/test.sh:393 stage_note with the same claim, kept in the final metrics.json
Suggested fix
Describe reward and partial_score separately and correctly in task.toml and the stage note.
partial_score gives no correctness credit unless every test passes
rubric · partial_score_contract · scoring
What's wrong
The final blend takes its correctness half from the binary reward.txt (0 or 1) and throws away the per-gate fraction test.sh computed. So partial_score is 0.5 × {0 or 1} + 0.5 × ux, and one failed assertion among ~150 tests costs the whole correctness half.
How an honest solution loses
A full clone that fails one test, with ux 0.875, scores 0.44. A UI on a basic backend with no IRC, cluster or crash support and ux 1.0 scores 0.50: the much weaker solution ranks higher.
Effect on score: partial_score misranks solutions on leaderboards.
Analyst note: Analyst checked: tests/cua_blend.sh:23 reads the binary reward.txt as the correctness term and :61 blends it 50/50 with ux; the gate fraction computed at tests/test.sh:376 is not used.
Evidence
tests/cua_blend.sh:23 correctness = contents of reward.txt tests/cua_blend.sh:61 partial_score = 0.5*correctness + 0.5*ux tests/test.sh:376 gate fraction computed, then overwritten
Suggested fix: Use the gate or test pass fraction as the correctness term of partial_score.
The UX rubric breaks the benchmark's own CUA grading rules
rubric · cua_rubric_benchmark_standard · cua-rubric
The benchmark's task-implementation standard asks for binary PASS/FAIL items with equal weights and per-item results in metrics.json. This task scores PARTIAL as 0.5, weights items 0.5–1.0, and keeps per-item verdicts only in a side file.
Effect on score: No direct score change; results aren't comparable with tasks that follow the standard.
Evidence
tests/cua_verifier.py:434 PARTIAL = 0.5 tests/rubric.json:26, :32 (0.75), :38 (0.5) tests/cua_blend.sh:70-80 only scalar fields added to metrics.json references: task-implementation.toml full_stack_clone_rubric_scoring, full_stack_clone_cua_metrics
Suggested fix
Bring the rubric and metrics output in line with the benchmark's standard.
The agent's server is still running when the final reward files are read
rubric · reward_channel_integrity · verifier · severity set by analyst (reviewer said critical)
What's wrong
The agent's server is started as root and stays alive through the UX grading and the final blend. The blend reads the correctness result from reward.txt and the UX result from ux/reward.json, files written seconds earlier. Nothing stops the server from rewriting them in between.
How an agent could reward-hack it
A thread in the agent's server overwrites reward.txt and ux/reward.json with full marks during the seconds between their writes and the blend reading them.
Effect on score: Would give reward 1 and partial_score 1.0, if the timing works; not confirmed.
Analyst note: Analyst re-graded: the route depends on the agent's server rewriting the reward files in the seconds between writes and reads, which the files do not settle. Kept as a lead that needs a run to confirm.
Evidence
tests/test.sh:143 starts the agent's server; tests/test.sh:73 kills it only in the EXIT trap tests/test.sh:409-410 writes reward.txt; tests/cua_blend.sh:23 reads it tests/cua_blend.sh:47-49 reads ux/reward.json
Suggested fix: Stop the agent's processes before the blend, or keep the stage results in memory or in a directory the agent can't write.
Search access control and several IRC commands are never tested
rubric · test_coverage_gaps · verifier
What's wrong
Search must be limited to the caller's accessible channels and honour a workspace_id filter, but the test only checks that the caller's own public message is found. The IRC commands NAMES, WHO, LIST, TOPIC and MODE are required but never sent by any test.
How a broken solution still passes
A search that returns other people's private-channel messages, and an IRC gateway that rejects NAMES and WHO, still pass.
Effect on score: No score effect for these missing requirements.
Evidence
instruction.md:85 search scope; tests/test_api.py:418-433 instruction.md:125 IRC commands; tests/test_irc.py has none of them
Suggested fix: Add a private-channel leak test for search and one test per required IRC command.
How tasks were audited
Rubric review: single reviewer per task (pilot). Rubric findings are one reviewer agent's judgement, not a multi-reviewer vote.
Static checks
Deterministic checks on the task files: required files, task.toml limits, answer leakage into the agent image, Docker build context, verifier network installs, grader config consistency, plus the benchmark's own CI checks where it ships them.
Rubric review
A reviewer agent reads every task file in a sandbox (`harbor check`) and judges it against a credibility rubric: hidden requirements, false-fail and false-pass risk, reference-solution validity, answer isolation, reproducibility.
Full description
A reviewer agent reads every task file in a sandbox (`harbor check`) and judges it against a credibility rubric: hidden requirements, false-fail and false-pass risk, reference-solution validity, answer isolation, reproducibility. When several reviewers are configured, a criterion fails only when enough of them agree.
Want this run on your tasks?
The same audit, on your benchmark or training environments.