13 of 13 audited tasks have a blocking finding.

[8T3hJZa] - instruction.md:76 tells the agent to 'Test your formatter against the reference files in /app/reference/test/'. environment/Dockerfile only does `COPY reference/ /app/reference/`, and environment/reference contains just deps/, lib/ and pubspec.yaml, with no test/ directory. An honest agent therefore has no golden examples to check byte-for-byte formatting against, which plausibly lowers its pass rate on a task scored entirely by exact match. If the v4 image does ship test/, it differs from the published build and those files are the hidden golden suite.

[8T3hJZa] - tests/test.sh:44-48 unpacks the full golden suite (inputs and expected outputs) in plaintext at <local path> in the same container where the graded formatter runs as root. run_tests.py:29 passes the golden directory on the parent's command line, which is readable via /proc/$PPID/cmdline. The '>>>'/'<<<' file format is documented in the reference's test_file.dart. A Haskell binary that finds the test case matching its stdin and prints the stored expected output passes every anti-cheat check (no execve, ≥3 .hs files, real ELF) and scores ≈1.0 with no formatter at all.

[8T3hJZa] - tests/test.sh:297-315: the only check for delegation or subprocesses is a single strace run on the input `void main() {}`. A thin Haskell main can execute directly for that input and exec `python3 /app/impl/fmt.py` (or any other program) for everything else. *.py files are recorded but never gated (compute_reward.py has no check on python_files_in_project), and files outside the project are not even recorded. This bypasses the 'in Haskell' and 'no script wrapper' rules and lets a far easier imperative Python port of the Dart code be scored at its full pass rate. Implementing the logic in C via cabal `c-sources` is not checked either.

[8T3hJZa] - tests/run_tests.py:29-40 calls subprocess.run(text=True) and only catches TimeoutExpired and OSError. If the formatter writes invalid UTF-8 on any single input (for example a Data.ByteString.Char8 output bug that truncates '×' or other non-ASCII characters, several of which appear in the golden inputs), UnicodeDecodeError ends run_tests.py. Under `set -euo pipefail`, test.sh:357 then exits before compute_reward.py runs, so no reward.json is written and the trial records an error or 0 instead of its pass rate.

[8T3hJZa] - tests/run_tests.py:34 gives each test a 30 s timeout but there is no overall budget, and the 5,224 tests run sequentially against task.toml verifier timeout_sec=86400. Any formatter that hangs on more than ~2,880 inputs (86400/30) runs past the verifier timeout, and compute_reward.py never runs. Example: a port whose line-splitter search fails to terminate on most inputs but passes 1,000 simple ones would score 0 instead of ~0.19. This also ranks it below a formatter that crashes quickly on the same inputs.

[XJwjrbS] - environment/workspace/verify_correctness.py:310-314 takes inputs from /app/media and prefers smooth gradient_ or colorbars_ images, so 12 of its 13 workloads run on gradients or flat colour bars (only the 320x240 bicubic case uses noise). tests/compute_reward.py:374-375 feeds uniform random noise into every plane of all 30 hidden workloads. With the same 60 dB per-plane bar, smooth inputs hide differences in chroma filtering, chroma siting and rounding that noise makes very large. An implementation can pass all 13 dev checks and fail one or more hidden 60 dB checks, scoring 0.5*passed/30 (e.g. 0.48) instead of 0.5+0.5*speedup. The size of this effect needs a run to confirm.
- The different comparison target (ASM ffmpeg CLI in the dev tool vs the C-only baseline in the verifier) is disclosed in instruction.md and is not counted.

[XJwjrbS] - 'Wrap... FFmpeg... libraries' is not reliably detected. test.sh:232 checks only dynamic symbols (nm -D). A Rust cdylib that statically links a surviving copy of the C libswscale.a keeps sws_* symbols local, so they never appear there. test.sh:106 matches API names only as literal text in the listed file extensions, so source that does not spell them out passes. Result: 30/30 and speedup about 1.0, a leaderboard score of about 1.0, for no reimplementation. (The surviving copy itself is filed under reference_removal_complete.)
- 'Use inline assembly' is not checked anywhere: no pattern covers Rust asm!/global_asm! or Zig asm, and .S/.asm files are scanned only for FFmpeg and verifier strings.
- 'in Zig or Rust' is not enforced: test.sh:195-198 accepts a Makefile ('make release') that builds any C code, including transplanted FFmpeg scalar C files that avoid the four public API names, compiled with -march=native.

[XJwjrbS] - tests/test.sh:161-165 skips target/, .zig-cache/ and zig-cache/ when removing prebuilt .so/.a files. Steps 182/190 then grade the first libswscale_candidate.so found anywhere in the tree, including those directories. Cargo and Zig reuse cached artifacts whose fingerprints are current. Nothing checks that the graded library came from this build (no clean build, no timestamp or output-path check). A precompiled library planted in target/ or the Zig cache, possibly built from forbidden C/asm or FFmpeg code, is graded as is.

[XJwjrbS] - tests/test.sh:20-34 deletes /reference and /usr/local/lib/lib{sw,av}*. The follow-up scan (36-48) looks only at executable ELF files under /app, matched by name. Static archives are 'current ar archive', not ELF, and are not executable, so a copy of /usr/local/lib/libswscale.a and libavutil.a made during the run survives: anywhere outside /app (/root, /opt, /tmp), or inside /app/swscale-impl/target/ or .zig-cache/ (excluded from the cleanup at 161-165). A build can statically link that copy. See forbidden_approach_detection for why the symbol checks miss it.

[XJwjrbS] - tests/compute_reward.py:419-442: the candidate runs in-process, and the try/except catches only Python exceptions. A segfault on one correctness workload kills the process, and the test.sh:280 fallback writes 0 without correctness fields, so the score is 0. A solution that instead gives wrong output on that same workload scores 0.5*29/30=0.483. A segfault in one benchmark workload likewise drops a fully correct solution from 0.5+0.5*speedup to 0, while a NULL return there still leaves about 0.5.

[XJwjrbS] - tests/compute_reward.py:522-536 calls swscale_process 110 times on the same context with the same src/dst pointers and unchanged source data, while correctness (433-436) calls each context only once. A candidate that returns immediately on a repeated identical call, because the destination already holds the result from the warmup, produces correct output with near-zero time. The median speedup reaches thousands, so the leaderboard score 0.5+0.5*speedup is in the hundreds. instruction.md allows precomputing only 'lookup tables and filter coefficients in swscale_create', not caching outputs.

[XJwjrbS] - tests/compute_reward.py:419-436 (correctness) and 527-536 (benchmark) call the candidate in-process with no per-workload or per-call timeout, and test.sh:272 has none either. A candidate that loops forever on one input, such as one odd format pair, blocks until the 3600 s verifier timeout. test.sh never reaches its fallback, so no reward is written. A solution correct on 29/30 workloads, worth 0.5*29/30=0.483, ends at 0 or a Harbor error. A hang in one benchmark workload likewise erases a fully correct solution's 0.5+0.5*speedup.

[aEcmJ5k] - task.toml:18-19 and Dockerfile:24-28 — scoring depends on the remote Tinker service accepting the pinned SDK and still holding the agent's checkpoint. The task's own comments record that the server dropping SDK 0.16.x "zeroed every trial in the Fable 5 scoring wave". A re-run or rescore after another SDK sunset or checkpoint expiry turns the same submission into 0 (compute_reward.py:1104-1108 records 0 solves on any Tinker failure).
- Dockerfile:51 — the tokenizer and chat template come from HF 'Qwen/Qwen3-8B' with no pinned revision, and transformers is unpinned (>=4.40). A rebuild could render prompts differently from what agents trained against (needs a run to confirm).

[aEcmJ5k] - instruction.md: "Must fine-tune from `Qwen/Qwen3-8B`". tests/compute_reward.py:554 and :1066-1067 accept any tinker:// path and sample it with its own base model, and never check the base model in adapter_config.json, which is only recorded for audit. A LoRA trained on a larger Qwen3 model on Tinker that shares the Qwen3 tokenizer (e.g. Qwen3-32B or Qwen3-235B-A22B) is scored the same way and very likely solves more boards, so the leaderboard score rises. How much needs a run to confirm.
- job.yaml passthrough_env TINKER_API_KEY is shared across n_concurrent_trials: 8 and across scoring waves. Nothing ties path.txt to a checkpoint created in this trial, so an agent could submit a checkpoint trained in another trial under the same key. Whether the Tinker REST API lists those checkpoints needs a run to confirm.

[aEcmJ5k] - instruction.md 'Output Parsing': "The verifier parses `<tool_call>` XML tags from the model output", and the system prompt says "You may call one or more functions". Reading A: every tool_call in a turn is executed. Reading B: only the first is. tests/compute_reward.py:449 uses `re.search`, so it executes only the first tag, and the history records only that one call (lines 765-779). A model trained under reading A (several place_frog calls per turn, which saves turns under max_tokens=2048) loses the rest of each turn's moves in the verifier and sees a history that differs from its training. That costs boards and score. How much needs a run to confirm.

[aEcmJ5k] - instruction.md 'Scoring' says: "runs both the base Qwen3-8B and your fine-tuned model on 500 unseen test boards. Weighted solve-rate improvement (easy=1×, medium=2×, hard=3×, expert=4×). 30pp weighted improvement = full marks." In fact tests/compute_reward.py:1119 sets reward = the raw count of boards the fine-tuned model solves, no base model is run, and score_from_reward.py divides by 500. That is unweighted, absolute (not an improvement) and has no 30pp cap. A rational agent following the instruction would favour expert/hard boards in its training curriculum, since they are worth 4×, and could stop once its own eval showed about 30pp weighted gain. Under the real formula an easy board is worth exactly as much as an expert one, and every extra board keeps paying, so both choices lower the leaderboard score.

[aEcmJ5k] - tests/compute_reward.py:743-748 / 843-847 — `sampling_client.sample(...).result()` has no timeout, and neither do the episodes. 500 boards × up to 30 sequential samples of ≤2048 tokens run with 40 workers, which is about 375 sequential sampling rounds against [verifier].timeout_sec = 7200. A slow Tinker queue or a model that writes long <think> blocks and rarely submits could exceed 7200 s. Results are written only at the end (line 1126), so the trial loses all credit for the boards it did solve. Needs a run to confirm.
- tests/compute_reward.py:305-337 — the informational board check runs the backtracking solver on every JSON under /app/boards with no limit, before the eval starts. RL pipelines often generate tens of thousands of boards, which uses up part of the same 7200 s (needs a run to confirm).

[RJTvUCP] - tests/test.sh:57-64: the 'no compiling the C source' rule is enforced only by counting lines in files named *.c plus a build.zig grep for git-src paths. C code kept under other file names is not counted, so a build that compiles git's C code, scoring near the oracle (~1.0), is not detected.
- tests/test.sh:78-83 and 175-206: the 'no wrapping git' rule is enforced by a single-line source grep and an strace of only three commands that ignores executables inside zig-out/bin. A binary that implements those three commands natively and delegates everything else is not detected (score near a real git's pass rate).

[RJTvUCP] - instruction.md ('Do not compile or link the C source or wrap around the existing git binary — write Zig'): it is unclear whether git's non-C components in /app/git-src (shell and Perl scripts such as git-submodule.sh, git-filter-branch.sh, git-sh-setup, git-send-email) may be shipped as they are or must be ported to Zig. Nothing in tests/test.sh distinguishes the two, because its checks look only at *.c files, ELF binaries and exec of a git binary. The readings therefore score differently: an agent that reuses the scripts collects the submodule, filter-branch and mergetool tests, while an agent that ports everything to Zig has to earn them.

[RJTvUCP] - tests/test.sh:107-113 and 126-140: the prebuilt-binary scan only flags *.o/*.a/*.so files and ELF files that have the executable bit, and test.sh:135 later chmods the graded file itself. Nothing confirms that zig-out/bin/git was compiled from the Zig sources by this build: build.zig can install an existing non-executable file as the output. A precompiled C git graded this way would score near the oracle (~1.0).

[RJTvUCP] - tests/test.sh:126 and 224-226 delete only /app/git-src, /usr/bin/git*, /usr/lib/git-core and /usr/libexec/git-core, and test.sh:262-263 wipe only /tmp and /var/tmp. Copies of the system git binary or of the C source that the agent leaves in its home directory or inside the workspace (non-executable, not named *.c) survive, and no scan outside /app/zig-port looks for them. Together with the gaps noted under forbidden_approach_detection, such a copy could be used at grading time.

[RJTvUCP] - tests/test.sh:180-185: the three strace smoke runs (init, status, hash-object) have no timeout, unlike every test script, which runs under `timeout 120`. A port with an infinite loop in `status` on an empty repository, a plausible partial-implementation bug, blocks the verifier until the sandbox expires. No reward is written, so the trial loses credit for every other test it passes.

[JUh7jnZ] - environment/Dockerfile:27-41: instruction.md lists as 'Pre-installed kernel tools' ThunderKittens (`import thunderkittens`), Helion (`import helion`), Triton-TLX, TileLang and CuTe DSL (`import cutlass`). The published Dockerfile installs only torch, flashinfer, sglang[srt], HF tools and requests. Nothing installs ThunderKittens, Helion or TLX. TileLang and cutlass-dsl can only arrive as unpinned transitive dependencies of sglang/flashinfer, if at all.
- instruction.md 'install additional packages from the pre-built cache (no internet)': no package or wheel cache is created anywhere. Every pip call uses --no-cache-dir, and allow_internet=false.
- Cost: an agent that plans around the advertised tools loses time. The direct score effect is modest because CUDA and Triton are present. Whether prebuilt image v4 contains these tools cannot be checked from the published files.

[JUh7jnZ] - environment/Dockerfile:34-38: `flashinfer` and `sglang[srt]` are unpinned, and the flashinfer install ends in `|| true`, so a failure is silently ignored. transformers and huggingface_hub are only lower-bounded. A rebuild next month gets a different SGLang/FlashInfer. That changes the baseline's speed and may change whether launch_baseline.sh flags (--mamba-scheduler-strategy extra_buffer, NEXTN for Qwen3.5) even exist. A rebuild could break the baseline (every trial scores 0) or change the speedup of the same submission.
- scripts/seed_modal_volume.py:44: the model weights come from snapshot_download of Qwen/Qwen3.5-4B with no revision, into a persistent Modal volume that is not part of the image.
- The verifier itself has no notable non-timing nondeterminism: prompt order is fixed and requests are sequential.

[JUh7jnZ] - instruction.md 'CANNOT access or reference /tests/': the only enforcement is the literal grep in test.sh:41-54. It skips every path with a dot-prefixed component (`-not -path "*/\.*"`), any path containing /model/, unscanned file types (.c, .cu, .md, files without an extension, compiled objects), all of site-packages, and any path built at runtime. The candidate server runs during verification, when /tests (prompts.jsonl, compute_reward.py with HIDDEN_WORKLOADS) is present. Code that reads those files at that point evades the scan and can tell correctness prompts from timed prompts. It can then answer the 1166 gated prompts exactly and use a lossy fast path on the timed ones, raising the speedup with no correctness loss.
- 'disable the timer daemon' is not enforced, but doing so does not raise the score.

[JUh7jnZ] - tests/compute_reward.py:918-951 and 1038-1045: any exception after the correctness gate passes produces reward 0 with no 'correctness' key. Examples: one candidate benchmark request returning HTTP 500 or exceeding 300 s, or the phase-3 baseline re-check server failing to restart, even though that re-check is diagnostic only. score_from_reward.py then gives 0. A candidate that failed the gate gets 0.5*exact_match_rate, up to about 0.47. So a fully correct, fast candidate can score below a wrong one because of one bad request or a baseline-side failure.
- tests/test.sh:2,79: under `set -euo pipefail`, a failing `tar xf` of the snapshot exits test.sh nonzero before compute_reward.py runs, so no reward is written.
- If compute_reward.py is killed by the verifier timeout, nothing is written. See resource_and_time_fit and verifier_hang_handling.

[JUh7jnZ] - tests/compute_reward.py:541-550 and 587-600: each timed workload sends the identical request (same messages and max_tokens, temperature 0) 20-40 times. The only reset is _flush_cache (line 515), which calls SGLang's /flush_cache and is ignored if the agent's server keeps its own response cache. A server that returns the stored greedy completion for a repeated identical request produces exactly correct output (the correctness prompts are unaffected, and duplicates there get identical answers). Its measured latency drops from about 1-3 s to milliseconds, giving per-workload speedups of roughly 100-1000x and a leaderboard score 0.5+0.5*s in the hundreds. The instruction ('Anything') does not forbid this.

[JUh7jnZ] - tests/compute_reward.py:40,62,694: there is no overall deadline. Each correctness request may wait REQUEST_TIMEOUT=300 s, and collection aborts only after 50 consecutive failures. If a candidate wedges on one degenerate prompt in prompts.jsonl, the remaining requests can wait up to 50 x 300 s = 15000 s, far past the 3600 s verifier timeout. Harbor kills the run with no reward written, so the 0.5*exact_match_rate partial credit the leaderboard would have paid is lost (score 0 instead of up to about 0.45).
- Startup waits of up to 1800 s per server use half the verifier budget on their own. See resource_and_time_fit.

tests/test.sh:67: if [ -f "${APP_DIR}/.oracle_solution" ]; then

[n32S7HQ] - instruction.md ('Test against the C expat source's test output as you go') — the image ships only /app/expat-src/lib (Dockerfile COPY expat-src/). There is no expat test suite, no C compiler (gcc is purged and only kept as an encrypted bundle), and no libexpat (encrypted and removed). An honest agent cannot produce the reference test output the instruction tells it to compare against, so it has to infer exact behaviour (error codes, handler sequences) from the source alone. All tools the instruction lists (nasm, as, ld, objdump, readelf, nm, strace, make) and /app/.timer are present.

[n32S7HQ] - tests/compute_reward.py:67-76 — pass/fail is read from PASS:/FAIL lines on the test binary's stdout, which the candidate library shares, and missing FAIL lines count as passes. At grading time the test sources sit unpacked under /tests/expat-test-suite. A library that, on its first API call, prints PASS lines for the registered test names and then exits normally produces a log with no FAIL lines. compute_reward then credits every mapped test, giving correctness 1.0 raw and 0.9 on the leaderboard → 0.45, the same as a perfect port, with no parser at all. Nothing checks the summary line or compares counts against the reference run.

[n32S7HQ] - tests/test.sh:162-169 — the only check for runtime loading is a text grep over files named *.s/*.asm. Assembler include files with other extensions are never scanned, and the built .so's own dynamic imports (an undefined dlopen symbol visible with nm -D) are never inspected.
- tests/test.sh:184-188 — the binary string check only matches '/usr/lib.*libexpat' or '/lib/.*libexpat'. A load by bare soname of the system libexpat (restored at test.sh:31-34 before the tests run) or of <local path> (built at test.sh:219 before step 6) passes. A wrapper that forwards the API to either library gets near-full correctness. That is ≈0.45 on the leaderboard (the maximum reachable), versus far less for honest partial ports.

[n32S7HQ] - extra_references/scripts/score_from_reward.py:37-39,72 — _LIBEXPAT_MODULE_WEIGHTS includes 'smoke_tier': 1, and total_w sums it (=10). compute_reward.py never writes a 'smoke_tier' module (its MODULE_WEIGHTS and additional_data.modules hold only basic/ns/misc/alloc/nsalloc/acc). Correctness is therefore capped at 9/10 = 0.9, so the `correctness >= 1.0` gate (line 197) is never met. Every trial, including a perfect port and the oracle, scores at most 0.5*0.9 = 0.45, and the performance subscore (ss[1].score) is never used.

[n32S7HQ] - tests/test.sh:91-96,143 — the instruction says the .so 'must be built from .s or .asm source files in /app/asm-port/', but the verifier never rebuilds. It grades whatever .so is present, provided at least one .s/.asm file exists and no GCC/clang .comment string is found. A .so assembled from a source the agent deleted, or from files the scan skips, is graded as-is, and nothing ties it to the scanned sources. This makes the source-level dlopen scan meaningless (wrapper score ≈0.45, as in forbidden_approach_detection).

[n32S7HQ] - tests/test.sh:31-34 and 219-224 — the reference is not removed at grading time but deliberately installed. The decrypted system libexpat.so.1 goes into /lib/x86_64-linux-gnu, and a fresh libexpat 2.6.4 build goes to <local path> Both happen before the agent's test binary runs (step 6) and before its benchmarks (step 7), so a library that loads the reference at run time finds it. The detection gaps are under forbidden_approach_detection.

[n32S7HQ] - tests/benchmark.c / test.sh:415 — the timed loop re-parses the identical buffer 100–100000 times through one parser (XML_ParserReset) with no handlers registered. The only thing checked is XML_Parse's return status. A library that remembers the result for an identical input, or skips real work when no handlers are set, gets an arbitrarily large ratio (compute_reward.py:222 'No cap').
- The timer is clock(), resolved through the benchmark binary's link order, which puts the agent's .so before libc. A library exporting its own clock symbol controls the measurement.
Today this inflates only the verifier's raw reward (0.5*corr+0.5*perf, gated on correctness>0). It reaches the leaderboard once correctness can hit 1.0 (blocked now by the smoke_tier bug).

[n32S7HQ] - tests/test.sh:366 with expat-test-suite/minicheck.c — all ~365 tests run in ONE process under a single `timeout 300`. There is no fork and no signal handler, and stdout goes to a file, so it is fully buffered and never flushed. One segfault or infinite loop in an early test during the first iteration (chunksize=0) loses every later test's PASS line, plus up to ~4 KB (~130 lines) of already-buffered PASS output. compute_reward.py counts all of those as failures, since totals come from source. A port that would pass ~80% of tests (leaderboard 0.5*0.8*0.9≈0.36) but crashes on, say, the 20th basic test scores ≈0, which is below a less-correct port that never crashes.

[yZQAunT] - All 182 tests are deterministic and read no input. tests/test.sh:385 gives the compiler up to 60 s per program, and nothing checks what the output binary computes. A compiler that evaluates the program at compile time and emits a binary that only writes the captured stdout passes every test, scoring about 1.0 with no code generation. The evaluator can be its own evaluator over the Protos, using the full runtime and stdlib in liblua-compile.a, or the surviving /verifier-data/lua. compile_stubs.c removes only luaV_execute.

[yZQAunT] - tests/test.sh:558-564: nothing detects the forbidden interpreter approach (instruction.md:27,69-73). An output binary can carry the agent's own bytecode interpreter, with its own loader, linked as prebuilt runtime support. It avoids the five parser symbol names checked at :435-441 and can score close to 1.0.
- tests/test.sh:473-490: the C-API ban is effectively unenforced. The check runs only when total==1, i.e. on the first test, and only if it compiled. It also looks only for undefined ('U') lua_* symbols, but liblua-runtime.a is static, so in a linked executable every lua_* symbol is defined ('T'). A compiler that emits lua_* call sequences passes.
- tests/test.sh:424-444: the parser-symbol check looks only at the first compiled binary, and a stripped binary shows no symbols.
- tests/test.sh:117-124: the 'no gcc on generated C' rule is checked only by a literal regex over source text. A command name built at runtime is not matched, and the leaderboard ignores the flag anyway (see leaderboard_score_reconstruction). A C-intermediate compiler can score close to 1.0.

[yZQAunT] - tests/compute_reward.py:57-60 always writes subscores[test_pass_rate].score = pass_rate, even when hard_fail_reasons sets reward to 0. score_from_reward.py:63-66 returns that subscore first. test.sh Step 5 runs the tests whatever ANTI_CHEAT_OK says, and also when BUILD_OK=0 if a stale luanatc exists. So a source-scan violation (the C-intermediate pattern or a /verifier-data reference) or a build failure still earns full partial credit. Example: 170/182 passing with anti_cheat_ok=0 gives reward 0 but a leaderboard score of 0.934.

[yZQAunT] - tests/test.sh:59-80 deletes /reference/lua, the /usr/local/bin and /usr/bin copies, and executables whose sha256 exactly matches. Two working copies survive:
  - /verifier-data/lua is in the image and readable by the root agent, and is never removed.
  - Any copy changed by even one byte (different hash) is also kept.
- An output binary can exec either copy on the original test path and match byte for byte, scoring about 1.0. The '/verifier-data' source scan at :98-104 covers only some file extensions, and the leaderboard ignores its result.

[SfjjYrn] - environment/Dockerfile:57: the unpinned `modular` package decides which MAX API and kernels the candidate runs on. A rebuild could change or break the same solution's outputs.
- Dockerfile:98 and 103-107: the reference repo and docs come from moving URLs.
The verifier itself is deterministic apart from GPU numerics.

[SfjjYrn] - tests/test.sh:66: the torch/transformers/diffusers check matches only imports at the start of a line (`^import torch`, ...). An import inside a function body, or a dynamic import through importlib, is not caught. torch and diffusers are installed in the image (Dockerfile:71-76), so a submission that runs WanPipeline this way reproduces the reference exactly and scores 1.0.
- test.sh:83 accepts any file with a `from max.` line, so a token MAX import satisfies the 'pure MAX' rule. Nothing checks that the model computation actually runs on MAX, and numpy compute is never checked.
- The scan covers only /app/submission. Code placed elsewhere (instruction.md forbids this) is never scanned.
See oracle_mode_isolation for the scan-skip marker.

[SfjjYrn] - environment/Dockerfile:117: hidden_workloads.json (prompts and seeds 100/200/300/400) ships inside the image at /opt/verifier-data.tar.gz, readable by root. A submission that special-cases those four prompts would score 1.0. See answer_isolation for the expected frames, which are in the same archive.

[SfjjYrn] - instruction.md 'Scoring' section: says the score is the uncapped geomean speedup vs diffusers and is zero if any workload fails. In fact test.sh:153-157 runs compute_reward_partial.py, which awards passed/4 with no speed measurement. The leaderboard (score_from_reward.py implementation branch) scores correctness only, capped at 1.0. An agent told that speed is the score will plausibly spend hours on speed and accept reduced-precision shortcuts that cut its PSNR margin. Speed is worth nothing, and each workload lost to PSNR costs 0.25.

[SfjjYrn] - score_from_reward.py:162 parses the first 'n/m' in reward['reason']. compute_reward_partial.py:88 copies the text of the candidate's import exception into reason ('IMPORT FAIL: {e}'). That import runs in a separate process after test.sh:113's own import check, so a candidate whose import fails only in the scoring process, with digits and a slash in the message, gets reward 0 but leaderboard correctness up to 1.0.
- Normal paths are fine: 'partial_correctness=k/4' gives k/4. The subscores carry no 'subtask' key, so the loop at lines 155-160 is never used.

[SfjjYrn] - build_and_push.sh:17 builds and pushes the tag 'latest', while task.toml names :v4, so nothing ties the published files to v4.
- environment/Dockerfile:57,98,103-107: `pip install modular` is unpinned, the Wan2.1 repo is cloned at HEAD, and the MAX docs are curled from live URLs. A rebuild today would get a different MAX SDK, which matters because the agent's code depends on the MAX API.

[SfjjYrn] - tests/test.sh:19-22 deletes only *.py files under /app/reference, and none of it is needed at runtime. The real delegation target, the diffusers/torch/transformers packages in system site-packages (Dockerfile:71-76), is never removed and stays importable during scoring. Combined with the import-scan gap (see forbidden_approach_detection), a submission can call the PyTorch pipeline.

[SfjjYrn] - tests/compute_reward_partial.py:111-197: the hidden workloads run with no per-workload timeout, and reward.json is written only after all of them finish (line 212). If one workload hangs (for example, a GPU deadlock on the 17-frame input), the whole 7200 s verifier timeout is used up and no reward is written. Workloads that already passed lose their credit: a 3/4 solution scores 0 instead of 0.75.

tests/test.sh:18: if [ ! -f "$APP_DIR/.oracle_solution" ]; then
tests/test.sh:53: if [ ! -f "$APP_DIR/.oracle_solution" ]; then
tests/test.sh:159: if [ -f "$APP_DIR/.oracle_solution" ]; then

[LWTZs67] - tests/compute_reward.py:120-123 and 452-458 — the instruction forbids merging inputs into one archive, but attribution only needs a file at each input path, and zero-byte placeholders are accepted. Compressing all 80 hidden notebooks into the output for the largest one and writing empty files for the rest gives r_i = artifact_term for 79 files. Computed on the real bundle (122,584,118 bytes, largest 38,413,468) with a 10 KB artifact and a 0.1-ratio codec: geom_mean about 9e-5, leaderboard about 0.9999, against about 0.89 for the same codec scored honestly.
- tests/scoring_core.py:114 / compute_reward.py:334-380 — "Decompress runs in a fresh environment with access only to <artifact_dir> and <compressed_dir>" is not enforced. Decompress runs in the same filesystem while the hidden inputs still exist, so compress can record or stash its inputs outside compressed_dir and decompress can restore them from there. Near-empty outputs then score about 1.0.
- compute_reward.py:285-305 — "Everything needed at decompress time must live in <artifact_dir>" is not enforced. A dictionary or model left in /app (up to 512 MiB) or written by fit outside artifact_dir is read at compress and decompress without counting toward artifact_bytes, removing the artifact_term penalty the metric is built around.

[LWTZs67] - environment/notebook-data/visible/ — the published visible corpus is three tiny placeholder notebooks (sample/test1.ipynb 555 B: "This is a sample notebook"; corpus1 and corpus2 about 1.3 KB each) plus stray .zst copies. environment/notebook-data/manifest.json describes 195 files totalling 136,111,403 bytes from about 45 source repos, and instruction.md calls it the "full visible notebook corpus". The real corpus can only be in the prebuilt image (task.toml docker_image ...:v5). A third party rebuilding from environment/ gets a corpus with essentially no training data. The verifier also re-runs `fit` on $DATA_ROOT/visible, so scores on a rebuilt image would differ materially.

[LWTZs67] - score_from_reward.py notebook-compression branch — a valid run scores 1 - geom_mean_ratio with no lower clamp, while any failure scores 0. That contradicts instruction.md ("hard FAIL ranked below every valid run"). Witness: a valid lossless codec whose artifact is about 150 MB (within the stated 8 GiB cap) has artifact_term about 1.22 and a score of about -0.27, below a submission that crashes (0). An identity-copy codec also scores slightly below 0.

[TrE9dWT] - environment/workspace/run_visible.py (speedup = min(speedup, 3.0)) — the dev tool caps each workload at 3x, but compute_reward.py:39 is uncapped. An agent choosing between configs giving 3x/3x and 10x/2x on two workloads sees geomean 3.0 vs 2.45 in the dev tool, but the verifier gives 3.0 vs 4.47, so the agent keeps the lower-scoring one.
- run_visible.py (missed target -> speedup = 0.0, 'Some workloads failed — reward would be 0.0') — the verifier gives partial credit target/final_ema (compute_reward.py:40-41) and, in practice, counts zeros as 1.0x. The dev tool reports 0 where the verifier pays roughly 0.9-1.0.
- run_visible.py ('Estimated reward = min(1.0, geo_mean / 3.0)') — the leaderboard score is the raw uncapped geomean (score_from_reward.py optimizer-design branch). The printed 'reward' is on a different scale from the graded one.

[TrE9dWT] - tests/test.sh:24-45 — the rule 'Cannot modify frozen files' is enforced against /app/.frozen_hashes.json, a file in the agent's writable /app (the agent is root, and chmod 444 does not bind root). If the file is missing, the check only prints WARN and continues. An agent can change train_workload.py (e.g. how target_reached_step is recorded) and refresh or delete the manifest, and grading uses the modified loop for all 10 workloads, giving speedups up to 100x each (leaderboard score about 100).
- tests/test.sh:68-96 — the 'no filesystem/network access, self-contained' rule is checked only by matching import statements in the AST. Allowed modules expose sys/os as attributes (warnings.sys and random._os exist), and builtins can import anything, so optimizer code can reach the caller's frames. It can then read workload.name (forbidden name branching) or mutate the WorkloadConfig/loop state (e.g. target_loss) inside the frozen loop. Nothing detects this, and the score rises without limit.
- 'Cannot branch on workload names or model class names' has no verifier check at all beyond the above.

[TrE9dWT] - instruction.md (Scoring) says 'geometric mean punishes zeros hard', but tests/compute_reward.py:45-49 drops zero speedups from the product while still dividing by the full count, so a zero counts exactly like 1.0x (the mechanism is under score_ranking_is_monotone). A rational agent following the instruction avoids aggressive configs that might diverge on some workloads. Under the real formula, divergence on a workload is neutral and beats any honest miss below 1.0, so the instruction pushes agents toward a lower-scoring strategy. The rest matches: the uncapped baseline_steps/steps and the partial credit target/final loss (verifier uses the EMA val loss and min(.,1)).

[TrE9dWT] - tests/compute_reward.py:45-49 — geometric_mean keeps only positive speedups but divides by len(values), so a workload with speedup 0 (exception, NaN or divergence: compute_speedup returns 0.0 because NaN>0 is false and target/inf is 0) contributes log(1), the same as 1.0x. Witness pair: optimizer A trains every workload and misses one narrowly (0.9 on it, 1.0 elsewhere) and gets 0.9^(1/10)=0.990. Optimizer B diverges to NaN or raises on that workload and gets 1.000. Likewise, deliberately failing on every workload except one with speedup s gives s^(1/10) instead of about 0. Better honest work scores below a crash.

[TrE9dWT] - tests/compute_reward.py:65-101 runs all 10 workloads sequentially with no per-workload or per-step time limit, and writes reward.json only once, at the very end (line 163). If the total run exceeds [verifier].timeout_sec=10800, Harbor kills test.sh before anything is written, so credit already earned on finished workloads is lost and the trial scores 0. Witness: an honest optimizer with costlier steps (e.g. a second-order/preconditioned method about 2x slower per step), which instruction.md does not forbid and which states no verifier time limit, could reach targets well ahead of the baseline in steps and still score 0 if the 10-workload sum passes 3 h. A step() that hangs on one hidden workload's parameter shape likewise erases the other nine. How close real runs come to 3 h needs a run to confirm.

[mBtxWuD] - `tests/inference_trace.py:173-174` (`if not _is_relative_to(path, app_root): continue`): reads outside /app, /tmp and the runtime directory are not checked, and the parameter count looks only at /app/checkpoint. A predict.py that loads a 200M-parameter `/root/model.pt` while /app/checkpoint holds a 1-parameter file with a matching `--count-params` passes the 50M cap check undetected.

[mBtxWuD] - `tests/test.sh:55`: the only enforcement of the 2D-only rule is a literal-name grep over .py/.sh files. `getattr(AllChem, 'Embed' + 'Molecule')`, or code kept in a non-.py file and exec'd, generates conformers without matching. 3D-derived descriptors can lower MAE, and nothing else (strace, trace) notices.

[mBtxWuD] - `tests/scoring_core.py:93,134` (load_prediction_frame / mean_absolute_error) are called from `tests/compute_reward.py` evaluate_prediction_file with no try/except. A predict.py that exits 0 but writes a CSV missing the `prediction` column, or one containing NaN or non-numeric predictions, raises in Python. test.sh runs under `set -e`, so it exits nonzero with no reward.json: Harbor records an error instead of 0 (or keeps a stale file; see reward_channel_integrity). The same happens if the holdout bundle fails to unzip (`unzip` fails under set -e).

[LSHG7ag] - tests/pg_compat_test.sh:43-53: check_no_error passes whenever psql exits 0. Without ON_ERROR_STOP, psql exits 0 even when the server returns an ERROR, so all 27 check_no_error items pass for a server that rejects every query.
- check_error for T8.4 accepts any message containing 'error' (:303).
- A server that only completes startup and answers every query with an ErrorResponse therefore earns about 28/72 compat items, ≈ 0.035 of the leaderboard score, for wrong output.
- Regression and TAP use upstream exact comparisons and are fine.

[LSHG7ag] - instruction.md forbids 'Wrapping a real PostgreSQL server'. The only enforcement is the literal-string scan at tests/test.sh:60-71 (the two real-binary paths) and a build.zig-only library check (:97-166). A candidate that copies the real server from /verifier-data during the run to an unscanned location, or builds the path at runtime, and execs it for postgres/initdb/pg_ctl passes every check. No process, binary or ldd check exists. Score ≈ the real PostgreSQL pass rate under this harness (likely most of the 810 items). Source of the binaries: see answer_isolation.
- Minor: the required language (Zig) is checked only for Rust/Cargo (:84-91). C sources compiled through build.sh are not detected, though this does not by itself raise the score.

[LSHG7ag] - tests/test.sh:182 deletes zig-out, then runs the agent-controlled build.sh. Nothing confirms that zig-out/bin/postgres-sqlite was compiled from the scanned Zig sources. build.sh can copy in a prebuilt executable stored anywhere on disk or under an unscanned extension in the workspace (the scan covers only .zig/.zon/.txt/.md/.sh/.json), and test.sh:190-205 grades it. This admits precompiled non-Zig or forbidden binaries, including a renamed real server (see answer_isolation). Score up to near 1.0.

[LSHG7ag] - tests/test.sh:398: `kill "${WATCHDOG_PID}" 2>/dev/null` runs under `set -euo pipefail` with no `|| true`. The watchdog subshell exits by itself after 'restart failed, giving up' (test.sh:377-378) or after 50 restarts (test.sh:380-382). In either case kill fails with 'No such process', the script exits nonzero, and compute_reward.py never runs. No reward.json is written.
- Witness: a partly working server that crashes during the regression suite and cannot be restarted within 30 s. It loses all credit, including compat items it had already passed (up to 72/810 ≈ 0.09) and every TAP pass, and scores 0 instead of its partial rate. High confidence; needs a run to confirm.
- More generally, the EXIT trap only removes BUILD_ROOT, and no fallback writes a reward on any unexpected command failure.

[LSHG7ag] - tests/test.sh's worst case is 900 s (compat, :341) + 1800 s (regression, :394) + 300 s for each of the 63 TAP directories (:528) = 21,600 s, plus configure/make/initdb. task.toml [verifier].timeout_sec is 7200, and the reward is only written at the very end (:626).
- Partial solutions often stall in TAP poll_query_until/pg_ctl waits, and PG_TEST_TIMEOUT_DEFAULT=600 (:310) exceeds the 300 s per-directory cap. If about 15-20 of the 63 directories reach their 300 s cap, Harbor stops the verifier before compute_reward.py runs. All partial credit is then lost (score 0 instead of, e.g., 0.1-0.2). Needs a run to confirm how many directories stall for a typical submission.

[6BBAK4P] - tests/test.sh SSIM_THRESHOLD = 0.95 (with ffmpeg ssim 'All', averaged over every frame) against instruction.md: 'near pixel-perfect matches ... Lossy tricks (frame skipping, quality reduction, resolution changes) will fail this gate'. Two examples:
  - Rendering every second frame and duplicating it: half the frames match exactly, and the rest differ by one frame of slow animation. The average stays well above 0.95 for the graphics scenes.
  - Encoding at a visibly lower quality or a faster x264 preset: on this synthetic content SSIM is typically 0.95-0.98.
  Both of these lossy shortcuts would likely clear the bar and pay their speedup in full under 0.5+0.5*speedup. Exact SSIM values need a run to confirm. The code comment admits the bar was loosened to absorb v0.4.2-vs-v0.4.4 drift.

[6BBAK4P] - tests/test.sh:34-46: the rule 'Access or reference /baseline/' is enforced only by a literal grep for '/baseline/revideo' in .ts/.js/.json/.py/.sh files under packages/. Code copied from /baseline/revideo during the agent run leaves no such string, and a renderer that builds the path at runtime to load modules from there is also missed. Either way the candidate becomes the baseline: speedup ~1, leaderboard score ~1.0. This is the same payoff as the /opt copy described under answer_isolation, reached through the path the instruction explicitly forbids.

[6BBAK4P] - The hidden scenes (tests/hidden-scenes.tar.gz) use only the three media files already in the agent's image (packages/benchmark/public/media/test_5s_720p.mp4 and test_10s_1080p.mp4). Five of the eight hidden scenes decode video. An agent can pre-decode these known files during its run and have the renderer read the stored frames, removing decode cost on 5/8 scenes while output stays identical. The resulting speedup feeds 0.5+0.5*speedup. The size of the gain needs a run to confirm.

[6BBAK4P] - extra_references/scripts/score_from_reward.py:109-117 (the ssim_threshold < 0.99 branch, which the CLI's default of 0.95 always takes): correctness comes only from correctness_details[*].ssim, and the score uses geometric_mean_speedup. hard_fail_reasons is ignored. compute_reward.py zeroes the reward for 'candidate_render_failed' and 'candidate_missing_scenes', but geometric_mean_speedup is computed only over scenes that succeeded. A candidate whose renderVideo writes a correct MP4 and then throws on the scenes where it is slower than baseline is hard-failed by the verifier (reward 0). The leaderboard instead pays 0.5+0.5*speedup computed over the favorable scenes only.
- The function default ssim_threshold=0.99 (line 54) differs from the CLI default of 0.95 (line 221). Under 0.99 the score uses the capped 'reward', and under 0.95 the uncapped geometric_mean_speedup.

[6BBAK4P] - tests/test.sh render_phase B1 then B2: the same scenes and inputs are rendered twice. The merge averages t1 and t2 but copies only run 1's MP4 for SSIM. A renderer that stores its first result (on disk or in /tmp) and returns it, or returns quickly, on the identical second call makes t2 ≈ 0. The candidate's average time then halves and the geometric-mean speedup roughly doubles: for example, a true 1.2x becomes ~2.4x, and the leaderboard score goes from 1.1 to 1.7. compute_reward.py only rejects ct <= 0.