- Staticpass19 checks, 0 with findings (0 blocking)
- Rubricfail15 criteria, 2 failed by a single reviewer: hidden_interface_requirements, test_coverage_gaps (1 minor)
- Validationpassoracle=1 nop=0
- Probepass1/1 probe(s) reproduced their finding by execution
critical
Verified by execution
Tests depend on names or interfaces the instruction doesn't specify
rubric · hidden_interface_requirements · verifier
Single-reviewer finding: one reviewer agent flagged this; it has not been through a multi-reviewer vote.
Verified by executionReference solution with only the rule's display name changed from 'Auto Table of Contents' to 'Auto TOC'.
Verifier reward: 0
41 of 1172 tests failed:
- [f2p] Auto Table of Contents No TOC markers present — text is unchanged
- [f2p] Auto Table of Contents Both markers present with empty TOC region generates TOC
- [f2p] Auto Table of Contents Only start marker present — inserts TOC and adds end marker
- [f2p] Auto Table of Contents End marker before start marker is ignored
- [f2p] Auto Table of Contents Multiple marker pairs — only the first pair is updated
- [f2p] Auto Table of Contents Updates existing TOC content between markers
- [f2p] Auto Table of Contents Nested headings produce correct indentation
- [f2p] Auto Table of Contents Numbered list style uses 1. prefix
- [f2p] Auto Table of Contents Indent size option controls indentation per level
- [f2p] Auto Table of Contents Bullet marker option controls the bullet character
- [f2p] Auto Table of Contents Ordered list style increment uses increasing numbers
- [f2p] Auto Table of Contents Explicit heading IDs are used as anchors when enabled
- [f2p] Auto Table of Contents Explicit heading IDs are deduplicated when repeated
- [f2p] Auto Table of Contents Strip formatting in TOC affects link text but not anchor generation
- [f2p] Auto Table of Contents Exclude headings option can exclude headings by literal match
... and 26 more
41 of the failures are 'missing from report'; 41 of those tests actually ran and PASSED under a different name, so the grader rejected working code only because of the name.
Evidence
[kzYvmAs] - Every f2p test ID depends on a display name that the instruction never states. tests/config.json sets grade.node_id = "name", and every f2p ID starts with "Auto Table of Contents" (e.g. "Auto Table of Contents No TOC markers present — text is unchanged").
- That prefix is not written in the test file. The base repo's __tests__/common.ts ruleTest() does `describe(rule.getName(), ...)`, and getName() resolves the rule's nameKey through the locale.
- The string comes only from the reference solution, which adds `'auto-toc': { 'name': 'Auto Table of Contents', ... }` to src/lang/locale/en.ts.
- instruction.md only says to "Implement a new rule, export default `AutoToc` from `src/rules/auto-toc.ts`, that generates or updates a TOC". It gives no rule name or locale entry.
- Consequence: an agent with perfect behaviour that names the rule "Auto TOC", "Table of Contents" or "Auto-TOC" gets all 41 f2p IDs reported as "missing from report" and scores 0.
minor
Reviewer finding, not yet verified
Stated requirements are not tested
rubric · test_coverage_gaps · verifier
Single-reviewer finding: one reviewer agent flagged this; it has not been through a multi-reviewer vote.
Evidence
[kzYvmAs] - Blank line after the end marker: the instruction says "Ensure blank lines ... after the end marker", but in every test input the end marker is followed by a blank line or end of text. The reference does not add one either: it outputs `${before}\n\n${tocContent}\n\n${after}` with `after` starting at the existing end marker. An implementation that skips this requirement still passes.
- Exclusion matching is only tested on plain literal and regex text. Formatted headings are never tested against excludeHeadings.
- `__bold__` and `_italic_` anchor stripping are never tested.