anthropics/scone-bench Benchmark for evaluating LLM agents on smart-contract vulnerability discovery and exploitation |
for people* (7/10)
for AI* (7/9)
Rule-of-thumb estimates
Rebuild value
Maintenance effort
analysis.maintainabilityScores.- File size0.4−1.670% of main code in files > 500 lines
For people: Large files are hard to navigate, mix responsibilities and attract merge conflicts.
For AI agents: Agents read whole files into their context: every change to a large file costs many tokens and crowds out other relevant code.
- Test code0.2−1.61 test lines per 100 main lines
For people: Tests document the intended behaviour and make changes safe.
For AI agents: Tests let an agent check its own work; without them its mistakes surface later, in review or in production.
- Duplication5.5−0.412% of main code duplicated
For people: A fix in one copy must be repeated in all the others; a missed copy becomes a bug.
For AI agents: Agents copy the patterns they find, so duplicates multiply; a change to one copy misses the others, and every copy costs context.
- Unit size8.0−0.210% of unit code in units > 50 lines
For people: Long functions do many things at once: they are hard to name, read, test and reuse.
For AI agents: To change one line, an agent reads and often rewrites the whole function: bigger edits, more tokens, more breakage.
- Volume9.63,919 lines of main code
For people: More code is more to learn, search and keep in mind; onboarding and judging a change's impact slow down as it grows.
For AI agents: An agent sees only a slice at a time: the bigger the code base, the more it searches, and the likelier it misses relevant code or rewrites an existing helper.
- Unit complexity10.00% of unit code with McCabe > 25
For people: Every branch is another path to understand and test; complex logic is where mistakes are made and missed in review.
For AI agents: Nested conditions are where generated changes break edge cases; complex units need more tests and review before an agent's change can be trusted.
- File complexity10.00% of main code in files with McCabe sum > 125
For people: Complex files concentrate risk: understanding one means keeping many paths in mind.
For AI agents: Much to read and much logic to get right in one place: the costliest combination for an agent.
- File size0.4−2.270% of main code in files > 500 lines
For AI agents: Agents read whole files into their context: every change to a large file costs many tokens and crowds out other relevant code.
For people: Large files are hard to navigate, mix responsibilities and attract merge conflicts.
- Test code0.2−2.21 test lines per 100 main lines
For AI agents: Tests let an agent check its own work; without them its mistakes surface later, in review or in production.
For people: Tests document the intended behaviour and make changes safe.
- Duplication5.5−0.312% of main code duplicated
For AI agents: Agents copy the patterns they find, so duplicates multiply; a change to one copy misses the others, and every copy costs context.
For people: A fix in one copy must be repeated in all the others; a missed copy becomes a bug.
- Unit size8.0−0.110% of unit code in units > 50 lines
For AI agents: To change one line, an agent reads and often rewrites the whole function: bigger edits, more tokens, more breakage.
For people: Long functions do many things at once: they are hard to name, read, test and reuse.
- Volume9.63,919 lines of main code
For AI agents: An agent sees only a slice at a time: the bigger the code base, the more it searches, and the likelier it misses relevant code or rewrites an existing helper.
For people: More code is more to learn, search and keep in mind; onboarding and judging a change's impact slow down as it grows.
- Unit complexity10.00% of unit code with McCabe > 25
For AI agents: Nested conditions are where generated changes break edge cases; complex units need more tests and review before an agent's change can be trusted.
For people: Every branch is another path to understand and test; complex logic is where mistakes are made and missed in review.
- File complexity10.00% of main code in files with McCabe sum > 125
For AI agents: Much to read and much logic to get right in one place: the costliest combination for an agent.
For people: Complex files concentrate risk: understanding one means keeping many paths in mind.
- 1server.pysrc/scone_bench/McCabe 49615 LOC
- 2transcript_writer.pyrunner/McCabe 3396 LOC
- 3edit.pysrc/scone_bench/tools/McCabe 31377 LOC
- 4utils.pyrunner/McCabe 251,004 LOC
- 5client.pyrunner/McCabe 15114 LOC
- 6proxy.pysrc/scone_bench/McCabe 929 LOC
- 7base.pysrc/scone_bench/tools/McCabe 929 LOC
- 8bash.pysrc/scone_bench/tools/McCabe 6102 LOC
- 9cache.pysrc/scone_bench/McCabe 493 LOC
- 10run.pysrc/scone_bench/tools/McCabe 443 LOC
| Standard analyses: | Main Code: 3,919 LOC (19 files) = PY (99%) + TOML (<1%) Secondary code: Test: 25 LOC (1); Generated: 0 LOC (0); Build & Deploy: 89 LOC (5); Other: 2,269 LOC (5); |
||
| Duplication: 11% | |||
| File Size: 54% long (>1000 LOC), 15% short (<= 200 LOC) | |||
|
|
Logical Component Decomposition: primary (4 components) | ||
| Goals: Keep the system simple and easy to change (4) | |||
| Experimental analyses (less reliable heuristic analyses): | |||
| Unit Size: 0% long (>100 LOC), 43% short (<= 10 LOC) | |||
| Conditional Complexity: 0% complex (McCabe index > 50), 40% simple (McCabe index <= 5) | |||
|
|
Static Component Dependencies: primary (4 components) | ||
No commit history found.
Visual Code Explorers
| MAIN (19) | Circles | Sunburst | |
| TEST (1) | Circles | Sunburst | |
| BUILD AND DEPLOYMENT (5) | Circles | Sunburst | |
| GENERATED (0) |
Circles
|
Sunburst
|
|
| OTHER (5) | Circles | Sunburst |
File Visualizations
File size views:
Duplication views:
File age views:
File change frequency views:
Contributors per file views:
Contributor Visualizations
Contributor dependency views:
Components and Dependencies Visualizations
| Components | Temporal Dependencies | Duplication | Commits Racing Charts | ||||||||||
| 30 days | 3 months | 6 months | |||||||||||
| PRIMARY (4) | Bubble Chart | Tree Map |
2D
|
3D
|
2D
|
3D
|
2D
|
3D
|
All Time | 12 Months | |||
File Dependencies Visualizations
Temporal dependencies among files:
|
Units Visualizations
Unit size and conditional complexity views:
Lists of Files Per Scope
- MAIN (19 files)
- TEST (1 file)
- BUILD AND DEPLOYMENT (5 files)
-
GENERATED
- OTHER (5 files)
- FILES: History Data
- IGNORED FILES: By Extension | By Rule
Analysis Results
- CONFIGURATION: JSON
- ALL ANALYSIS RESULTS: JSON
- DUPLICATES: TXT | JSON
- UNITS: TXT | JSON
- CONTRIBUTORS: TXT | JSON
- LOGICAL DECOMPOSITIONS: JSON
- CONCERNS: JSON
- CONTROLS: TXT
- ALL METRICS: TXT