anthropics / scone-bench

Benchmark for evaluating LLM agents on smart-contract vulnerability discovery and exploitation

GitHub Repo

3.9K

lines of main code

0.03K

2.4K

3.9K

0.02K

toml

	docker
Standard analyses:
		Main Code: 3,919 LOC (19 files) = PY (99%) + TOML (<1%) Secondary code: Test: 25 LOC (1); Generated: 0 LOC (0); Build & Deploy: 89 LOC (5); Other: 2,269 LOC (5);
		Duplication: 11%
		File Size: 54% long (>1000 LOC), 15% short (<= 200 LOC)
		Logical Component Decomposition: primary (4 components)
		Goals: Keep the system simple and easy to change (4)
Experimental analyses (less reliable heuristic analyses):
		Unit Size: 0% long (>100 LOC), 43% short (<= 10 LOC)
		Conditional Complexity: 0% complex (McCabe index > 50), 40% simple (McCabe index <= 5)
		Static Component Dependencies: primary (4 components)

All Analysis Reports

Source Code Overview

Components

Component Dependencies*

Duplication

File Size

Unit Size*

Conditional Complexity*

Features of Interest

All Metrics

Goals & Controls

File Age & Freshness

File Churn

Temporal Dependencies

Contributors

Notes & Findings

No commit history found.

Visual Code Explorers

MAIN (19)	Circles	Sunburst
TEST (1)	Circles	Sunburst
BUILD AND DEPLOYMENT (5)	Circles	Sunburst
GENERATED (0)	Circles	Sunburst
OTHER (5)	Circles	Sunburst

File Visualizations

File size views:

3D view of file size
files grouped by size category
files grouped by folder

Duplication views:

2D force graph of duplication among files
3D force graph of duplication among files
2D view of duplication among files (with duplicates)
3D view of duplication among files (with duplicates)

File age views:

	files grouped by age category files grouped by folder
	files grouped by freshness category files grouped by folder

File change frequency views:

files grouped by change frequency category
files grouped by folder

Contributors per file views:

files grouped by number of contributors category
files grouped by folder

Contributor Visualizations

Contributor dependency views:

past 30 days: 2D graph | 2D graph (with files) | 2D graph (with shared files only) | 3D graph | 3D graph (with files) | 3D graph (with shared files only)
past 3 months: 2D graph | 2D graph (with files) | 2D graph (with shared files only) | 3D graph | 3D graph (with files) | 3D graph (with shared files only)
past 6 months: 2D graph | 2D graph (with files) | 2D graph (with shared files only) | 3D graph | 3D graph (with files) | 3D graph (with shared files only)
past year: 2D graph | 2D graph (with files) | 2D graph (with shared files only) | 3D graph | 3D graph (with files) | 3D graph (with shared files only)

Components and Dependencies Visualizations

Components

Temporal Dependencies

Duplication

Commits Racing Charts

30 days

3 months

6 months

PRIMARY (4)

File Dependencies Visualizations

Temporal dependencies among files:

past 30 days: no dependencies
past 3 months: no dependencies
past 6 months: no dependencies

Units Visualizations

Unit size and conditional complexity views:

3D view of unit size
3D view of unit complexity

Lists of Files Per Scope

MAIN (19 files)
TEST (1 file)
BUILD AND DEPLOYMENT (5 files)
GENERATED
OTHER (5 files)
FILES: History Data
IGNORED FILES: By Extension | By Rule

Analysis Results

CONFIGURATION: JSON
ALL ANALYSIS RESULTS: JSON
DUPLICATES: TXT | JSON
UNITS: TXT | JSON
CONTRIBUTORS: TXT | JSON
LOGICAL DECOMPOSITIONS: JSON
CONCERNS: JSON
CONTROLS: TXT
ALL METRICS: TXT

Zipped Files

GIT HISTORY: ZIP
ALL FILES IN ALL ANALYSIS SCOPES: ZIP

AI Prompts

Generative AI tools, like ChatGPT or Gemini, can help you explore and discuss various aspects of source code repositories using simple prompts and file uploads. Sokrates provides you with curated data that you can use to analyze your source code further.

Example Prompt 1: Repository Evolution Analyzer (based on git history)

Try on: OpenAI ChatGPT | Claude.ai Chat | Google Gemini

Files to upload: git-history.zip

Prompt:

You are an experienced software architect tasked with analyzing Git commit history.
Context:
- Repository Name: anthropics / scone-bench
- Repository Description: Benchmark for evaluating LLM agents on smart-contract vulnerability discovery and exploitation
- Data Provided: A plain text file containing Git commit records. Source code is not available.
- Analysis Basis: Your analysis must be based solely on inferring context from the provided file paths, file names, file extensions, commit timestamps, and author information within the commit records.

Input Format:
The commit records are provided in the following format per line:
[YYYY-MM-DD] [author email] [commit hash] [file path changed]

Requested Insights:

Please analyze the provided commit history and generate a report covering the following sections:

1. Management Summary
- Provide a concise, high-level overview suitable for non-technical stakeholders.
- Summarize the repository's likely purpose, main development phases observed, and key takeaways about the team structure/activity.

2. Functional Overview
- Based only on file/module/class names, infer and describe the primary functions and capabilities of the software in this repository.
- What services or features does the codebase seem to support?

3. Key Business/Domain Concepts
- Identify and list core business or domain concepts reflected in the file and directory names (e.g., reporting, analytics, data export, user management, sourcecode analysis, landscape, visualization).

4. Inferred Architecture & Design
- Infer potential architectural patterns (e.g., modular, layered, microservices) based on directory structures and file naming conventions.
- Highlight notable design choices evident from the structure, such as separation of concerns (e.g., common module), use of configuration files, or templating engines.

5. Inferred Technology Stack
- List the programming languages, frameworks, libraries, and tools likely used, based on file extensions (.java, .py, .js, .html, .xml, .yml, pom.xml, dockerfile), configuration files, and common directory structures (e.g., /src/main/java).
- Categorize these technologies where possible (e.g., Backend, Frontend, Build, Infrastructure).

6. Logical Components / Modules
- Identify the major logical components or modules suggested by the directory structure (e.g., codeanalyzer, reports, cli, common, web).
- Briefly describe the inferred purpose of each major component.

7. Evolution by Year
- Summarize the apparent focus of development activity for each year present in the commit history.
- Assign a thematic title to each year (e.g., "Year 1: Foundation & Core Analysis", "Year 2: Reporting & Visualization Expansion").

8. Trends Over Time
Identify and describe significant trends observed across the years, such as:
- Shifts in technology usage (e.g., introduction of Docker).
- Changes in module activity (e.g., increased focus on landscape analysis).
- Signs of project maturity, expansion into new areas, or periods of focused maintenance.
9. Team Dynamics
- Analyze contributor patterns based on author emails and commit frequency:
 - Identify core maintainer(s) vs. occasional contributors.
 - Describe observable contribution trends (e.g., stable team, high turnover, reliance on a single developer).
 - Note any appearance of automated contributors (e.g., Dependabot).
10. Risk Analysis:
- Based on the findings in all previous sections (Team Dynamics, Technology, Architecture, Evolution), identify potential key risks for the project.
- Examples: Key person dependency, technology obsolescence, maintainability challenges due to complexity, potential scalability bottlenecks, bus factor.

Input File:
You will receive the commit records in a plain text file. Please parse this file to perform your analysis.

Example Prompt 2: File name conventions

Try on: OpenAI ChatGPT | Claude.ai Chat | Google Gemini

Files to upload: files.json

Prompt:

Example Prompt 3: Technology analyzer (based of file paths)

Try on: OpenAI ChatGPT | Claude.ai Chat | Google Gemini

Files to upload: files.json

Prompt:

Analyze the provided list of file paths to identify as many technologies, frameworks, tools, programming languages, cloud services, and architectural patterns used in the project as possible.

**Input:**
Please analyze the following list of file structures (provided as a list of strings or a JSON object containing file paths and optionally extensions/metadata):

*(Example: A JSON array like the one previously provided, or a simple list like `["src/server.ts", "Dockerfile", "tests/e2e/spec.cy.js", "infra/main.tf"]`)*

**Analysis Instructions:**
When performing the analysis, please consider the following:

1. **File Extensions:** Identify languages and configuration types (e.g., `.ts`, `.js`, `.py`, `.java`, `.tf`, `.yml`, `.hcl`, `.css`, `.html`).
2. **Directory Structure & Naming:** Look for conventional directory names (`src`, `tests`, `infra`, `docs`, `cicd`) and names indicating specific components or frameworks (`packages/bff`, `features`, `controllers`, `services`, `cypress`).
3. **Specific Filenames:** Identify standard configuration or definition files (e.g., `package.json`, `pom.xml`, `docker-compose.yml`, `Dockerfile`, `Jenkinsfile`, `webpack.config.js`, `tsconfig.json`, `requirements.txt`, `terragrunt.hcl`, `lint-staged.config.js`, `commitlint.config.js`).
4. **Keywords in Paths:** Look for names of known technologies, tools, frameworks, or cloud providers within the paths (e.g., `aws`, `azure`, `lambda`, `sqs`, `react`, `nest`, `k6`, `datadog`, `pm2`).

**Output:**
Please present the findings as a list of identified technologies. If possible, group them into relevant categories such as:

* Programming Languages
* Frameworks & Runtimes
* Testing Tools
* Infrastructure as Code / Cloud Providers
* Containerization & Orchestration
* Build Tools / Bundlers
* Databases (if identifiable)
* Monitoring & Logging
* CI/CD & Dev Tools
* Architectural Patterns

generated by sokrates.dev (configuration) on 2026-06-24