openai / SWELancer-Benchmark

36K

py

0.1K

toml

0

commits
(30 days)

0

contributors
(30 days)

	47
	12
	2025

	github actions docker
Standard analyses:
		Main Code: 35,801 LOC (348 files) = PY (99%) + TOML (<1%) Secondary code: Test: 35,299 LOC (401); Generated: 0 LOC (0); Build & Deploy: 31 LOC (1); Other: 14,667 LOC (1726);
		Duplication: 50%
		File Size: 4% long (>1000 LOC), 81% short (<= 200 LOC)
		Logical Component Decomposition: primary (5 components)
		2 months old 0% of code older than 365 days 0% of code not updated in the past 365 days
		0% of code updated more than 50 times Also see temporal dependencies for files frequently changed in same commits.
		Goals: Keep the system simple and easy to change (4)
Experimental analyses (less reliable heuristic analyses):
		Unit Size: 1% long (>100 LOC), 35% short (<= 10 LOC)
		Conditional Complexity: <1% complex (McCabe index > 50), 77% simple (McCabe index <= 5)
		Static Component Dependencies: primary (5 components)

	Features of interest: TODOs 3 files

All Analysis Reports

Source Code Overview

Components

Temporal Dependencies

Duplication

File Size

File Age & Freshness

File Change Frequency

Commits

Contributors

Unit Size*

Conditional Complexity*

Features of Interest

All Metrics

Goals & Controls

Component Dependencies*

Trend

Notes & Findings

Overall Activity Per Year

Latest commit date: 2025-04-03

Reference analysis date: 2025-05-04

0

commits
(30 days)

0

contributors
(30 days)

	47
	12
	2025

Activity Per File Extension

commits

	2025
	30	py
	1	toml

contributors

	2025
	9	py
	1	toml

Visual Code Explorers

MAIN (348)	Circles	Sunburst
TEST (401)	Circles	Sunburst
BUILD AND DEPLOYMENT (1)	Circles	Sunburst
GENERATED (0)	Circles	Sunburst
OTHER (1726)	Circles	Sunburst

File Visualizations

File size views:

Duplication views:

File age views:

	files grouped by age category files grouped by folder
	files grouped by freshness category files grouped by folder

File change frequency views:

Contributors per file views:

Contributor Visualizations

Contributor dependency views:

past 30 days: graphviz | 2D graph | 2D graph (with files) | 2D graph (with shared files only) | 3D graph | 3D graph (with files) | 3D graph (with shared files only)
past 3 months: graphviz | 2D graph | 2D graph (with files) | 2D graph (with shared files only) | 3D graph | 3D graph (with files) | 3D graph (with shared files only)
past 6 months: graphviz | 2D graph | 2D graph (with files) | 2D graph (with shared files only) | 3D graph | 3D graph (with files) | 3D graph (with shared files only)
past year: graphviz | 2D graph | 2D graph (with files) | 2D graph (with shared files only) | 3D graph | 3D graph (with files) | 3D graph (with shared files only)

Components and Dependencies Visualizations

Components

Temporal Dependencies

Duplication

Commits Racing Charts

30 days

3 months

6 months

PRIMARY (5)

Bubble Chart

Tree Map

graphviz

2D

3D

graphviz

2D

3D

graphviz

2D

3D

Duplication Graph

All Time

12 Months

File Dependencies Visualizations

Temporal dependencies among files:

past 30 days: no dependencies
past 3 months: graphviz | 2D graph | 2D graph (with commits) | 3D graph | 3D graph (with commits)
past 6 months: graphviz | 2D graph | 2D graph (with commits) | 3D graph | 3D graph (with commits)

Units Visualizations

Unit size and conditional complexity views:

Lists of Files Per Scope

Analysis Results

CONFIGURATION: JSON
ALL ANALYSIS RESULTS: JSON
DUPLICATES: TXT | JSON
UNITS: TXT | JSON
CONTRIBUTORS: TXT | JSON
LOGICAL DECOMPOSITIONS: JSON
CONCERNS: JSON
CONTROLS: TXT
ALL METRICS: TXT

Zipped Files

GIT HISTORY: ZIP
ALL FILES IN ALL ANALYSIS SCOPES: ZIP

Generative AI tools, like ChatGPT or Gemini, can help you explore and discuss various aspects of source code repositories using simple prompts and file uploads. Sokrates provides you with curated data that you can use to analyze your source code further.

Example Prompt 1: Repository Evolution Analyzer (based on git history)

Try on: OpenAI ChatGPT | Claude.ai Chat | Google Gemini

Files to upload: git-history.zip

Prompt:

You are an experienced software architect tasked with analyzing Git commit history.
Context:
- Repository Name: openai / SWELancer-Benchmark
- Repository Description: This repo contains the dataset and code for the paper "SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?"
- Data Provided: A plain text file containing Git commit records. Source code is not available.
- Analysis Basis: Your analysis must be based solely on inferring context from the provided file paths, file names, file extensions, commit timestamps, and author information within the commit records.

Input Format:
The commit records are provided in the following format per line:
[YYYY-MM-DD] [author email] [commit hash] [file path changed]

Requested Insights:

Please analyze the provided commit history and generate a report covering the following sections:

1. Management Summary
- Provide a concise, high-level overview suitable for non-technical stakeholders.
- Summarize the repository's likely purpose, main development phases observed, and key takeaways about the team structure/activity.

2. Functional Overview
- Based only on file/module/class names, infer and describe the primary functions and capabilities of the software in this repository.
- What services or features does the codebase seem to support?

3. Key Business/Domain Concepts
- Identify and list core business or domain concepts reflected in the file and directory names (e.g., reporting, analytics, data export, user management, sourcecode analysis, landscape, visualization).

4. Inferred Architecture & Design
- Infer potential architectural patterns (e.g., modular, layered, microservices) based on directory structures and file naming conventions.
- Highlight notable design choices evident from the structure, such as separation of concerns (e.g., common module), use of configuration files, or templating engines.

5. Inferred Technology Stack
- List the programming languages, frameworks, libraries, and tools likely used, based on file extensions (.java, .py, .js, .html, .xml, .yml, pom.xml, dockerfile), configuration files, and common directory structures (e.g., /src/main/java).
- Categorize these technologies where possible (e.g., Backend, Frontend, Build, Infrastructure).

6. Logical Components / Modules
- Identify the major logical components or modules suggested by the directory structure (e.g., codeanalyzer, reports, cli, common, web).
- Briefly describe the inferred purpose of each major component.

7. Evolution by Year
- Summarize the apparent focus of development activity for each year present in the commit history.
- Assign a thematic title to each year (e.g., "Year 1: Foundation & Core Analysis", "Year 2: Reporting & Visualization Expansion").

8. Trends Over Time
Identify and describe significant trends observed across the years, such as:
- Shifts in technology usage (e.g., introduction of Docker).
- Changes in module activity (e.g., increased focus on landscape analysis).
- Signs of project maturity, expansion into new areas, or periods of focused maintenance.
9. Team Dynamics
- Analyze contributor patterns based on author emails and commit frequency:
 - Identify core maintainer(s) vs. occasional contributors.
 - Describe observable contribution trends (e.g., stable team, high turnover, reliance on a single developer).
 - Note any appearance of automated contributors (e.g., Dependabot).
10. Risk Analysis:
- Based on the findings in all previous sections (Team Dynamics, Technology, Architecture, Evolution), identify potential key risks for the project.
- Examples: Key person dependency, technology obsolescence, maintainability challenges due to complexity, potential scalability bottlenecks, bus factor.

Input File:
You will receive the commit records in a plain text file. Please parse this file to perform your analysis.

Example Prompt 2: File name conventions

Try on: OpenAI ChatGPT | Claude.ai Chat | Google Gemini

Files to upload: files.json

Prompt:

Example Prompt 3: Technology analyzer (based of file paths)

Try on: OpenAI ChatGPT | Claude.ai Chat | Google Gemini

Files to upload: files.json

Prompt:

Analyze the provided list of file paths to identify as many technologies, frameworks, tools, programming languages, cloud services, and architectural patterns used in the project as possible.

**Input:**
Please analyze the following list of file structures (provided as a list of strings or a JSON object containing file paths and optionally extensions/metadata):

*(Example: A JSON array like the one previously provided, or a simple list like `["src/server.ts", "Dockerfile", "tests/e2e/spec.cy.js", "infra/main.tf"]`)*

**Analysis Instructions:**
When performing the analysis, please consider the following:

1. **File Extensions:** Identify languages and configuration types (e.g., `.ts`, `.js`, `.py`, `.java`, `.tf`, `.yml`, `.hcl`, `.css`, `.html`).
2. **Directory Structure & Naming:** Look for conventional directory names (`src`, `tests`, `infra`, `docs`, `cicd`) and names indicating specific components or frameworks (`packages/bff`, `features`, `controllers`, `services`, `cypress`).
3. **Specific Filenames:** Identify standard configuration or definition files (e.g., `package.json`, `pom.xml`, `docker-compose.yml`, `Dockerfile`, `Jenkinsfile`, `webpack.config.js`, `tsconfig.json`, `requirements.txt`, `terragrunt.hcl`, `lint-staged.config.js`, `commitlint.config.js`).
4. **Keywords in Paths:** Look for names of known technologies, tools, frameworks, or cloud providers within the paths (e.g., `aws`, `azure`, `lambda`, `sqs`, `react`, `nest`, `k6`, `datadog`, `pm2`).

**Output:**
Please present the findings as a list of identified technologies. If possible, group them into relevant categories such as:

* Programming Languages
* Frameworks & Runtimes
* Testing Tools
* Infrastructure as Code / Cloud Providers
* Containerization & Orchestration
* Build Tools / Bundlers
* Databases (if identifiable)
* Monitoring & Logging
* CI/CD & Dev Tools
* Architectural Patterns