openai/evals
File Complexity*
The distribution of files by complexity: the sum of the McCabe indexes of their units.
File Complexity Overall
  • There are 266 files with units, with 27,437 lines of code.
    • 0 very complex files (McCabe index sum 251+; 0 lines of code)
    • 1 complex files (McCabe index sum 126-250; 1,014 lines of code)
    • 19 medium complex files (McCabe index sum 51-125; 5,577 lines of code)
    • 30 slightly complex files (McCabe index sum 26-50; 4,955 lines of code)
    • 216 simple files (McCabe index sum 1-25; 15,891 lines of code)
  • 1,229 main files have no units (e.g. languages without unit analysis, or files without functions) and are not classified.
system0% | 3% | 20% | 18% | 57%
Legend:
251+
126-250
51-125
26-50
1-25


explore: grouped by folders | grouped by complexity
File Complexity per Extension
251+
126-250
51-125
26-50
1-25
py0% | 3% | 20% | 18% | 57%
js0% | 0% | 0% | 0% | 100%
File Complexity per Logical Decomposition
primary
251+
126-250
51-125
26-50
1-25
evals0% | 3% | 20% | 18% | 57%
scripts0% | 0% | 0% | 0% | 100%
.github0% | 0% | 0% | 0% | 100%
Most Complex Files (Top 50)

The McCabe index of a file grows with its size: per 100 lines tells a large but plain file from a dense one, and most complex unit whether the complexity sits in one unit or is spread over many.

File# lines# unitsMcCabe index
(sum)		per 100
lines		most complex
unit
actions.py
in evals/elsuite/multistep_web_tasks/webarena/browser_env
 1014 50 160 16 26
tools.py
in evals/elsuite/bugged_tools
 497 37 114 23 7
cards.py
in evals/elsuite/bluff/bluff
 206 37 105 51 18
record.py
in evals
 450 54 104 23 12
solve.py
in evals/registry/data/solve-for-variable/tools
 231 30 94 41 23
core.py
in evals/elsuite/make_me_say
 223 18 89 40 18
processors.py
in evals/elsuite/multistep_web_tasks/webarena/browser_env
 495 21 87 18 19
eval.py
in evals/elsuite/function_deduction
 244 14 86 35 19
environment.py
in evals/elsuite/hr_ml_agent_bench
 283 21 71 25 17
session.py
in evals/elsuite/multistep_web_tasks
 416 28 69 17 10
registry.py
in evals
 242 27 66 27 7
make_plots.py
in evals/elsuite/identifying_variables/scripts
 325 17 65 20 13
make_plots.py
in evals/elsuite/error_recovery/scripts
 446 20 63 14 7
data.py
in evals
 148 21 62 42 11
wave_function_collapse.py
in evals/registry/data/simple_physics_engine
 157 17 59 38 9
eval.py
in evals/elsuite/identifying_variables
 227 14 58 26 13
eval.py
in evals/elsuite/bugged_tools
 210 9 57 27 13
playwright_api.py
in evals/elsuite/multistep_web_tasks/webarena/core
 279 32 55 20 8
make_plots.py
in evals/elsuite/track_the_stat/scripts
 235 9 54 23 16
make_plots.py
in evals/elsuite/already_said_that/scripts
 263 8 52 20 18
openai_solver.py
in evals/solvers/providers/openai
 181 18 48 27 7
corrset.py
in evals/elsuite/identifying_variables/renderers
 216 11 45 21 12
board.py
in evals/elsuite/cant_do_that_anymore/chess
 162 13 43 27 10
players.py
in evals/elsuite/bluff/bluff
 107 13 43 40 8
utils.py
in evals/elsuite
 150 13 43 29 9
oaieval.py
in evals/cli
 253 6 41 16 25
make_plots.py
in evals/elsuite/ballots/scripts
 233 12 40 17 10
eval.py
in evals/elsuite/bluff
 164 9 40 24 20
diagonal_dataset_creation.py
in evals/elsuite/cant_do_that_anymore/scripts
 216 6 37 17 17
low_level_actions.py
in evals/elsuite/hr_ml_agent_bench
 304 13 37 12 12
baselines.py
in evals/elsuite/function_deduction
 91 10 36 40 8
tester.py
in evals/registry/data/solve-for-variable/tools
 94 9 36 38 18
together_solver.py
in evals/solvers/providers/together
 68 10 34 50 20
evaluators.py
in evals/elsuite/multistep_web_tasks/webarena/evaluation_harness
 273 19 34 12 5
utils.py
in evals/elsuite/skill_acquisition
 115 15 33 29 5
nxn_maze_eval_generator.py
in evals/registry/data/mazes
 162 9 33 20 6
custom_datasets.py
in evals/elsuite/steganography/scripts/dataset
 197 12 32 16 8
app.py
in evals/elsuite/multistep_web_tasks/docker/flask-playwright
 165 8 32 19 7
eval.py
in evals/elsuite/error_recovery
 204 9 32 16 17
strong_solver.py
in evals/elsuite/multistep_web_tasks/solvers/strong_solver
 173 12 31 18 11
solver.py
in evals/registry/data/simple_physics_engine
 90 1 31 34 31
solver_tools_convo.py
in evals/elsuite
 181 11 30 17 10
eval.py
in evals/elsuite/self_prompting
 210 7 30 14 16
bugged_tools.py
in evals/elsuite/bugged_tools
 132 6 30 23 8
consistency_plots.py
in evals/elsuite/sandbagging/scripts
 88 6 30 34 8
basic_bash_env.py
in evals/elsuite/multistep_web_tasks/webarena/bash_env
 163 13 29 18 5
utils.py
in evals/elsuite/bugged_tools
 48 6 27 56 13
plot_experiments.py
in evals/elsuite/bugged_tools/scripts
 107 3 26 24 23
eval.py
in evals/elsuite/skill_acquisition
 313 7 26 8 13
poker_analysis_sample_generator.py
in evals/registry/data/poker_analysis
 95 6 26 27 13
About This Analysis
  • File complexity is the sum of the McCabe indexes (conditional complexity) of the units (methods, functions...) in a file.
  • Files are classified in five categories based on that sum: 1-25 (simple files), 26-50 (slightly complex files), 51-125 (medium complex files), 126-250 (complex files), 251+ (very complex files). The default bands are five times the unit complexity bands, as a file is a group of units.
  • Only main code files with units are classified. Files without units (languages without unit analysis, or files without functions) have no complexity to measure and are left out rather than counted as simple.
  • A very complex file holds many decisions, which often means it has too many responsibilities: a candidate to split, and a place where changes need more testing.
Learn more...