openai / evals
File Churn

File change frequency (churn) shows the distribution of file updates (days with at least one commit).

Overview
File Change Frequency Overall
  • There are 1,489 files with 41,192 lines of code.
    • 0 files changed more than 100 times (0 lines of code)
    • 0 files changed 51-100 times (0 lines of code)
    • 1 file changed 21-50 times (242 lines of code)
    • 13 files changed 6-20 times (1,728 lines of code)
    • 1,475 files changed 1-5 times (39,222 lines of code)
0% | 0% | <1% | 4% | 95%
Legend:
101+
51-100
21-50
6-20
1-5

explore: grouped by folders | grouped by update frequency | data
Contributors Count Frequency Overall
  • There are 1,489 files with 41,192 lines of code.
    • 0 files changed by more than 25 contributors (0 lines of code)
    • 2 files changed by 11-25 contributors (495 lines of code)
    • 8 files changed by 6-10 contributors (1,225 lines of code)
    • 135 files changed by 2-5 contributors (8,433 lines of code)
    • 1,344 files changed by 1 contributor (31,039 lines of code)
0% | 1% | 2% | 20% | 75%
Legend:
26+
11-25
6-10
2-5
1

explore: grouped by folders | grouped by contributors count | data
File Change Frequency per File Extension
jsonl, yaml, py, md, txt, sh, json, gitignore, ipynb, gitattributes, html, js, in, ini, toml
File Change Frequency per Extension
The number of recorded file updates
101+
51-100
21-50
6-20
1-5
py0% | 0% | <1% | 5% | 93%
yaml0% | 0% | 0% | <1% | 99%
jsonl0% | 0% | 0% | 0% | 100%
ipynb0% | 0% | 0% | 0% | 100%
html0% | 0% | 0% | 0% | 100%
js0% | 0% | 0% | 0% | 100%
in0% | 0% | 0% | 0% | 100%
File Change Frequency per Logical Decomposition
primary
primary (file change frequency)
The number of recorded file updates
101+
51-100
21-50
6-20
1-5
evals0% | 0% | <1% | 3% | 95%
scripts0% | 0% | 0% | 100% | 0%
ROOT0% | 0% | 0% | 0% | 100%
.github0% | 0% | 0% | 0% | 100%
Most Frequently Changed Files (Top 50)

See data for all files...

File# lines# unitscreatedlast modified# changes
(days)		# contributorsline
churn		first
contributor		latest
contributor
registry.py
in evals
 242 27 2023-03-21 2024-03-28 22 17 +316 / -145 343165+rlbayes@users.norepl... oliver.jaffe@hotmail.co.uk
classify.py
in evals/elsuite/modelgraded
 97 3 2023-03-16 2023-09-26 19 7 +312 / -479 shane@openai.com bomarni@googlemail.com
oaieval.py
in evals/cli
 253 6 2023-03-18 2024-09-30 19 18 +369 / -276 120423412+andrew-openai@use... steven@openai.com
record.py
in evals
 450 54 2023-03-16 2024-01-26 13 9 +218 / -52 shane@openai.com 140545726+ianmckenzie-oai@u...
api_utils.py
in evals/utils
 15 1 2023-03-18 2024-03-26 10 8 +152 / -144 120423412+andrew-openai@use... oliver.jaffe@hotmail.co.uk
utils.py
in evals/elsuite
 150 13 2023-03-22 2023-09-26 10 6 +101 / -53 343165+rlbayes@users.norepl... bomarni@googlemail.com
data.py
in evals
 148 21 2023-03-17 2024-04-02 9 10 +70 / -56 343165+rlbayes@users.norepl... 150190178+josnyder-2@users....
eval.py
in evals
 170 15 2023-03-20 2024-03-28 9 9 +161 / -45 2406911+zhangmarvin@users.n... oliver.jaffe@hotmail.co.uk
includes.py
in evals/elsuite/basic
 48 3 2023-04-11 2023-06-12 7 4 +44 / -25 73198383+hwchung27@users.no... 120423412+andrew-openai@use...
test-modelgraded.yaml
in evals/registry/evals
 81 - 2023-03-20 2023-09-18 7 3 +56 / -48 343165+rlbayes@users.norepl... 55913678+cholotook@users.no...
utils.py
in evals/elsuite/make_me_say
 34 4 2023-09-19 2023-12-20 6 5 +67 / -21 140545726+ianmckenzie-oai@u... inwaves@users.noreply.githu...
48 3 2023-03-15 2024-01-10 6 6 +137 / -16 jasonwei@openai.com 140545726+ianmckenzie-oai@u...
49 1 2023-03-16 2024-01-10 6 5 +29 / -71 shane@openai.com 140545726+ianmckenzie-oai@u...
185 1 2023-03-16 2024-01-10 6 5 +528 / -17 shane@openai.com 140545726+ianmckenzie-oai@u...
test-all.yaml
in evals/registry/eval_sets
 21 - 2023-03-16 2023-04-24 5 3 +8 / -3 shane@openai.com 343165+rlbayes@users.norepl...
test-modelgraded-battle.yaml
in evals/registry/evals
 36 - 2023-03-16 2023-04-24 5 3 +11 / -9 shane@openai.com 343165+rlbayes@users.norepl...
fuzzy_match.py
in evals/elsuite/basic
 49 3 2023-04-11 2023-06-05 5 4 +28 / -17 73198383+hwchung27@users.no... jwang47@users.noreply.githu...
base.py
in evals
 51 1 2023-03-20 2023-09-26 5 4 +13 / -76 343165+rlbayes@users.norepl... lukevanseters@gmail.com
match.py
in evals/elsuite/basic
 57 3 2023-04-11 2023-09-26 5 4 +27 / -7 73198383+hwchung27@users.no... lukevanseters@gmail.com
api.py
in evals
 61 5 2023-03-28 2023-06-08 5 4 +57 / -200 jwang47@users.noreply.githu... 131678108+wingsdrafterwork@...
oaievalset.py
in evals/cli
 111 8 2023-03-21 2024-01-10 5 4 +57 / -12 343165+rlbayes@users.norepl... 140545726+ianmckenzie-oai@u...
classify_utils.py
in evals/elsuite/modelgraded
 145 8 2023-04-04 2023-09-18 5 2 +269 / -82 343165+rlbayes@users.norepl... 97272807+sohenze@users.nore...
openai.py
in evals/completion_fns
 147 10 2023-04-11 2024-03-26 5 6 +203 / -22 jwang47@users.noreply.githu... oliver.jaffe@hotmail.co.uk
best.yaml
in evals/registry/modelgraded
 10 - 2023-03-20 2023-09-18 4 2 +15 / -3 343165+rlbayes@users.norepl... 55913678+cholotook@users.no...
base.py
in evals/elsuite/modelgraded
 16 - 2023-04-04 2023-04-27 4 1 +187 / -162 343165+rlbayes@users.norepl... 343165+rlbayes@users.norepl...
human_cli_solver.py
in evals/solvers
 29 3 2023-11-09 2024-03-13 4 3 +60 / -12 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
utils.py
in evals/solvers
 37 2 2023-11-09 2024-03-28 4 2 +137 / -82 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
json_validator.py
in evals/elsuite/basic
 41 4 2023-04-14 2023-06-12 4 2 +58 / -9 120423412+andrew-openai@use... 120423412+andrew-openai@use...
coqa-ex.yaml
in evals/registry/evals
 55 - 2023-03-17 2023-04-24 4 3 +57 / -18 343165+rlbayes@users.norepl... 343165+rlbayes@users.norepl...
base.py
in evals/prompt
 64 8 2023-03-29 2024-03-13 4 3 +24 / -17 343165+rlbayes@users.norepl... oliver.jaffe@hotmail.co.uk
hhh.py
in evals/solvers/prompts
 99 - 2023-11-09 2024-03-13 4 3 +212 / -90 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
autoeval.py
in evals/elsuite/make_me_say
 116 3 2023-09-19 2023-12-05 4 4 +149 / -6 140545726+ianmckenzie-oai@u... erik.t.ritter@gmail.com
solver.py
in evals/solvers
 125 17 2023-11-09 2024-03-28 4 2 +265 / -49 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
validators.py
in evals/registry/data/word_association/corpus_tools
 151 11 2023-07-04 2024-01-10 4 4 +345 / -39 douglas.monsky@gmail.com 140545726+ianmckenzie-oai@u...
in
 4 - 2023-03-30 2023-04-12 3 3 +3 / -2 jwang47@users.noreply.githu... 120423412+andrew-openai@use...
test-modelgraded-generated.yaml
in evals/registry/evals
 9 - 2023-03-16 2023-03-27 3 2 +37 / -1 shane@openai.com 343165+rlbayes@users.norepl...
test-modelgraded.yaml
in evals/registry/eval_sets
 14 - 2023-03-17 2023-04-06 3 2 +4 / -3 343165+rlbayes@users.norepl... 343165+rlbayes@users.norepl...
langchain_math.py
in evals/completion_fns
 21 4 2023-04-11 2023-12-05 3 4 +44 / -13 jwang47@users.noreply.githu... erik.t.ritter@gmail.com
sql.yaml
in evals/registry/modelgraded
 24 - 2023-04-21 2023-05-30 3 2 +25 / -0 mark@haym.me 133797909+jorge-openai@user...
langchain_llms.yaml
in evals/registry/completion_fns
 24 - 2023-04-11 2023-12-21 3 3 +28 / -1 73198383+hwchung27@users.no... lorenzo.pacchiardi@stats.ox...
prompts.py
in evals/elsuite/schelling_point
 25 - 2023-09-19 2024-01-10 3 2 +337 / -1 140545726+ianmckenzie-oai@u... 140545726+ianmckenzie-oai@u...
32 - 2023-07-13 2026-04-14 3 3 +20 / -5 pan93412@gmail.com 146999853+mcgrew-oai@users....
ballots.yaml
in evals/registry/evals
 34 - 2023-09-19 2024-03-13 3 3 +39 / -0 140545726+ianmckenzie-oai@u... oliver.jaffe@hotmail.co.uk
prompts.py
in evals/elsuite/ballots
 44 - 2023-09-19 2024-01-10 3 2 +146 / -6 140545726+ianmckenzie-oai@u... 140545726+ianmckenzie-oai@u...
utils.py
in evals/elsuite/make_me_pay
 47 6 2023-09-19 2023-12-05 3 3 +125 / -50 140545726+ianmckenzie-oai@u... erik.t.ritter@gmail.com
eval.py
in evals/elsuite/make_me_say
 48 3 2023-09-19 2024-03-28 3 2 +75 / -16 140545726+ianmckenzie-oai@u... oliver.jaffe@hotmail.co.uk
metrics.py
in evals
 52 8 2023-06-02 2023-09-26 3 3 +13 / -16 21045365+kjbilton@users.nor... bomarni@googlemail.com
task_description.py
in evals/elsuite/make_me_pay
 57 - 2023-11-15 2024-03-13 3 2 +64 / -4 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
make-me-pay.yaml
in evals/registry/evals
 59 - 2023-09-19 2024-03-13 3 2 +262 / -195 140545726+ianmckenzie-oai@u... oliver.jaffe@hotmail.co.uk
cot_solver.py
in evals/solvers/nested
 61 7 2024-01-29 2024-03-28 3 2 +87 / -2 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
Files With Most Churn (Top 50)
Based on the number of lines added and deleted across all commits.

See data for all files...

File# lines# unitscreatedlast modified# changes
(days)		# contributorsline
churn		first
contributor		latest
contributor
jsonl
logiqa.jsonl
in evals/registry/data/logiqa
 3 - 2023-04-11 2023-04-11 1 2 +1357 / -1354 kondrich@openai.com 62553364+csitfun@users.nore...
generate_samples.ipynb
in evals/registry/data/backgammon
 1349 - 2023-06-22 2023-06-22 1 1 +1349 / -0 bakebrain@gmail.com bakebrain@gmail.com
actions.py
in evals/elsuite/multistep_web_tasks/webarena/browser_env
 1014 50 2024-03-21 2024-03-21 1 1 +1264 / -0 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
defaults.yaml
in evals/registry/solvers
 294 - 2024-01-29 2024-03-28 3 2 +651 / -291 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
classify.py
in evals/elsuite/modelgraded
 97 3 2023-03-16 2023-09-26 19 7 +312 / -479 shane@openai.com bomarni@googlemail.com
solve.py
in evals/registry/data/solve-for-variable/tools
 231 30 2023-06-02 2023-09-26 2 2 +590 / -141 130226176+juan-de-salgado@u... bomarni@googlemail.com
tools.py
in evals/elsuite/bugged_tools
 497 37 2024-03-19 2024-03-19 1 1 +653 / -0 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
oaieval.py
in evals/cli
 253 6 2023-03-18 2024-09-30 19 18 +369 / -276 120423412+andrew-openai@use... steven@openai.com
processors.py
in evals/elsuite/multistep_web_tasks/webarena/browser_env
 495 21 2024-03-21 2024-03-21 1 1 +626 / -0 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
make_plots.py
in evals/elsuite/error_recovery/scripts
 446 20 2024-03-19 2024-03-19 1 1 +597 / -0 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
185 1 2023-03-16 2024-01-10 6 5 +528 / -17 shane@openai.com 140545726+ianmckenzie-oai@u...
session.py
in evals/elsuite/multistep_web_tasks
 416 28 2024-03-21 2024-03-21 1 1 +518 / -0 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
jsonl
samples.v0.jsonl
in evals/registry/data/pattern_identification
 3 - 2023-03-15 2023-03-16 2 2 +256 / -250 jasonwei@openai.com jasonwei@openai.com
theory_of_mind.yaml
in evals/registry/solvers
 394 - 2024-01-29 2024-03-28 3 2 +456 / -35 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
gen_data.py
in evals/elsuite/identifying_variables/scripts
 319 13 2024-03-19 2024-03-19 1 1 +467 / -0 giulio.starace@gmail.com giulio.starace@gmail.com
registry.py
in evals
 242 27 2023-03-21 2024-03-28 22 17 +316 / -145 343165+rlbayes@users.norepl... oliver.jaffe@hotmail.co.uk
make-me-pay.yaml
in evals/registry/evals
 59 - 2023-09-19 2024-03-13 3 2 +262 / -195 140545726+ianmckenzie-oai@u... oliver.jaffe@hotmail.co.uk
mmlu.yaml
in evals/registry/evals
 399 - 2023-05-08 2023-05-08 1 1 +455 / -0 jwang47@users.noreply.githu... jwang47@users.noreply.githu...
plot_experiments.py
in evals/elsuite/hr_ml_agent_bench/scripts
 307 - 2024-03-21 2024-03-21 1 1 +442 / -0 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
nxn_maze_eval_generator.py
in evals/registry/data/mazes
 162 9 2023-06-14 2023-09-26 2 2 +414 / -23 douglas.monsky@gmail.com bomarni@googlemail.com
eval.py
in evals/elsuite/skill_acquisition
 313 7 2024-03-19 2024-03-19 1 1 +428 / -0 inwaves@users.noreply.githu... inwaves@users.noreply.githu...
mmmu.yaml
in evals/registry/evals
 390 - 2023-12-21 2023-12-21 1 1 +419 / -0 erik.t.ritter@gmail.com erik.t.ritter@gmail.com
make_plots.py
in evals/elsuite/identifying_variables/scripts
 325 17 2024-03-19 2024-03-19 1 1 +400 / -0 giulio.starace@gmail.com giulio.starace@gmail.com
makemepay.py
in evals/elsuite/make_me_pay
 222 8 2023-09-19 2023-11-15 2 2 +345 / -50 140545726+ianmckenzie-oai@u... oliver.jaffe@hotmail.co.uk
validators.py
in evals/registry/data/word_association/corpus_tools
 151 11 2023-07-04 2024-01-10 4 4 +345 / -39 douglas.monsky@gmail.com 140545726+ianmckenzie-oai@u...
environment.py
in evals/elsuite/hr_ml_agent_bench
 283 21 2024-03-21 2024-03-21 1 1 +383 / -0 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
playwright_api.py
in evals/elsuite/multistep_web_tasks/webarena/core
 279 32 2024-03-21 2024-03-21 1 1 +378 / -0 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
low_level_actions.py
in evals/elsuite/hr_ml_agent_bench
 304 13 2024-03-21 2024-03-21 1 1 +370 / -0 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
evaluators.py
in evals/elsuite/multistep_web_tasks/webarena/evaluation_harness
 273 19 2024-03-21 2024-03-21 1 1 +356 / -0 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
eval_run.py
in evals/elsuite/multistep_web_tasks/webarena
 277 14 2024-03-21 2024-03-21 1 1 +356 / -0 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
classify_utils.py
in evals/elsuite/modelgraded
 145 8 2023-04-04 2023-09-18 5 2 +269 / -82 343165+rlbayes@users.norepl... 97272807+sohenze@users.nore...
base.py
in evals/elsuite/modelgraded
 16 - 2023-04-04 2023-04-27 4 1 +187 / -162 343165+rlbayes@users.norepl... 343165+rlbayes@users.norepl...
corrset.py
in evals/elsuite/identifying_variables/renderers
 216 11 2024-03-19 2024-03-19 1 1 +346 / -0 giulio.starace@gmail.com giulio.starace@gmail.com
prompts.py
in evals/elsuite/schelling_point
 25 - 2023-09-19 2024-01-10 3 2 +337 / -1 140545726+ianmckenzie-oai@u... 140545726+ianmckenzie-oai@u...
parsePosDe.py
in evals/registry/data/german-part-of-speech
 172 - 2023-06-01 2023-09-26 2 2 +270 / -66 vasco@piranhafleisch.de bomarni@googlemail.com
eval.py
in evals/elsuite/schelling_point
 73 3 2023-09-19 2023-12-15 2 2 +215 / -119 140545726+ianmckenzie-oai@u... 129281094+james-aung@users....
prompts.py
in evals/elsuite/steganography
 55 1 2023-09-19 2023-09-19 1 1 +329 / -0 140545726+ianmckenzie-oai@u... 140545726+ianmckenzie-oai@u...
skill_acquisition.yaml
in evals/registry/solvers
 267 - 2024-03-19 2024-03-28 2 2 +308 / -21 inwaves@users.noreply.githu... oliver.jaffe@hotmail.co.uk
core.py
in evals/elsuite/make_me_say
 223 18 2023-09-19 2024-03-13 3 2 +325 / -3 140545726+ianmckenzie-oai@u... oliver.jaffe@hotmail.co.uk
make_plots.py
in evals/elsuite/already_said_that/scripts
 263 8 2024-03-19 2024-03-19 1 1 +328 / -0 giulio.starace@gmail.com giulio.starace@gmail.com
diagonal_dataset_creation.py
in evals/elsuite/cant_do_that_anymore/scripts
 216 6 2024-03-19 2024-03-19 1 1 +316 / -0 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
solver.py
in evals/solvers
 125 17 2023-11-09 2024-03-28 4 2 +265 / -49 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
raven-matrices.yaml
in evals/registry/evals
 224 - 2023-06-08 2023-06-08 1 1 +313 / -0 44269117+ggendro@users.nore... 44269117+ggendro@users.nore...
dataset_creation.py
in evals/elsuite/cant_do_that_anymore/scripts
 235 8 2024-03-19 2024-03-19 1 1 +312 / -0 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
hhh.py
in evals/solvers/prompts
 99 - 2023-11-09 2024-03-13 4 3 +212 / -90 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
eval.py
in evals/elsuite/function_deduction
 244 14 2024-03-19 2024-03-19 1 1 +302 / -0 129281094+james-aung@users.... 129281094+james-aung@users....
make_plots.py
in evals/elsuite/ballots/scripts
 233 12 2024-03-13 2024-03-13 1 1 +299 / -0 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
openai_solver.py
in evals/solvers/providers/openai
 181 18 2024-03-28 2024-03-28 1 1 +297 / -0 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
api_utils.py
in evals/utils
 15 1 2023-03-18 2024-03-26 10 8 +152 / -144 120423412+andrew-openai@use... oliver.jaffe@hotmail.co.uk
make_plots.py
in evals/elsuite/track_the_stat/scripts
 235 9 2024-03-19 2024-03-19 1 1 +296 / -0 giulio.starace@gmail.com giulio.starace@gmail.com
Files With Most Contributors (Top 50)
Based on the number of unique email addresses found in commits.

See data for all files...

File# lines# unitscreatedlast modified# changes
(days)		# contributorsline
churn		first
contributor		latest
contributor
oaieval.py
in evals/cli
 253 6 2023-03-18 2024-09-30 19 18 +369 / -276 120423412+andrew-openai@use... steven@openai.com
registry.py
in evals
 242 27 2023-03-21 2024-03-28 22 17 +316 / -145 343165+rlbayes@users.norepl... oliver.jaffe@hotmail.co.uk
data.py
in evals
 148 21 2023-03-17 2024-04-02 9 10 +70 / -56 343165+rlbayes@users.norepl... 150190178+josnyder-2@users....
record.py
in evals
 450 54 2023-03-16 2024-01-26 13 9 +218 / -52 shane@openai.com 140545726+ianmckenzie-oai@u...
eval.py
in evals
 170 15 2023-03-20 2024-03-28 9 9 +161 / -45 2406911+zhangmarvin@users.n... oliver.jaffe@hotmail.co.uk
api_utils.py
in evals/utils
 15 1 2023-03-18 2024-03-26 10 8 +152 / -144 120423412+andrew-openai@use... oliver.jaffe@hotmail.co.uk
classify.py
in evals/elsuite/modelgraded
 97 3 2023-03-16 2023-09-26 19 7 +312 / -479 shane@openai.com bomarni@googlemail.com
utils.py
in evals/elsuite
 150 13 2023-03-22 2023-09-26 10 6 +101 / -53 343165+rlbayes@users.norepl... bomarni@googlemail.com
48 3 2023-03-15 2024-01-10 6 6 +137 / -16 jasonwei@openai.com 140545726+ianmckenzie-oai@u...
openai.py
in evals/completion_fns
 147 10 2023-04-11 2024-03-26 5 6 +203 / -22 jwang47@users.noreply.githu... oliver.jaffe@hotmail.co.uk
utils.py
in evals/elsuite/make_me_say
 34 4 2023-09-19 2023-12-20 6 5 +67 / -21 140545726+ianmckenzie-oai@u... inwaves@users.noreply.githu...
185 1 2023-03-16 2024-01-10 6 5 +528 / -17 shane@openai.com 140545726+ianmckenzie-oai@u...
49 1 2023-03-16 2024-01-10 6 5 +29 / -71 shane@openai.com 140545726+ianmckenzie-oai@u...
includes.py
in evals/elsuite/basic
 48 3 2023-04-11 2023-06-12 7 4 +44 / -25 73198383+hwchung27@users.no... 120423412+andrew-openai@use...
match.py
in evals/elsuite/basic
 57 3 2023-04-11 2023-09-26 5 4 +27 / -7 73198383+hwchung27@users.no... lukevanseters@gmail.com
fuzzy_match.py
in evals/elsuite/basic
 49 3 2023-04-11 2023-06-05 5 4 +28 / -17 73198383+hwchung27@users.no... jwang47@users.noreply.githu...
base.py
in evals
 51 1 2023-03-20 2023-09-26 5 4 +13 / -76 343165+rlbayes@users.norepl... lukevanseters@gmail.com
api.py
in evals
 61 5 2023-03-28 2023-06-08 5 4 +57 / -200 jwang47@users.noreply.githu... 131678108+wingsdrafterwork@...
oaievalset.py
in evals/cli
 111 8 2023-03-21 2024-01-10 5 4 +57 / -12 343165+rlbayes@users.norepl... 140545726+ianmckenzie-oai@u...
autoeval.py
in evals/elsuite/make_me_say
 116 3 2023-09-19 2023-12-05 4 4 +149 / -6 140545726+ianmckenzie-oai@u... erik.t.ritter@gmail.com
validators.py
in evals/registry/data/word_association/corpus_tools
 151 11 2023-07-04 2024-01-10 4 4 +345 / -39 douglas.monsky@gmail.com 140545726+ianmckenzie-oai@u...
langchain_math.py
in evals/completion_fns
 21 4 2023-04-11 2023-12-05 3 4 +44 / -13 jwang47@users.noreply.githu... erik.t.ritter@gmail.com
langchain_llm.py
in evals/completion_fns
 70 7 2023-04-11 2024-01-03 3 4 +99 / -10 jwang47@users.noreply.githu... z@hyperf.io
test-modelgraded.yaml
in evals/registry/evals
 81 - 2023-03-20 2023-09-18 7 3 +56 / -48 343165+rlbayes@users.norepl... 55913678+cholotook@users.no...
test-modelgraded-battle.yaml
in evals/registry/evals
 36 - 2023-03-16 2023-04-24 5 3 +11 / -9 shane@openai.com 343165+rlbayes@users.norepl...
test-all.yaml
in evals/registry/eval_sets
 21 - 2023-03-16 2023-04-24 5 3 +8 / -3 shane@openai.com 343165+rlbayes@users.norepl...
base.py
in evals/prompt
 64 8 2023-03-29 2024-03-13 4 3 +24 / -17 343165+rlbayes@users.norepl... oliver.jaffe@hotmail.co.uk
hhh.py
in evals/solvers/prompts
 99 - 2023-11-09 2024-03-13 4 3 +212 / -90 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
human_cli_solver.py
in evals/solvers
 29 3 2023-11-09 2024-03-13 4 3 +60 / -12 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
coqa-ex.yaml
in evals/registry/evals
 55 - 2023-03-17 2023-04-24 4 3 +57 / -18 343165+rlbayes@users.norepl... 343165+rlbayes@users.norepl...
in
 4 - 2023-03-30 2023-04-12 3 3 +3 / -2 jwang47@users.noreply.githu... 120423412+andrew-openai@use...
utils.py
in evals/elsuite/make_me_pay
 47 6 2023-09-19 2023-12-05 3 3 +125 / -50 140545726+ianmckenzie-oai@u... erik.t.ritter@gmail.com
solvers.py
in evals/elsuite/sandbagging
 152 16 2023-11-15 2024-01-29 3 3 +221 / -3 oliver.jaffe@hotmail.co.uk junshern@users.noreply.gith...
strategy_solver.py
in evals/elsuite/bluff
 88 5 2023-11-15 2024-03-28 3 3 +195 / -19 33967107+johny-b@users.nore... oliver.jaffe@hotmail.co.uk
metrics.py
in evals
 52 8 2023-06-02 2023-09-26 3 3 +13 / -16 21045365+kjbilton@users.nor... bomarni@googlemail.com
retrieval.py
in evals/completion_fns
 68 6 2023-04-12 2024-01-10 3 3 +128 / -8 120423412+andrew-openai@use... 140545726+ianmckenzie-oai@u...
schelling_point.yaml
in evals/registry/evals
 66 - 2023-09-19 2024-03-13 3 3 +71 / -0 140545726+ianmckenzie-oai@u... oliver.jaffe@hotmail.co.uk
ballots.yaml
in evals/registry/evals
 34 - 2023-09-19 2024-03-13 3 3 +39 / -0 140545726+ianmckenzie-oai@u... oliver.jaffe@hotmail.co.uk
langchain_llms.yaml
in evals/registry/completion_fns
 24 - 2023-04-11 2023-12-21 3 3 +28 / -1 73198383+hwchung27@users.no... lorenzo.pacchiardi@stats.ox...
32 - 2023-07-13 2026-04-14 3 3 +20 / -5 pan93412@gmail.com 146999853+mcgrew-oai@users....
parse_yaml.py
in .github/workflows
 10 1 2023-03-18 2023-09-26 2 3 +28 / -1 120423412+andrew-openai@use... bomarni@googlemail.com
classify_utils.py
in evals/elsuite/modelgraded
 145 8 2023-04-04 2023-09-18 5 2 +269 / -82 343165+rlbayes@users.norepl... 97272807+sohenze@users.nore...
json_validator.py
in evals/elsuite/basic
 41 4 2023-04-14 2023-06-12 4 2 +58 / -9 120423412+andrew-openai@use... 120423412+andrew-openai@use...
solver.py
in evals/solvers
 125 17 2023-11-09 2024-03-28 4 2 +265 / -49 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
utils.py
in evals/solvers
 37 2 2023-11-09 2024-03-28 4 2 +137 / -82 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
best.yaml
in evals/registry/modelgraded
 10 - 2023-03-20 2023-09-18 4 2 +15 / -3 343165+rlbayes@users.norepl... 55913678+cholotook@users.no...
eval.py
in evals/elsuite/ballots
 161 3 2023-09-19 2023-12-20 3 2 +202 / -8 140545726+ianmckenzie-oai@u... 129281094+james-aung@users....
prompts.py
in evals/elsuite/ballots
 44 - 2023-09-19 2024-01-10 3 2 +146 / -6 140545726+ianmckenzie-oai@u... 140545726+ianmckenzie-oai@u...
task_description.py
in evals/elsuite/make_me_pay
 57 - 2023-11-15 2024-03-13 3 2 +64 / -4 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
eval.py
in evals/elsuite/make_me_pay
 126 3 2023-09-19 2024-03-13 3 2 +188 / -33 140545726+ianmckenzie-oai@u... oliver.jaffe@hotmail.co.uk
Files With Least Contributors (Top 50)
Based on the number of unique email addresses found in commits.

See data for all files...

File# lines# unitscreatedlast modified# changes
(days)		# contributorsline
churn		first
contributor		latest
contributor
14 1
jsonl
samples.jsonl
in evals/registry/data/coqa
 3 -
jsonl
match.jsonl
in evals/registry/data/coqa
 3 -
generate_samples.ipynb
in evals/registry/data/backgammon
 1349 - 2023-06-22 2023-06-22 1 1 +1349 / -0 bakebrain@gmail.com bakebrain@gmail.com
actions.py
in evals/elsuite/multistep_web_tasks/webarena/browser_env
 1014 50 2024-03-21 2024-03-21 1 1 +1264 / -0 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
tools.py
in evals/elsuite/bugged_tools
 497 37 2024-03-19 2024-03-19 1 1 +653 / -0 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
processors.py
in evals/elsuite/multistep_web_tasks/webarena/browser_env
 495 21 2024-03-21 2024-03-21 1 1 +626 / -0 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
make_plots.py
in evals/elsuite/error_recovery/scripts
 446 20 2024-03-19 2024-03-19 1 1 +597 / -0 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
session.py
in evals/elsuite/multistep_web_tasks
 416 28 2024-03-21 2024-03-21 1 1 +518 / -0 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
mmlu.yaml
in evals/registry/evals
 399 - 2023-05-08 2023-05-08 1 1 +455 / -0 jwang47@users.noreply.githu... jwang47@users.noreply.githu...
mmmu.yaml
in evals/registry/evals
 390 - 2023-12-21 2023-12-21 1 1 +419 / -0 erik.t.ritter@gmail.com erik.t.ritter@gmail.com
make_plots.py
in evals/elsuite/identifying_variables/scripts
 325 17 2024-03-19 2024-03-19 1 1 +400 / -0 giulio.starace@gmail.com giulio.starace@gmail.com
gen_data.py
in evals/elsuite/identifying_variables/scripts
 319 13 2024-03-19 2024-03-19 1 1 +467 / -0 giulio.starace@gmail.com giulio.starace@gmail.com
eval.py
in evals/elsuite/skill_acquisition
 313 7 2024-03-19 2024-03-19 1 1 +428 / -0 inwaves@users.noreply.githu... inwaves@users.noreply.githu...
plot_experiments.py
in evals/elsuite/hr_ml_agent_bench/scripts
 307 - 2024-03-21 2024-03-21 1 1 +442 / -0 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
low_level_actions.py
in evals/elsuite/hr_ml_agent_bench
 304 13 2024-03-21 2024-03-21 1 1 +370 / -0 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
environment.py
in evals/elsuite/hr_ml_agent_bench
 283 21 2024-03-21 2024-03-21 1 1 +383 / -0 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
constants.py
in evals/elsuite/multistep_web_tasks/webarena/browser_env
 282 - 2024-03-21 2024-03-21 1 1 +295 / -0 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
playwright_api.py
in evals/elsuite/multistep_web_tasks/webarena/core
 279 32 2024-03-21 2024-03-21 1 1 +378 / -0 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
eval_run.py
in evals/elsuite/multistep_web_tasks/webarena
 277 14 2024-03-21 2024-03-21 1 1 +356 / -0 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
evaluators.py
in evals/elsuite/multistep_web_tasks/webarena/evaluation_harness
 273 19 2024-03-21 2024-03-21 1 1 +356 / -0 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
make_plots.py
in evals/elsuite/already_said_that/scripts
 263 8 2024-03-19 2024-03-19 1 1 +328 / -0 giulio.starace@gmail.com giulio.starace@gmail.com
eval.py
in evals/elsuite/function_deduction
 244 14 2024-03-19 2024-03-19 1 1 +302 / -0 129281094+james-aung@users.... 129281094+james-aung@users....
make_plots.py
in evals/elsuite/track_the_stat/scripts
 235 9 2024-03-19 2024-03-19 1 1 +296 / -0 giulio.starace@gmail.com giulio.starace@gmail.com
dataset_creation.py
in evals/elsuite/cant_do_that_anymore/scripts
 235 8 2024-03-19 2024-03-19 1 1 +312 / -0 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
make_plots.py
in evals/elsuite/ballots/scripts
 233 12 2024-03-13 2024-03-13 1 1 +299 / -0 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
eval.py
in evals/elsuite/identifying_variables
 227 14 2024-03-19 2024-03-19 1 1 +292 / -0 giulio.starace@gmail.com giulio.starace@gmail.com
raven-matrices.yaml
in evals/registry/evals
 224 - 2023-06-08 2023-06-08 1 1 +313 / -0 44269117+ggendro@users.nore... 44269117+ggendro@users.nore...
corrset.py
in evals/elsuite/identifying_variables/renderers
 216 11 2024-03-19 2024-03-19 1 1 +346 / -0 giulio.starace@gmail.com giulio.starace@gmail.com
diagonal_dataset_creation.py
in evals/elsuite/cant_do_that_anymore/scripts
 216 6 2024-03-19 2024-03-19 1 1 +316 / -0 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
eval.py
in evals/elsuite/bugged_tools
 210 9 2024-03-19 2024-03-19 1 1 +285 / -0 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
eval.py
in evals/elsuite/self_prompting
 210 7 2023-11-15 2023-11-15 1 1 +261 / -0 junshern@users.noreply.gith... junshern@users.noreply.gith...
cards.py
in evals/elsuite/bluff/bluff
 206 37 2023-11-15 2023-11-15 1 1 +288 / -0 33967107+johny-b@users.nore... 33967107+johny-b@users.nore...
eval.py
in evals/elsuite/error_recovery
 204 9 2024-03-19 2024-03-19 1 1 +284 / -0 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
pieces.py
in evals/elsuite/cant_do_that_anymore/chess
 203 9 2024-03-19 2024-03-19 1 1 +263 / -0 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
custom_datasets.py
in evals/elsuite/steganography/scripts/dataset
 197 12 2024-03-13 2024-03-13 1 1 +246 / -0 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
make_plots.py
in evals/elsuite/function_deduction/scripts
 195 4 2024-03-19 2024-03-19 1 1 +256 / -0 129281094+james-aung@users.... 129281094+james-aung@users....
basic_browser_env.py
in evals/elsuite/multistep_web_tasks/webarena/browser_env
 191 11 2024-03-21 2024-03-21 1 1 +248 / -0 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
high_level_actions.py
in evals/elsuite/hr_ml_agent_bench
 191 4 2024-03-21 2024-03-21 1 1 +260 / -0 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
utils.py
in evals/elsuite/multistep_web_tasks/webarena/core
 188 7 2024-03-21 2024-03-21 1 1 +223 / -0 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
openai_assistants_solver.py
in evals/solvers/providers/openai
 186 11 2024-03-28 2024-03-28 1 1 +272 / -0 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
solver_tools_convo.py
in evals/elsuite
 181 11 2024-03-19 2024-03-19 1 1 +240 / -0 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
openai_solver.py
in evals/solvers/providers/openai
 181 18 2024-03-28 2024-03-28 1 1 +297 / -0 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
ukraine-gec.yaml
in evals/registry/evals
 180 - 2023-06-02 2023-06-02 1 1 +198 / -0 roman.mashta@gmail.com roman.mashta@gmail.com
utils.py
in evals/elsuite/cant_do_that_anymore
 178 12 2024-03-19 2024-03-19 1 1 +250 / -0 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
strong_solver.py
in evals/elsuite/multistep_web_tasks/solvers/strong_solver
 173 12 2024-03-21 2024-03-21 1 1 +226 / -0 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
autoeval.py
in evals/elsuite/hr_ml_agent_bench
 172 2 2024-03-21 2024-03-21 1 1 +214 / -0 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
eval.py
in evals/elsuite/cant_do_that_anymore
 170 4 2024-03-19 2024-03-19 1 1 +201 / -0 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
eval.py
in evals/elsuite/twenty_questions
 169 4 2024-03-19 2024-03-19 1 1 +204 / -0 inwaves@users.noreply.githu... inwaves@users.noreply.githu...
app.py
in evals/elsuite/multistep_web_tasks/docker/flask-playwright
 165 8 2024-03-21 2024-03-21 1 1 +218 / -0 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
Correlations

File Size vs. Number of Changes: 1489 points

.pre-commit-config.yaml x: 32 lines of code y: 3 # changes evals/cli/oaieval.py x: 253 lines of code y: 19 # changes evals/registry/data/imo_exact_answers/samples.jsonl x: 3 lines of code y: 1 # changes evals/registry/evals/imo_exact_answers.yaml x: 8 lines of code y: 1 # changes evals/data.py x: 148 lines of code y: 9 # changes evals/completion_fns/solver_completion_fn.py x: 47 lines of code y: 1 # changes evals/elsuite/bluff/strategy_solver.py x: 88 lines of code y: 3 # changes evals/elsuite/bugged_tools/task_description.py x: 9 lines of code y: 2 # changes evals/elsuite/function_deduction/prompts.py x: 6 lines of code y: 2 # changes evals/elsuite/function_deduction/solvers.py x: 140 lines of code y: 2 # changes evals/elsuite/hr_ml_agent_bench/solvers/baseline.py x: 90 lines of code y: 2 # changes evals/elsuite/make_me_say/eval.py x: 48 lines of code y: 3 # changes evals/eval.py x: 170 lines of code y: 9 # changes evals/registry.py x: 242 lines of code y: 22 # changes evals/registry/solvers/already_said_that.yaml x: 75 lines of code y: 2 # changes evals/registry/solvers/bluff.yaml x: 80 lines of code y: 2 # changes evals/registry/solvers/cant_do_that_anymore.yaml x: 16 lines of code y: 2 # changes evals/registry/solvers/defaults.yaml x: 294 lines of code y: 3 # changes evals/registry/solvers/error_recovery.yaml x: 33 lines of code y: 2 # changes evals/registry/solvers/function_deduction.yaml x: 174 lines of code y: 2 # changes evals/registry/solvers/hr-ml-agent-bench.yaml x: 37 lines of code y: 2 # changes evals/registry/solvers/make-me-pay.yaml x: 101 lines of code y: 2 # changes evals/registry/solvers/self_prompting.yaml x: 96 lines of code y: 2 # changes evals/registry/solvers/skill_acquisition.yaml x: 267 lines of code y: 2 # changes evals/registry/solvers/theory_of_mind.yaml x: 394 lines of code y: 3 # changes evals/registry/solvers/together.yaml x: 85 lines of code y: 2 # changes evals/solvers/memory.py x: 50 lines of code y: 1 # changes evals/solvers/nested/cot_solver.py x: 61 lines of code y: 3 # changes evals/solvers/nested/self_consistency_solver.py x: 118 lines of code y: 3 # changes evals/solvers/prompts/cot.py x: 4 lines of code y: 2 # changes evals/solvers/providers/openai/openai_assistants_solver.py x: 186 lines of code y: 1 # changes evals/solvers/providers/openai/openai_solver.py x: 181 lines of code y: 1 # changes evals/solvers/providers/together/together_solver.py x: 68 lines of code y: 1 # changes evals/solvers/solver.py x: 125 lines of code y: 4 # changes evals/solvers/utils.py x: 37 lines of code y: 4 # changes evals/registry/solvers/gemini.yaml x: 15 lines of code y: 1 # changes evals/solvers/providers/google/gemini_solver.py x: 157 lines of code y: 1 # changes evals/completion_fns/openai.py x: 147 lines of code y: 5 # changes evals/utils/api_utils.py x: 15 lines of code y: 10 # changes evals/registry/solvers/anthropic.yaml x: 90 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/__init__.py x: 1 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/actions.py x: 37 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/auto_marking.py x: 53 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/autoeval.py x: 172 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/ant/baselines/human.py x: 36 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/ant/baselines/naive.py x: 28 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/ant/scripts/grade.py x: 43 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/bipedal_walker/baselines/human.py x: 57 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/bipedal_walker/scripts/grade.py x: 31 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/cartpole/scripts/grade.py x: 40 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/cifar10/env/train.py x: 105 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/cifar10/scripts/prepare.py x: 5 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/feedback/scripts/prepare.py x: 24 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/house_price/scripts/grade.py x: 33 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/house_price/scripts/prepare.py x: 20 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/humanoid/scripts/grade.py x: 45 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/ogbn_arxiv/env/train.py x: 126 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/parkinsons_disease/env/train.py x: 124 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/parkinsons_disease/scripts/prepare.py x: 98 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/spaceship_titanic/scripts/grade.py x: 26 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/vectorization/env/train.py x: 118 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/vectorization/scripts/grade.py x: 64 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/vectorization/scripts/human_baseline.py x: 83 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/environment.py x: 283 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/eval.py x: 91 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/high_level_actions.py x: 191 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/low_level_actions.py x: 304 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/scripts/plot_experiments.py x: 307 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/scripts/run_experiments.py x: 60 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/utils.py x: 112 lines of code y: 1 # changes evals/registry/evals/hr-ml-agent-bench.yaml x: 137 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/constants.py x: 70 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/docker/flask-playwright/app.py x: 165 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/docker/homepage/templates/index.html x: 108 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/reproducibility/make_plots.py x: 94 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/session.py x: 416 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/solvers/strong_solver/strong_solver.py x: 173 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/solvers/webarena_solvers/webarena_prompts.py x: 12 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/bash_env/basic_bash_env.py x: 163 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/browser_env/actions.py x: 1014 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/browser_env/auto_login.py x: 100 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/browser_env/browser_utils.py x: 78 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/browser_env/constants.py x: 282 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/browser_env/helper_functions.py x: 129 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/browser_env/processors.py x: 495 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/core/env.py x: 75 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/core/playwright_api.py x: 279 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/core/utils.py x: 188 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/eval_run.py x: 277 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/evaluation_harness/evaluators.py x: 273 lines of code y: 1 # changes evals/registry/solvers/multistep_web_tasks.yaml x: 56 lines of code y: 1 # changes evals/elsuite/function_deduction/eval.py x: 244 lines of code y: 1 # changes evals/elsuite/function_deduction/scripts/make_plots.py x: 195 lines of code y: 1 # changes evals/elsuite/identifying_variables/eval.py x: 227 lines of code y: 1 # changes evals/elsuite/identifying_variables/graph_utils.py x: 130 lines of code y: 1 # changes evals/elsuite/identifying_variables/renderers/corrset.py x: 216 lines of code y: 1 # changes evals/elsuite/identifying_variables/scripts/gen_data.py x: 319 lines of code y: 1 # changes evals/elsuite/identifying_variables/scripts/make_plots.py x: 325 lines of code y: 1 # changes evals/elsuite/identifying_variables/structs.py x: 18 lines of code y: 1 # changes evals/elsuite/identifying_variables/utils.py x: 72 lines of code y: 1 # changes evals/elsuite/track_the_stat/scripts/make_plots.py x: 235 lines of code y: 1 # changes evals/elsuite/already_said_that/distractors.py x: 81 lines of code y: 1 # changes evals/elsuite/already_said_that/scripts/make_plots.py x: 263 lines of code y: 1 # changes evals/elsuite/already_said_that/utils.py x: 104 lines of code y: 1 # changes evals/utils/log_utils.py x: 53 lines of code y: 2 # changes evals/elsuite/twenty_questions/eval.py x: 169 lines of code y: 1 # changes evals/elsuite/skill_acquisition/eval.py x: 313 lines of code y: 1 # changes evals/elsuite/skill_acquisition/scraping/scrape_distractor_articles.py x: 76 lines of code y: 1 # changes evals/elsuite/skill_acquisition/solvers.py x: 13 lines of code y: 1 # changes evals/elsuite/skill_acquisition/utils.py x: 115 lines of code y: 1 # changes evals/elsuite/error_recovery/eval.py x: 204 lines of code y: 1 # changes evals/elsuite/error_recovery/scripts/make_plots.py x: 446 lines of code y: 1 # changes evals/elsuite/cant_do_that_anymore/chess/move_variants.py x: 113 lines of code y: 1 # changes evals/elsuite/cant_do_that_anymore/chess/notation.py x: 61 lines of code y: 1 # changes evals/elsuite/cant_do_that_anymore/utils.py x: 178 lines of code y: 1 # changes evals/elsuite/bugged_tools/bugged_tools.py x: 132 lines of code y: 1 # changes evals/elsuite/bugged_tools/eval.py x: 210 lines of code y: 1 # changes evals/elsuite/bugged_tools/tools.py x: 497 lines of code y: 1 # changes evals/elsuite/ballots/scripts/make_plots.py x: 233 lines of code y: 1 # changes evals/elsuite/basic/match_with_solvers.py x: 65 lines of code y: 2 # changes evals/elsuite/bluff/solver_player.py x: 82 lines of code y: 2 # changes evals/elsuite/make_me_pay/eval.py x: 126 lines of code y: 3 # changes evals/elsuite/make_me_pay/task_description.py x: 57 lines of code y: 3 # changes evals/elsuite/make_me_say/core.py x: 223 lines of code y: 3 # changes evals/prompt/base.py x: 64 lines of code y: 4 # changes evals/registry/evals/make-me-pay.yaml x: 59 lines of code y: 3 # changes evals/registry/evals/schelling_point.yaml x: 66 lines of code y: 3 # changes evals/registry/evals/self_prompting.yaml x: 19 lines of code y: 2 # changes evals/registry/evals/theory_of_mind.yaml x: 48 lines of code y: 2 # changes evals/solvers/human_cli_solver.py x: 29 lines of code y: 4 # changes evals/solvers/nested/fewshot_solver.py x: 91 lines of code y: 2 # changes evals/solvers/prompts/hhh.py x: 99 lines of code y: 4 # changes evals/elsuite/sandbagging/solvers.py x: 152 lines of code y: 3 # changes evals/elsuite/self_prompting/solvers/baselines.py x: 45 lines of code y: 2 # changes evals/elsuite/self_prompting/solvers/custom_cot_solver.py x: 57 lines of code y: 2 # changes evals/record.py x: 450 lines of code y: 13 # changes evals/cli/oaievalset.py x: 111 lines of code y: 5 # changes evals/elsuite/ballots/prompts.py x: 44 lines of code y: 3 # changes evals/elsuite/sandbagging/defaults.py x: 12 lines of code y: 2 # changes evals/elsuite/sandbagging/sandbagging_eval.py x: 68 lines of code y: 2 # changes evals/elsuite/schelling_point/prompts.py x: 25 lines of code y: 3 # changes evals/elsuite/schelling_point/utils.py x: 61 lines of code y: 2 # changes evals/elsuite/steganography/eval.py x: 69 lines of code y: 2 # changes evals/registry/data/word_association/corpus_tools/validators.py x: 151 lines of code y: 4 # changes scripts/battle_generator.py x: 49 lines of code y: 6 # changes scripts/modelgraded_generator.py x: 185 lines of code y: 6 # changes scripts/pattern_identification_generator.py x: 48 lines of code y: 6 # changes evals/elsuite/mmmu/eval.py x: 159 lines of code y: 3 # changes evals/completion_fns/langchain_llm.py x: 70 lines of code y: 3 # changes evals/registry/completion_fns/langchain_llms.yaml x: 24 lines of code y: 3 # changes evals/registry/evals/mmmu.yaml x: 390 lines of code y: 1 # changes evals/elsuite/make_me_say/utils.py x: 34 lines of code y: 6 # changes evals/elsuite/ballots/eval.py x: 161 lines of code y: 3 # changes evals/elsuite/schelling_point/eval.py x: 73 lines of code y: 2 # changes evals/elsuite/bluff/eval.py x: 164 lines of code y: 2 # changes evals/completion_fns/langchain_math.py x: 21 lines of code y: 3 # changes evals/elsuite/make_me_say/autoeval.py x: 116 lines of code y: 4 # changes evals/elsuite/bluff/bluff/cards.py x: 206 lines of code y: 1 # changes evals/elsuite/make_me_pay/makemepay.py x: 222 lines of code y: 2 # changes evals/elsuite/lambada.py x: 39 lines of code y: 2 # changes evals/elsuite/modelgraded/classify.py x: 97 lines of code y: 19 # changes evals/elsuite/utils.py x: 150 lines of code y: 10 # changes evals/metrics.py x: 52 lines of code y: 3 # changes evals/registry/data/canto_wu_pronunciation/csv_to_json.py x: 55 lines of code y: 2 # changes evals/registry/data/german-part-of-speech/parsePosDe.py x: 172 lines of code y: 2 # changes evals/registry/data/mapping_to_matricies/data_generator.py x: 30 lines of code y: 2 # changes evals/registry/data/mazes/nxn_maze_eval_generator.py x: 162 lines of code y: 2 # changes evals/registry/data/simple_physics_engine/wave_function_collapse.py x: 157 lines of code y: 2 # changes evals/registry/data/solve-for-variable/tools/solve.py x: 231 lines of code y: 2 # changes evals/registry/data/solve-for-variable/tools/tester.py x: 94 lines of code y: 2 # changes evals/registry/data/word_association/corpus_tools/pipelines.py x: 14 lines of code y: 2 # changes evals/registry/data/word_association/corpus_tools/processor.py x: 36 lines of code y: 2 # changes evals/registry/data/word_association/corpus_tools/sample_generators.py x: 154 lines of code y: 2 # changes evals/base.py x: 51 lines of code y: 5 # changes evals/elsuite/basic/match.py x: 57 lines of code y: 5 # changes evals/registry/evals/test-modelgraded.yaml x: 81 lines of code y: 7 # changes evals/registry/modelgraded/best.yaml x: 10 lines of code y: 4 # changes evals/elsuite/modelgraded/classify_utils.py x: 145 lines of code y: 5 # changes evals/registry/data/backgammon/generate_samples.ipynb x: 1349 lines of code y: 1 # changes evals/elsuite/basic/includes.py x: 48 lines of code y: 7 # changes evals/elsuite/basic/json_validator.py x: 41 lines of code y: 4 # changes evals/api.py x: 61 lines of code y: 5 # changes evals/registry/evals/raven-matrices.yaml x: 224 lines of code y: 1 # changes evals/registry/evals/mmlu.yaml x: 399 lines of code y: 1 # changes evals/elsuite/modelgraded/base.py x: 16 lines of code y: 4 # changes evals/registry/eval_sets/test-all.yaml x: 21 lines of code y: 5 # changes evals/registry/evals/coqa-ex.yaml x: 55 lines of code y: 4 # changes evals/registry/evals/test-modelgraded-battle.yaml x: 36 lines of code y: 5 # changes evals/registry/modelgraded/closedqa.yaml x: 21 lines of code y: 2 # changes MANIFEST.in x: 4 lines of code y: 3 # changes evals/registry/eval_sets/test-modelgraded.yaml x: 14 lines of code y: 3 # changes evals/registry/evals/stock-options.yaml x: 96 lines of code y: 1 # changes evals/registry/evals/test-modelgraded-generated.yaml x: 9 lines of code y: 3 # changes
22.0
# changes
  min: 1.0
  average: 1.24
  25th percentile: 1.0
  median: 1.0
  75th percentile: 1.0
  max: 22.0
0 1349.0
lines of code
min: 1.0 | average: 27.66 | 25th percentile: 3.0 | median: 3.0 | 75th percentile: 16.0 | max: 1349.0

Number of Contributors vs. Number of Changes: 1489 points

.pre-commit-config.yaml x: 3 # contributors y: 3 # changes evals/cli/oaieval.py x: 18 # contributors y: 19 # changes evals/registry/data/imo_exact_answers/samples.jsonl x: 1 # contributors y: 1 # changes evals/data.py x: 10 # contributors y: 9 # changes evals/elsuite/bugged_tools/task_description.py x: 1 # contributors y: 2 # changes evals/elsuite/function_deduction/prompts.py x: 2 # contributors y: 2 # changes evals/elsuite/make_me_say/eval.py x: 2 # contributors y: 3 # changes evals/eval.py x: 9 # contributors y: 9 # changes evals/registry.py x: 17 # contributors y: 22 # changes evals/solvers/solver.py x: 2 # contributors y: 4 # changes evals/completion_fns/openai.py x: 6 # contributors y: 5 # changes evals/utils/api_utils.py x: 8 # contributors y: 10 # changes evals/prompt/base.py x: 3 # contributors y: 4 # changes evals/record.py x: 9 # contributors y: 13 # changes evals/cli/oaievalset.py x: 4 # contributors y: 5 # changes evals/registry/data/word_association/corpus_tools/validators.py x: 4 # contributors y: 4 # changes scripts/battle_generator.py x: 5 # contributors y: 6 # changes scripts/pattern_identification_generator.py x: 6 # contributors y: 6 # changes evals/elsuite/mmmu/eval.py x: 1 # contributors y: 3 # changes evals/completion_fns/langchain_llm.py x: 4 # contributors y: 3 # changes .github/workflows/parse_yaml.py x: 3 # contributors y: 2 # changes evals/elsuite/modelgraded/classify.py x: 7 # contributors y: 19 # changes evals/elsuite/utils.py x: 6 # contributors y: 10 # changes evals/registry/evals/test-modelgraded.yaml x: 3 # contributors y: 7 # changes evals/elsuite/modelgraded/classify_utils.py x: 2 # contributors y: 5 # changes evals/elsuite/basic/includes.py x: 4 # contributors y: 7 # changes evals/elsuite/modelgraded/base.py x: 1 # contributors y: 4 # changes evals/registry/eval_sets/test-all.yaml x: 3 # contributors y: 5 # changes evals/registry/data/logiqa/logiqa.jsonl x: 2 # contributors y: 1 # changes
22.0
# changes
  min: 1.0
  average: 1.24
  25th percentile: 1.0
  median: 1.0
  75th percentile: 1.0
  max: 22.0
0 18.0
# contributors
min: 1.0 | average: 1.18 | 25th percentile: 1.0 | median: 1.0 | 75th percentile: 1.0 | max: 18.0

Number of Contributors vs. File Size: 1489 points

.pre-commit-config.yaml x: 3 # contributors y: 32 lines of code evals/cli/oaieval.py x: 18 # contributors y: 253 lines of code evals/registry/data/imo_exact_answers/samples.jsonl x: 1 # contributors y: 3 lines of code evals/registry/evals/imo_exact_answers.yaml x: 1 # contributors y: 8 lines of code evals/data.py x: 10 # contributors y: 148 lines of code evals/completion_fns/solver_completion_fn.py x: 1 # contributors y: 47 lines of code evals/elsuite/bluff/strategy_solver.py x: 3 # contributors y: 88 lines of code evals/elsuite/function_deduction/prompts.py x: 2 # contributors y: 6 lines of code evals/elsuite/function_deduction/solvers.py x: 2 # contributors y: 140 lines of code evals/elsuite/hr_ml_agent_bench/solvers/baseline.py x: 2 # contributors y: 90 lines of code evals/elsuite/make_me_say/eval.py x: 2 # contributors y: 48 lines of code evals/eval.py x: 9 # contributors y: 170 lines of code evals/registry.py x: 17 # contributors y: 242 lines of code evals/registry/solvers/already_said_that.yaml x: 2 # contributors y: 75 lines of code evals/registry/solvers/bluff.yaml x: 2 # contributors y: 80 lines of code evals/registry/solvers/cant_do_that_anymore.yaml x: 1 # contributors y: 16 lines of code evals/registry/solvers/defaults.yaml x: 2 # contributors y: 294 lines of code evals/registry/solvers/error_recovery.yaml x: 1 # contributors y: 33 lines of code evals/registry/solvers/function_deduction.yaml x: 2 # contributors y: 174 lines of code evals/registry/solvers/hr-ml-agent-bench.yaml x: 2 # contributors y: 37 lines of code evals/registry/solvers/make-me-pay.yaml x: 2 # contributors y: 101 lines of code evals/registry/solvers/self_prompting.yaml x: 2 # contributors y: 96 lines of code evals/registry/solvers/skill_acquisition.yaml x: 2 # contributors y: 267 lines of code evals/registry/solvers/theory_of_mind.yaml x: 2 # contributors y: 394 lines of code evals/solvers/memory.py x: 1 # contributors y: 50 lines of code evals/solvers/nested/cot_solver.py x: 2 # contributors y: 61 lines of code evals/solvers/nested/self_consistency_solver.py x: 2 # contributors y: 118 lines of code evals/solvers/prompts/cot.py x: 2 # contributors y: 4 lines of code evals/solvers/providers/openai/openai_assistants_solver.py x: 1 # contributors y: 186 lines of code evals/solvers/providers/openai/openai_solver.py x: 1 # contributors y: 181 lines of code evals/solvers/providers/together/together_solver.py x: 1 # contributors y: 68 lines of code evals/solvers/solver.py x: 2 # contributors y: 125 lines of code evals/registry/solvers/gemini.yaml x: 1 # contributors y: 15 lines of code evals/solvers/providers/google/gemini_solver.py x: 1 # contributors y: 157 lines of code evals/completion_fns/openai.py x: 6 # contributors y: 147 lines of code evals/utils/api_utils.py x: 8 # contributors y: 15 lines of code evals/registry/solvers/anthropic.yaml x: 1 # contributors y: 90 lines of code evals/elsuite/hr_ml_agent_bench/autoeval.py x: 1 # contributors y: 172 lines of code evals/elsuite/hr_ml_agent_bench/benchmarks/ant/baselines/naive.py x: 1 # contributors y: 28 lines of code evals/elsuite/hr_ml_agent_bench/benchmarks/bipedal_walker/baselines/human.py x: 1 # contributors y: 57 lines of code evals/elsuite/hr_ml_agent_bench/benchmarks/cartpole/scripts/grade.py x: 1 # contributors y: 40 lines of code evals/elsuite/hr_ml_agent_bench/benchmarks/cifar10/env/train.py x: 1 # contributors y: 105 lines of code evals/elsuite/hr_ml_agent_bench/benchmarks/feedback/scripts/prepare.py x: 1 # contributors y: 24 lines of code evals/elsuite/hr_ml_agent_bench/benchmarks/ogbn_arxiv/env/train.py x: 1 # contributors y: 126 lines of code evals/elsuite/hr_ml_agent_bench/benchmarks/parkinsons_disease/scripts/prepare.py x: 1 # contributors y: 98 lines of code evals/elsuite/hr_ml_agent_bench/benchmarks/vectorization/env/train.py x: 1 # contributors y: 118 lines of code evals/elsuite/hr_ml_agent_bench/benchmarks/vectorization/scripts/human_baseline.py x: 1 # contributors y: 83 lines of code evals/elsuite/hr_ml_agent_bench/environment.py x: 1 # contributors y: 283 lines of code evals/elsuite/hr_ml_agent_bench/eval.py x: 1 # contributors y: 91 lines of code evals/elsuite/hr_ml_agent_bench/low_level_actions.py x: 1 # contributors y: 304 lines of code evals/elsuite/hr_ml_agent_bench/scripts/run_experiments.py x: 1 # contributors y: 60 lines of code evals/elsuite/hr_ml_agent_bench/utils.py x: 1 # contributors y: 112 lines of code evals/registry/evals/hr-ml-agent-bench.yaml x: 1 # contributors y: 137 lines of code evals/elsuite/multistep_web_tasks/constants.py x: 1 # contributors y: 70 lines of code evals/elsuite/multistep_web_tasks/docker/flask-playwright/app.py x: 1 # contributors y: 165 lines of code evals/elsuite/multistep_web_tasks/docker/homepage/templates/index.html x: 1 # contributors y: 108 lines of code evals/elsuite/multistep_web_tasks/session.py x: 1 # contributors y: 416 lines of code evals/elsuite/multistep_web_tasks/webarena/bash_env/basic_bash_env.py x: 1 # contributors y: 163 lines of code evals/elsuite/multistep_web_tasks/webarena/browser_env/actions.py x: 1 # contributors y: 1014 lines of code evals/elsuite/multistep_web_tasks/webarena/browser_env/browser_utils.py x: 1 # contributors y: 78 lines of code evals/elsuite/multistep_web_tasks/webarena/browser_env/helper_functions.py x: 1 # contributors y: 129 lines of code evals/elsuite/multistep_web_tasks/webarena/browser_env/processors.py x: 1 # contributors y: 495 lines of code evals/elsuite/multistep_web_tasks/webarena/core/playwright_api.py x: 1 # contributors y: 279 lines of code evals/elsuite/multistep_web_tasks/webarena/evaluation_harness/evaluators.py x: 1 # contributors y: 273 lines of code evals/elsuite/function_deduction/eval.py x: 1 # contributors y: 244 lines of code evals/elsuite/function_deduction/scripts/make_plots.py x: 1 # contributors y: 195 lines of code evals/elsuite/identifying_variables/eval.py x: 1 # contributors y: 227 lines of code evals/elsuite/identifying_variables/renderers/corrset.py x: 1 # contributors y: 216 lines of code evals/elsuite/identifying_variables/scripts/gen_data.py x: 1 # contributors y: 319 lines of code evals/elsuite/identifying_variables/scripts/make_plots.py x: 1 # contributors y: 325 lines of code evals/elsuite/track_the_stat/scripts/make_plots.py x: 1 # contributors y: 235 lines of code evals/elsuite/already_said_that/scripts/make_plots.py x: 1 # contributors y: 263 lines of code evals/elsuite/skill_acquisition/eval.py x: 1 # contributors y: 313 lines of code evals/elsuite/error_recovery/eval.py x: 1 # contributors y: 204 lines of code evals/elsuite/error_recovery/scripts/make_plots.py x: 1 # contributors y: 446 lines of code evals/elsuite/cant_do_that_anymore/utils.py x: 1 # contributors y: 178 lines of code evals/elsuite/bugged_tools/eval.py x: 1 # contributors y: 210 lines of code evals/elsuite/ballots/scripts/make_plots.py x: 1 # contributors y: 233 lines of code evals/elsuite/basic/match_with_solvers.py x: 2 # contributors y: 65 lines of code evals/elsuite/make_me_pay/task_description.py x: 2 # contributors y: 57 lines of code evals/elsuite/make_me_say/core.py x: 2 # contributors y: 223 lines of code evals/elsuite/steganography/scripts/dataset/custom_datasets.py x: 1 # contributors y: 197 lines of code evals/prompt/base.py x: 3 # contributors y: 64 lines of code evals/registry/evals/self_prompting.yaml x: 2 # contributors y: 19 lines of code evals/solvers/human_cli_solver.py x: 3 # contributors y: 29 lines of code evals/solvers/nested/fewshot_solver.py x: 2 # contributors y: 91 lines of code evals/solvers/prompts/hhh.py x: 3 # contributors y: 99 lines of code evals/elsuite/sandbagging/solvers.py x: 3 # contributors y: 152 lines of code evals/record.py x: 9 # contributors y: 450 lines of code evals/cli/oaievalset.py x: 4 # contributors y: 111 lines of code evals/elsuite/ballots/prompts.py x: 2 # contributors y: 44 lines of code evals/elsuite/sandbagging/defaults.py x: 2 # contributors y: 12 lines of code evals/elsuite/schelling_point/prompts.py x: 2 # contributors y: 25 lines of code evals/registry/data/word_association/corpus_tools/validators.py x: 4 # contributors y: 151 lines of code scripts/battle_generator.py x: 5 # contributors y: 49 lines of code scripts/modelgraded_generator.py x: 5 # contributors y: 185 lines of code scripts/pattern_identification_generator.py x: 6 # contributors y: 48 lines of code evals/completion_fns/langchain_llm.py x: 4 # contributors y: 70 lines of code evals/registry/completion_fns/langchain_llms.yaml x: 3 # contributors y: 24 lines of code evals/registry/evals/mmmu.yaml x: 1 # contributors y: 390 lines of code evals/elsuite/make_me_say/utils.py x: 5 # contributors y: 34 lines of code evals/elsuite/ballots/eval.py x: 2 # contributors y: 161 lines of code evals/elsuite/schelling_point/eval.py x: 2 # contributors y: 73 lines of code evals/completion_fns/langchain_math.py x: 4 # contributors y: 21 lines of code evals/elsuite/make_me_pay/utils.py x: 3 # contributors y: 47 lines of code evals/elsuite/make_me_say/autoeval.py x: 4 # contributors y: 116 lines of code .github/workflows/parse_yaml.py x: 3 # contributors y: 10 lines of code evals/elsuite/lambada.py x: 2 # contributors y: 39 lines of code evals/elsuite/modelgraded/classify.py x: 7 # contributors y: 97 lines of code evals/elsuite/utils.py x: 6 # contributors y: 150 lines of code evals/metrics.py x: 3 # contributors y: 52 lines of code evals/registry/data/mapping_to_matricies/data_generator.py x: 2 # contributors y: 30 lines of code evals/registry/data/simple_physics_engine/wave_function_collapse.py x: 2 # contributors y: 157 lines of code evals/registry/data/solve-for-variable/tools/solve.py x: 2 # contributors y: 231 lines of code evals/registry/data/word_association/corpus_tools/sample_generators.py x: 2 # contributors y: 154 lines of code evals/base.py x: 4 # contributors y: 51 lines of code evals/elsuite/basic/match.py x: 4 # contributors y: 57 lines of code evals/registry/evals/test-modelgraded.yaml x: 3 # contributors y: 81 lines of code evals/elsuite/modelgraded/classify_utils.py x: 2 # contributors y: 145 lines of code evals/registry/data/backgammon/generate_samples.ipynb x: 1 # contributors y: 1349 lines of code evals/api.py x: 4 # contributors y: 61 lines of code evals/registry/evals/mmlu.yaml x: 1 # contributors y: 399 lines of code evals/registry/eval_sets/test-all.yaml x: 3 # contributors y: 21 lines of code evals/registry/evals/coqa-ex.yaml x: 3 # contributors y: 55 lines of code MANIFEST.in x: 3 # contributors y: 4 lines of code
1349.0
lines of code
  min: 1.0
  average: 27.66
  25th percentile: 3.0
  median: 3.0
  75th percentile: 16.0
  max: 1349.0
0 18.0
# contributors
min: 1.0 | average: 1.18 | 25th percentile: 1.0 | median: 1.0 | 75th percentile: 1.0 | max: 18.0

Number of Changes vs. Line Churn: 1741 points

.pre-commit-config.yaml x: 3 # changes y: 25 line churn .github/workflows/run_tests.yaml x: 6 # changes y: 46 line churn .github/workflows/test_eval.yaml x: 5 # changes y: 142 line churn evals/elsuite/incontext_rl/README.md x: 2 # changes y: 148 line churn evals/elsuite/incontext_rl/anti-cot_solver.py x: 2 # changes y: 76 line churn evals/elsuite/incontext_rl/baselines.py x: 2 # changes y: 236 line churn evals/elsuite/incontext_rl/defaults.py x: 2 # changes y: 60 line churn evals/elsuite/incontext_rl/env_setup.py x: 2 # changes y: 24 line churn evals/elsuite/incontext_rl/eval.py x: 2 # changes y: 598 line churn evals/elsuite/incontext_rl/requirements.txt x: 2 # changes y: 6 line churn evals/elsuite/incontext_rl/scripts/plot_experiments.py x: 2 # changes y: 726 line churn evals/elsuite/incontext_rl/scripts/qlearning_baseline.ipynb x: 2 # changes y: 804 line churn evals/registry/evals/incontext_rl.yaml x: 2 # changes y: 124 line churn evals/registry/solvers/incontext_rl.yaml x: 3 # changes y: 58 line churn README.md x: 16 # changes y: 136 line churn evals/cli/oaieval.py x: 19 # changes y: 645 line churn evals/registry/data/imo_exact_answers/samples.jsonl x: 1 # changes y: 3 line churn pyproject.toml x: 20 # changes y: 155 line churn evals/registry/solvers/citation_prediction.yaml x: 2 # changes y: 286 line churn evals/data.py x: 9 # changes y: 126 line churn evals/completion_fns/solver_completion_fn.py x: 1 # changes y: 73 line churn evals/elsuite/bluff/strategy_solver.py x: 3 # changes y: 214 line churn evals/elsuite/function_deduction/prompts.py x: 2 # changes y: 47 line churn evals/elsuite/function_deduction/solvers.py x: 2 # changes y: 175 line churn evals/elsuite/function_deduction/solvers_test.py x: 2 # changes y: 167 line churn evals/elsuite/hr_ml_agent_bench/solvers/baseline.py x: 2 # changes y: 117 line churn evals/elsuite/make_me_say/eval.py x: 3 # changes y: 91 line churn evals/eval.py x: 9 # changes y: 206 line churn evals/registry.py x: 22 # changes y: 461 line churn evals/registry/solvers/already_said_that.yaml x: 2 # changes y: 91 line churn evals/registry/solvers/bluff.yaml x: 2 # changes y: 105 line churn evals/registry/solvers/cant_do_that_anymore.yaml x: 2 # changes y: 21 line churn evals/registry/solvers/defaults.yaml x: 3 # changes y: 942 line churn evals/registry/solvers/function_deduction.yaml x: 2 # changes y: 222 line churn evals/registry/solvers/skill_acquisition.yaml x: 2 # changes y: 329 line churn evals/registry/solvers/theory_of_mind.yaml x: 3 # changes y: 491 line churn evals/solvers/README.md x: 2 # changes y: 79 line churn evals/solvers/memory.py x: 1 # changes y: 64 line churn evals/solvers/nested/cot_solver.py x: 3 # changes y: 89 line churn evals/solvers/nested/self_consistency_solver.py x: 3 # changes y: 154 line churn evals/solvers/openai_assistants_solver.py x: 4 # changes y: 584 line churn evals/solvers/openai_assistants_solver_test.py x: 2 # changes y: 434 line churn evals/solvers/openai_solver.py x: 4 # changes y: 862 line churn evals/solvers/providers/openai/openai_assistants_solver.py x: 1 # changes y: 272 line churn evals/solvers/providers/openai/openai_assistants_solver_test.py x: 1 # changes y: 220 line churn evals/solvers/providers/openai/openai_solver.py x: 1 # changes y: 297 line churn evals/solvers/providers/together/together_solver.py x: 1 # changes y: 146 line churn evals/solvers/providers/together/together_solver_test.py x: 1 # changes y: 117 line churn evals/solvers/solver.py x: 4 # changes y: 314 line churn evals/solvers/together_solver.py x: 2 # changes y: 298 line churn evals/solvers/together_solver_test.py x: 2 # changes y: 234 line churn evals/solvers/utils.py x: 4 # changes y: 219 line churn evals/registry/solvers/gemini.yaml x: 1 # changes y: 23 line churn evals/solvers/providers/google/gemini_solver.py x: 1 # changes y: 211 line churn evals/completion_fns/openai.py x: 5 # changes y: 225 line churn evals/solvers/providers/anthropic/anthropic_solver.py x: 2 # changes y: 184 line churn evals/utils/api_utils.py x: 10 # changes y: 296 line churn docs/run-evals.md x: 4 # changes y: 18 line churn evals/registry/solvers/anthropic.yaml x: 1 # changes y: 125 line churn evals/elsuite/hr_ml_agent_bench/README.md x: 1 # changes y: 226 line churn evals/elsuite/hr_ml_agent_bench/auto_marking.py x: 1 # changes y: 79 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/ant/baselines/human.py x: 1 # changes y: 52 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/ant/scripts/grade.py x: 1 # changes y: 94 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/cartpole/baselines/human.py x: 1 # changes y: 43 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/cifar10/env/train.py x: 1 # changes y: 140 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/feedback/env/evaluation_details.txt x: 1 # changes y: 17 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/house_price/env/data_description.txt x: 1 # changes y: 523 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/ogbn_arxiv/env/train.py x: 1 # changes y: 165 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/parkinsons_disease/env/train.py x: 1 # changes y: 184 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/vectorization/env/train.py x: 1 # changes y: 200 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/vectorization/scripts/grade.py x: 1 # changes y: 101 line churn evals/elsuite/hr_ml_agent_bench/environment.py x: 1 # changes y: 383 line churn evals/elsuite/hr_ml_agent_bench/high_level_actions.py x: 1 # changes y: 260 line churn evals/elsuite/hr_ml_agent_bench/low_level_actions.py x: 1 # changes y: 370 line churn evals/elsuite/hr_ml_agent_bench/scripts/plot_experiments.py x: 1 # changes y: 442 line churn evals/elsuite/multistep_web_tasks/reproducibility/all_tasks.json x: 1 # changes y: 280 line churn evals/elsuite/multistep_web_tasks/webarena/browser_env/actions.py x: 1 # changes y: 1264 line churn evals/elsuite/multistep_web_tasks/webarena/browser_env/basic_browser_env.py x: 1 # changes y: 248 line churn evals/elsuite/multistep_web_tasks/webarena/browser_env/processors.py x: 1 # changes y: 626 line churn evals/elsuite/multistep_web_tasks/webarena/eval_run.py x: 1 # changes y: 356 line churn evals/elsuite/function_deduction/eval.py x: 1 # changes y: 302 line churn evals/elsuite/identifying_variables/README.md x: 1 # changes y: 177 line churn evals/elsuite/identifying_variables/renderers/corrset.py x: 1 # changes y: 346 line churn evals/elsuite/identifying_variables/scripts/gen_data.py x: 1 # changes y: 467 line churn evals/elsuite/identifying_variables/scripts/make_plots.py x: 1 # changes y: 400 line churn LICENSE.md x: 3 # changes y: 119 line churn evals/elsuite/already_said_that/scripts/make_plots.py x: 1 # changes y: 328 line churn evals/elsuite/skill_acquisition/eval.py x: 1 # changes y: 428 line churn evals/elsuite/skill_acquisition/scraping/human_rights.html x: 1 # changes y: 2839 line churn evals/elsuite/error_recovery/scripts/make_plots.py x: 1 # changes y: 597 line churn evals/elsuite/cant_do_that_anymore/chess/board.py x: 1 # changes y: 244 line churn evals/elsuite/cant_do_that_anymore/scripts/diagonal_dataset_creation.py x: 1 # changes y: 316 line churn evals/elsuite/bugged_tools/tools.py x: 1 # changes y: 653 line churn evals/elsuite/make_me_pay/readme.md x: 3 # changes y: 141 line churn evals/elsuite/make_me_pay/scripts/run_experiments_personality.sh x: 2 # changes y: 37 line churn evals/elsuite/make_me_pay/task_description.py x: 3 # changes y: 68 line churn evals/elsuite/make_me_say/core.py x: 3 # changes y: 328 line churn evals/elsuite/make_me_say/makemesay_test.py x: 4 # changes y: 940 line churn evals/elsuite/schelling_point/README.md x: 3 # changes y: 131 line churn evals/prompt/base.py x: 4 # changes y: 41 line churn evals/registry/evals/ballots.yaml x: 3 # changes y: 39 line churn evals/registry/evals/make-me-pay.yaml x: 3 # changes y: 457 line churn evals/solvers/human_cli_solver.py x: 4 # changes y: 72 line churn evals/solvers/prompts/hhh.py x: 4 # changes y: 302 line churn evals/elsuite/sandbagging/solvers.py x: 3 # changes y: 224 line churn evals/solvers/openai_chat_completion_cot_solver.py x: 4 # changes y: 262 line churn evals/record.py x: 13 # changes y: 270 line churn evals/cli/oaievalset.py x: 5 # changes y: 69 line churn evals/elsuite/make_me_say/defaults.py x: 2 # changes y: 208 line churn evals/elsuite/schelling_point/prompts.py x: 3 # changes y: 338 line churn evals/registry/data/word_association/corpus_tools/validators.py x: 4 # changes y: 384 line churn scripts/battle_generator.py x: 6 # changes y: 100 line churn scripts/modelgraded_generator.py x: 6 # changes y: 545 line churn scripts/pattern_identification_generator.py x: 6 # changes y: 153 line churn evals/elsuite/mmmu/eval.py x: 3 # changes y: 184 line churn evals/completion_fns/langchain_llm.py x: 3 # changes y: 109 line churn evals/registry/evals/mmmu.yaml x: 1 # changes y: 419 line churn evals/elsuite/make_me_say/utils.py x: 6 # changes y: 88 line churn evals/elsuite/ballots/eval.py x: 3 # changes y: 210 line churn docs/custom-eval.md x: 4 # changes y: 23 line churn evals/elsuite/make_me_pay/utils.py x: 3 # changes y: 175 line churn evals/elsuite/make_me_say/autoeval.py x: 4 # changes y: 155 line churn evals/elsuite/make_me_pay/makemepay.py x: 2 # changes y: 395 line churn evals/elsuite/make_me_pay/makemepay_test.py x: 2 # changes y: 368 line churn .gitignore x: 7 # changes y: 19 line churn docs/eval-templates.md x: 5 # changes y: 19 line churn .github/PULL_REQUEST_TEMPLATE.md x: 8 # changes y: 92 line churn evals/elsuite/modelgraded/classify.py x: 19 # changes y: 791 line churn evals/elsuite/utils.py x: 10 # changes y: 154 line churn evals/registry/data/german-part-of-speech/parsePosDe.py x: 2 # changes y: 336 line churn evals/registry/data/mazes/nxn_maze_eval_generator.py x: 2 # changes y: 437 line churn evals/registry/data/solve-for-variable/tools/problem.py x: 2 # changes y: 252 line churn evals/registry/data/solve-for-variable/tools/solve.py x: 2 # changes y: 731 line churn evals/registry/data/word_association/corpus_tools/related_words.py x: 2 # changes y: 199 line churn evals/base.py x: 5 # changes y: 89 line churn evals/elsuite/basic/match.py x: 5 # changes y: 34 line churn evals/registry/evals/test-modelgraded.yaml x: 7 # changes y: 104 line churn evals/elsuite/modelgraded/classify_utils.py x: 5 # changes y: 351 line churn evals/registry/data/GPT-model-text-detection/source.md x: 1 # changes y: 614 line churn evals/registry/data/backgammon/generate_samples.ipynb x: 1 # changes y: 1349 line churn evals/elsuite/basic/includes.py x: 7 # changes y: 69 line churn evals/elsuite/basic/json_validator.py x: 4 # changes y: 67 line churn evals/api.py x: 5 # changes y: 257 line churn evals/elsuite/basic/fuzzy_match.py x: 5 # changes y: 45 line churn docs/build-eval.md x: 12 # changes y: 80 line churn evals/registry/data/integer-sequence-predictions/misc-recent-sequences-info.txt x: 1 # changes y: 641 line churn evals/registry/evals/mmlu.yaml x: 1 # changes y: 455 line churn evals/elsuite/modelgraded/base.py x: 4 # changes y: 349 line churn evals/registry/eval_sets/test-all.yaml x: 5 # changes y: 11 line churn MANIFEST.in x: 3 # changes y: 5 line churn evals/registry/data/logiqa/logiqa.jsonl x: 1 # changes y: 2711 line churn evals/registry/data/README.md x: 3 # changes y: 21 line churn evals/registry/data/pattern_identification/samples.v0.jsonl x: 2 # changes y: 506 line churn
2839.0
line churn
  min: 1.0
  average: 51.03
  25th percentile: 3.0
  median: 8.0
  75th percentile: 38.5
  max: 2839.0
0 22.0
# changes
min: 1.0 | average: 1.31 | 25th percentile: 1.0 | median: 1.0 | 75th percentile: 1.0 | max: 22.0

Number of Contributors vs. Line Churn: 1481 points

.pre-commit-config.yaml x: 3 # contributors y: 25 line churn evals/cli/oaieval.py x: 18 # contributors y: 645 line churn evals/registry/data/imo_exact_answers/samples.jsonl x: 1 # contributors y: 3 line churn evals/data.py x: 10 # contributors y: 126 line churn evals/completion_fns/solver_completion_fn.py x: 1 # contributors y: 73 line churn evals/elsuite/bluff/strategy_solver.py x: 3 # contributors y: 214 line churn evals/elsuite/bugged_tools/task_description.py x: 1 # contributors y: 60 line churn evals/elsuite/function_deduction/prompts.py x: 2 # contributors y: 47 line churn evals/elsuite/function_deduction/solvers.py x: 2 # contributors y: 175 line churn evals/elsuite/hr_ml_agent_bench/solvers/baseline.py x: 2 # contributors y: 117 line churn evals/elsuite/make_me_say/eval.py x: 2 # contributors y: 91 line churn evals/eval.py x: 9 # contributors y: 206 line churn evals/registry.py x: 17 # contributors y: 461 line churn evals/registry/solvers/bluff.yaml x: 2 # contributors y: 105 line churn evals/registry/solvers/cant_do_that_anymore.yaml x: 1 # contributors y: 21 line churn evals/registry/solvers/defaults.yaml x: 2 # contributors y: 942 line churn evals/registry/solvers/error_recovery.yaml x: 1 # contributors y: 46 line churn evals/registry/solvers/function_deduction.yaml x: 2 # contributors y: 222 line churn evals/registry/solvers/make-me-pay.yaml x: 2 # contributors y: 125 line churn evals/registry/solvers/skill_acquisition.yaml x: 2 # contributors y: 329 line churn evals/registry/solvers/theory_of_mind.yaml x: 2 # contributors y: 491 line churn evals/solvers/nested/self_consistency_solver.py x: 2 # contributors y: 154 line churn evals/solvers/prompts/cot.py x: 2 # contributors y: 8 line churn evals/solvers/providers/openai/openai_assistants_solver.py x: 1 # contributors y: 272 line churn evals/solvers/providers/openai/openai_solver.py x: 1 # contributors y: 297 line churn evals/solvers/providers/together/together_solver.py x: 1 # contributors y: 146 line churn evals/solvers/solver.py x: 2 # contributors y: 314 line churn evals/registry/solvers/gemini.yaml x: 1 # contributors y: 23 line churn evals/solvers/providers/google/gemini_solver.py x: 1 # contributors y: 211 line churn evals/completion_fns/openai.py x: 6 # contributors y: 225 line churn evals/solvers/providers/anthropic/anthropic_solver.py x: 2 # contributors y: 184 line churn evals/utils/api_utils.py x: 8 # contributors y: 296 line churn evals/registry/solvers/anthropic.yaml x: 1 # contributors y: 125 line churn evals/elsuite/hr_ml_agent_bench/auto_marking.py x: 1 # contributors y: 79 line churn evals/elsuite/hr_ml_agent_bench/autoeval.py x: 1 # contributors y: 214 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/ant/scripts/grade.py x: 1 # contributors y: 94 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/ogbn_arxiv/env/train.py x: 1 # contributors y: 165 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/ogbn_arxiv/scripts/grade.py x: 1 # contributors y: 97 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/parkinsons_disease/env/train.py x: 1 # contributors y: 184 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/parkinsons_disease/scripts/prepare.py x: 1 # contributors y: 135 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/vectorization/env/train.py x: 1 # contributors y: 200 line churn evals/elsuite/hr_ml_agent_bench/environment.py x: 1 # contributors y: 383 line churn evals/elsuite/hr_ml_agent_bench/high_level_actions.py x: 1 # contributors y: 260 line churn evals/elsuite/hr_ml_agent_bench/low_level_actions.py x: 1 # contributors y: 370 line churn evals/elsuite/hr_ml_agent_bench/scripts/plot_experiments.py x: 1 # contributors y: 442 line churn evals/elsuite/hr_ml_agent_bench/utils.py x: 1 # contributors y: 180 line churn evals/registry/eval_sets/hr-ml-agent-bench.yaml x: 1 # contributors y: 35 line churn evals/registry/evals/hr-ml-agent-bench.yaml x: 1 # contributors y: 152 line churn evals/elsuite/multistep_web_tasks/docker/homepage/templates/calculator.html x: 1 # contributors y: 109 line churn evals/elsuite/multistep_web_tasks/session.py x: 1 # contributors y: 518 line churn evals/elsuite/multistep_web_tasks/solvers/strong_solver/strong_solver.py x: 1 # contributors y: 226 line churn evals/elsuite/multistep_web_tasks/webarena/browser_env/actions.py x: 1 # contributors y: 1264 line churn evals/elsuite/multistep_web_tasks/webarena/browser_env/basic_browser_env.py x: 1 # contributors y: 248 line churn evals/elsuite/multistep_web_tasks/webarena/browser_env/processors.py x: 1 # contributors y: 626 line churn evals/elsuite/multistep_web_tasks/webarena/eval_run.py x: 1 # contributors y: 356 line churn evals/elsuite/function_deduction/eval.py x: 1 # contributors y: 302 line churn evals/elsuite/identifying_variables/renderers/corrset.py x: 1 # contributors y: 346 line churn evals/elsuite/identifying_variables/scripts/gen_data.py x: 1 # contributors y: 467 line churn evals/elsuite/identifying_variables/scripts/make_plots.py x: 1 # contributors y: 400 line churn evals/elsuite/already_said_that/scripts/make_plots.py x: 1 # contributors y: 328 line churn evals/utils/log_utils.py x: 2 # contributors y: 82 line churn evals/elsuite/skill_acquisition/eval.py x: 1 # contributors y: 428 line churn evals/elsuite/error_recovery/eval.py x: 1 # contributors y: 284 line churn evals/elsuite/error_recovery/scripts/make_plots.py x: 1 # contributors y: 597 line churn evals/elsuite/cant_do_that_anymore/chess/board.py x: 1 # contributors y: 244 line churn evals/elsuite/cant_do_that_anymore/scripts/dataset_creation.py x: 1 # contributors y: 312 line churn evals/elsuite/bugged_tools/tools.py x: 1 # contributors y: 653 line churn evals/elsuite/make_me_pay/task_description.py x: 2 # contributors y: 68 line churn evals/prompt/base.py x: 3 # contributors y: 41 line churn evals/registry/evals/make-me-pay.yaml x: 2 # contributors y: 457 line churn evals/registry/evals/schelling_point.yaml x: 3 # contributors y: 71 line churn evals/registry/evals/self_prompting.yaml x: 2 # contributors y: 25 line churn evals/solvers/prompts/hhh.py x: 3 # contributors y: 302 line churn evals/record.py x: 9 # contributors y: 270 line churn evals/cli/oaievalset.py x: 4 # contributors y: 69 line churn evals/completion_fns/retrieval.py x: 3 # contributors y: 136 line churn evals/elsuite/make_me_pay/solvers/prompts.py x: 2 # contributors y: 21 line churn evals/elsuite/sandbagging/defaults.py x: 2 # contributors y: 57 line churn evals/elsuite/schelling_point/prompts.py x: 2 # contributors y: 338 line churn evals/registry/data/word_association/corpus_tools/validators.py x: 4 # contributors y: 384 line churn scripts/battle_generator.py x: 5 # contributors y: 100 line churn scripts/modelgraded_generator.py x: 5 # contributors y: 545 line churn scripts/pattern_identification_generator.py x: 6 # contributors y: 153 line churn evals/completion_fns/langchain_llm.py x: 4 # contributors y: 109 line churn evals/registry/evals/mmmu.yaml x: 1 # contributors y: 419 line churn evals/elsuite/make_me_say/utils.py x: 5 # contributors y: 88 line churn evals/elsuite/ballots/eval.py x: 2 # contributors y: 210 line churn evals/completion_fns/langchain_math.py x: 4 # contributors y: 57 line churn evals/elsuite/make_me_pay/utils.py x: 3 # contributors y: 175 line churn evals/elsuite/make_me_say/autoeval.py x: 4 # contributors y: 155 line churn evals/elsuite/make_me_pay/makemepay.py x: 2 # contributors y: 395 line churn evals/elsuite/modelgraded/classify.py x: 7 # contributors y: 791 line churn evals/registry/data/mazes/nxn_maze_eval_generator.py x: 2 # contributors y: 437 line churn evals/registry/data/simple_physics_engine/wave_function_collapse.py x: 2 # contributors y: 280 line churn evals/registry/data/solve-for-variable/tools/problem.py x: 2 # contributors y: 252 line churn evals/registry/data/solve-for-variable/tools/solve.py x: 2 # contributors y: 731 line churn evals/registry/data/word_association/corpus_tools/related_words.py x: 2 # contributors y: 199 line churn evals/base.py x: 4 # contributors y: 89 line churn evals/elsuite/basic/match.py x: 4 # contributors y: 34 line churn evals/registry/evals/test-modelgraded.yaml x: 3 # contributors y: 104 line churn evals/elsuite/modelgraded/classify_utils.py x: 2 # contributors y: 351 line churn evals/registry/data/backgammon/generate_samples.ipynb x: 1 # contributors y: 1349 line churn evals/api.py x: 4 # contributors y: 257 line churn evals/elsuite/basic/fuzzy_match.py x: 4 # contributors y: 45 line churn evals/registry/evals/mmlu.yaml x: 1 # contributors y: 455 line churn evals/registry/eval_sets/test-all.yaml x: 3 # contributors y: 11 line churn evals/registry/evals/coqa-ex.yaml x: 3 # contributors y: 75 line churn MANIFEST.in x: 3 # contributors y: 5 line churn evals/registry/data/logiqa/logiqa.jsonl x: 2 # contributors y: 2711 line churn evals/registry/evals/test-modelgraded-generated.yaml x: 2 # contributors y: 38 line churn evals/registry/data/pattern_identification/samples.v0.jsonl x: 2 # contributors y: 506 line churn
2711.0
line churn
  min: 1.0
  average: 44.06
  25th percentile: 3.0
  median: 8.0
  75th percentile: 25.0
  max: 2711.0
0 18.0
# contributors
min: 1.0 | average: 1.18 | 25th percentile: 1.0 | median: 1.0 | 75th percentile: 1.0 | max: 18.0

File Size vs. Line Churn: 1481 points

.pre-commit-config.yaml x: 32 lines of code y: 25 line churn evals/cli/oaieval.py x: 253 lines of code y: 645 line churn evals/registry/data/imo_exact_answers/samples.jsonl x: 3 lines of code y: 3 line churn evals/registry/evals/imo_exact_answers.yaml x: 8 lines of code y: 9 line churn evals/data.py x: 148 lines of code y: 126 line churn evals/completion_fns/solver_completion_fn.py x: 47 lines of code y: 73 line churn evals/elsuite/bluff/strategy_solver.py x: 88 lines of code y: 214 line churn evals/elsuite/bugged_tools/task_description.py x: 9 lines of code y: 60 line churn evals/elsuite/function_deduction/prompts.py x: 6 lines of code y: 47 line churn evals/elsuite/function_deduction/solvers.py x: 140 lines of code y: 175 line churn evals/elsuite/hr_ml_agent_bench/solvers/baseline.py x: 90 lines of code y: 117 line churn evals/elsuite/make_me_say/eval.py x: 48 lines of code y: 91 line churn evals/eval.py x: 170 lines of code y: 206 line churn evals/registry.py x: 242 lines of code y: 461 line churn evals/registry/solvers/already_said_that.yaml x: 75 lines of code y: 91 line churn evals/registry/solvers/bluff.yaml x: 80 lines of code y: 105 line churn evals/registry/solvers/cant_do_that_anymore.yaml x: 16 lines of code y: 21 line churn evals/registry/solvers/defaults.yaml x: 294 lines of code y: 942 line churn evals/registry/solvers/error_recovery.yaml x: 33 lines of code y: 46 line churn evals/registry/solvers/function_deduction.yaml x: 174 lines of code y: 222 line churn evals/registry/solvers/hr-ml-agent-bench.yaml x: 37 lines of code y: 46 line churn evals/registry/solvers/make-me-pay.yaml x: 101 lines of code y: 125 line churn evals/registry/solvers/self_prompting.yaml x: 96 lines of code y: 124 line churn evals/registry/solvers/skill_acquisition.yaml x: 267 lines of code y: 329 line churn evals/registry/solvers/theory_of_mind.yaml x: 394 lines of code y: 491 line churn evals/registry/solvers/together.yaml x: 85 lines of code y: 110 line churn evals/solvers/memory.py x: 50 lines of code y: 64 line churn evals/solvers/nested/cot_solver.py x: 61 lines of code y: 89 line churn evals/solvers/nested/self_consistency_solver.py x: 118 lines of code y: 154 line churn evals/solvers/providers/openai/openai_assistants_solver.py x: 186 lines of code y: 272 line churn evals/solvers/providers/openai/openai_solver.py x: 181 lines of code y: 297 line churn evals/solvers/providers/together/together_solver.py x: 68 lines of code y: 146 line churn evals/solvers/solver.py x: 125 lines of code y: 314 line churn evals/solvers/utils.py x: 37 lines of code y: 219 line churn evals/registry/solvers/gemini.yaml x: 15 lines of code y: 23 line churn evals/solvers/providers/google/gemini_solver.py x: 157 lines of code y: 211 line churn evals/completion_fns/openai.py x: 147 lines of code y: 225 line churn evals/solvers/providers/anthropic/anthropic_solver.py x: 89 lines of code y: 184 line churn evals/utils/api_utils.py x: 15 lines of code y: 296 line churn evals/registry/solvers/anthropic.yaml x: 90 lines of code y: 125 line churn evals/elsuite/hr_ml_agent_bench/actions.py x: 37 lines of code y: 60 line churn evals/elsuite/hr_ml_agent_bench/auto_marking.py x: 53 lines of code y: 79 line churn evals/elsuite/hr_ml_agent_bench/autoeval.py x: 172 lines of code y: 214 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/ant/baselines/human.py x: 36 lines of code y: 52 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/ant/baselines/naive.py x: 28 lines of code y: 48 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/ant/scripts/grade.py x: 43 lines of code y: 94 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/bipedal_walker/baselines/human.py x: 57 lines of code y: 80 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/bipedal_walker/scripts/grade.py x: 31 lines of code y: 76 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/cartpole/baselines/human.py x: 30 lines of code y: 43 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/cartpole/scripts/grade.py x: 40 lines of code y: 81 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/cifar10/env/train.py x: 105 lines of code y: 140 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/cifar10/scripts/grade.py x: 35 lines of code y: 70 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/cifar10/scripts/prepare.py x: 5 lines of code y: 8 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/feedback/env/train.py x: 66 lines of code y: 95 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/feedback/scripts/grade.py x: 36 lines of code y: 77 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/feedback/scripts/prepare.py x: 24 lines of code y: 31 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/house_price/env/train.py x: 41 lines of code y: 64 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/house_price/scripts/grade.py x: 33 lines of code y: 78 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/house_price/scripts/prepare.py x: 20 lines of code y: 28 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/humanoid/scripts/grade.py x: 45 lines of code y: 94 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/imdb/env/train.py x: 21 lines of code y: 43 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/imdb/scripts/grade.py x: 27 lines of code y: 62 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/inverted_pendulum/scripts/grade.py x: 28 lines of code y: 66 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/ogbn_arxiv/env/train.py x: 126 lines of code y: 165 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/ogbn_arxiv/scripts/grade.py x: 53 lines of code y: 97 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/parkinsons_disease/env/train.py x: 124 lines of code y: 184 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/parkinsons_disease/scripts/grade.py x: 22 lines of code y: 58 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/parkinsons_disease/scripts/prepare.py x: 98 lines of code y: 135 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/pong/baselines/human.py x: 43 lines of code y: 59 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/spaceship_titanic/env/train.py x: 34 lines of code y: 64 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/spaceship_titanic/scripts/grade.py x: 26 lines of code y: 64 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/vectorization/env/train.py x: 118 lines of code y: 200 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/vectorization/scripts/grade.py x: 64 lines of code y: 101 line churn evals/elsuite/hr_ml_agent_bench/benchmarks/vectorization/scripts/human_baseline.py x: 83 lines of code y: 162 line churn evals/elsuite/hr_ml_agent_bench/environment.py x: 283 lines of code y: 383 line churn evals/elsuite/hr_ml_agent_bench/eval.py x: 91 lines of code y: 120 line churn evals/elsuite/hr_ml_agent_bench/high_level_actions.py x: 191 lines of code y: 260 line churn evals/elsuite/hr_ml_agent_bench/low_level_actions.py x: 304 lines of code y: 370 line churn evals/elsuite/hr_ml_agent_bench/prepare_task.py x: 40 lines of code y: 65 line churn evals/elsuite/hr_ml_agent_bench/prompts.py x: 26 lines of code y: 69 line churn evals/elsuite/hr_ml_agent_bench/schema.py x: 46 lines of code y: 65 line churn evals/elsuite/hr_ml_agent_bench/scripts/plot_experiments.py x: 307 lines of code y: 442 line churn evals/elsuite/hr_ml_agent_bench/scripts/run_experiments.py x: 60 lines of code y: 88 line churn evals/elsuite/hr_ml_agent_bench/utils.py x: 112 lines of code y: 180 line churn evals/registry/eval_sets/hr-ml-agent-bench.yaml x: 34 lines of code y: 35 line churn evals/registry/evals/hr-ml-agent-bench.yaml x: 137 lines of code y: 152 line churn evals/elsuite/multistep_web_tasks/constants.py x: 70 lines of code y: 75 line churn evals/elsuite/multistep_web_tasks/docker/flask-playwright/app.py x: 165 lines of code y: 218 line churn evals/elsuite/multistep_web_tasks/docker/homepage/templates/calculator.html x: 106 lines of code y: 109 line churn evals/elsuite/multistep_web_tasks/docker/homepage/templates/index.html x: 108 lines of code y: 162 line churn evals/elsuite/multistep_web_tasks/docker/homepage/templates/scratchpad.html x: 105 lines of code y: 122 line churn evals/elsuite/multistep_web_tasks/reproducibility/make_plots.py x: 94 lines of code y: 142 line churn evals/elsuite/multistep_web_tasks/reproducibility/run_environments.py x: 29 lines of code y: 36 line churn evals/elsuite/multistep_web_tasks/session.py x: 416 lines of code y: 518 line churn evals/elsuite/multistep_web_tasks/solvers/strong_solver/strong_prompts.py x: 4 lines of code y: 83 line churn evals/elsuite/multistep_web_tasks/solvers/strong_solver/strong_solver.py x: 173 lines of code y: 226 line churn evals/elsuite/multistep_web_tasks/solvers/webarena_solvers/webarena_prompts.py x: 12 lines of code y: 151 line churn evals/elsuite/multistep_web_tasks/solvers/webarena_solvers/webarena_solvers.py x: 94 lines of code y: 123 line churn evals/elsuite/multistep_web_tasks/webarena/bash_browser_env/bash_browser_utils.py x: 11 lines of code y: 14 line churn evals/elsuite/multistep_web_tasks/webarena/bash_env/basic_bash_env.py x: 163 lines of code y: 203 line churn evals/elsuite/multistep_web_tasks/webarena/browser_env/actions.py x: 1014 lines of code y: 1264 line churn evals/elsuite/multistep_web_tasks/webarena/browser_env/basic_browser_env.py x: 191 lines of code y: 248 line churn evals/elsuite/multistep_web_tasks/webarena/browser_env/browser_utils.py x: 78 lines of code y: 113 line churn evals/elsuite/multistep_web_tasks/webarena/browser_env/constants.py x: 282 lines of code y: 295 line churn evals/elsuite/multistep_web_tasks/webarena/browser_env/helper_functions.py x: 129 lines of code y: 181 line churn evals/elsuite/multistep_web_tasks/webarena/browser_env/processors.py x: 495 lines of code y: 626 line churn evals/elsuite/multistep_web_tasks/webarena/core/env.py x: 75 lines of code y: 129 line churn evals/elsuite/multistep_web_tasks/webarena/core/playwright_api.py x: 279 lines of code y: 378 line churn evals/elsuite/multistep_web_tasks/webarena/core/utils.py x: 188 lines of code y: 223 line churn evals/elsuite/multistep_web_tasks/webarena/eval_run.py x: 277 lines of code y: 356 line churn evals/elsuite/multistep_web_tasks/webarena/evaluation_harness/evaluators.py x: 273 lines of code y: 356 line churn evals/elsuite/multistep_web_tasks/webarena/evaluation_harness/helper_functions.py x: 110 lines of code y: 159 line churn evals/elsuite/multistep_web_tasks/webarena/task_description.py x: 2 lines of code y: 43 line churn evals/registry/evals/multistep_web_tasks.yaml x: 60 lines of code y: 80 line churn evals/registry/solvers/multistep_web_tasks.yaml x: 56 lines of code y: 63 line churn evals/elsuite/function_deduction/baselines.py x: 91 lines of code y: 133 line churn evals/elsuite/function_deduction/eval.py x: 244 lines of code y: 302 line churn evals/elsuite/function_deduction/scripts/dataset/create_dataset.py x: 44 lines of code y: 62 line churn evals/elsuite/function_deduction/scripts/make_plots.py x: 195 lines of code y: 256 line churn evals/registry/evals/function-deduction.yaml x: 4 lines of code y: 37 line churn evals/elsuite/identifying_variables/eval.py x: 227 lines of code y: 292 line churn evals/elsuite/identifying_variables/graph_utils.py x: 130 lines of code y: 254 line churn evals/elsuite/identifying_variables/metrics.py x: 54 lines of code y: 105 line churn evals/elsuite/identifying_variables/prompts.py x: 8 lines of code y: 103 line churn evals/elsuite/identifying_variables/renderers/corrset.py x: 216 lines of code y: 346 line churn evals/elsuite/identifying_variables/renderers/tabular.py x: 125 lines of code y: 200 line churn evals/elsuite/identifying_variables/scripts/gen_data.py x: 319 lines of code y: 467 line churn evals/elsuite/identifying_variables/scripts/make_plots.py x: 325 lines of code y: 400 line churn evals/elsuite/identifying_variables/scripts/plotting_utils.py x: 128 lines of code y: 163 line churn evals/elsuite/identifying_variables/structs.py x: 18 lines of code y: 49 line churn evals/elsuite/identifying_variables/utils.py x: 72 lines of code y: 91 line churn evals/registry/evals/identifying_variables.yaml x: 123 lines of code y: 136 line churn evals/elsuite/track_the_stat/eval.py x: 80 lines of code y: 96 line churn evals/elsuite/track_the_stat/prompts/__init__.py x: 11 lines of code y: 27 line churn evals/elsuite/track_the_stat/prompts/median.py x: 2 lines of code y: 33 line churn evals/elsuite/track_the_stat/prompts/mode.py x: 2 lines of code y: 29 line churn evals/elsuite/track_the_stat/scripts/make_plots.py x: 235 lines of code y: 296 line churn evals/elsuite/track_the_stat/solvers.py x: 72 lines of code y: 98 line churn evals/elsuite/track_the_stat/utils.py x: 47 lines of code y: 78 line churn evals/elsuite/already_said_that/distractors.py x: 81 lines of code y: 133 line churn evals/elsuite/already_said_that/eval.py x: 130 lines of code y: 160 line churn evals/elsuite/already_said_that/scripts/gen_data.py x: 50 lines of code y: 73 line churn evals/elsuite/already_said_that/scripts/make_plots.py x: 263 lines of code y: 328 line churn evals/elsuite/already_said_that/utils.py x: 104 lines of code y: 171 line churn evals/registry/evals/already_said_that.yaml x: 45 lines of code y: 50 line churn evals/elsuite/twenty_questions/scripts/make_plots.py x: 111 lines of code y: 142 line churn evals/elsuite/skill_acquisition/eval.py x: 313 lines of code y: 428 line churn evals/elsuite/skill_acquisition/scraping/scrape_distractor_articles.py x: 76 lines of code y: 96 line churn evals/elsuite/skill_acquisition/scraping/scrape_miskito.py x: 106 lines of code y: 135 line churn evals/elsuite/skill_acquisition/solvers.py x: 13 lines of code y: 22 line churn evals/elsuite/skill_acquisition/task_description.py x: 1 lines of code y: 3 line churn evals/elsuite/skill_acquisition/utils.py x: 115 lines of code y: 179 line churn evals/registry/evals/skill_acquisition.yaml x: 94 lines of code y: 107 line churn evals/elsuite/error_recovery/eval.py x: 204 lines of code y: 284 line churn evals/elsuite/error_recovery/scripts/dataset_creation.py x: 106 lines of code y: 156 line churn evals/elsuite/error_recovery/scripts/make_plots.py x: 446 lines of code y: 597 line churn evals/registry/evals/error_recovery.yaml x: 31 lines of code y: 36 line churn evals/elsuite/cant_do_that_anymore/chess/board.py x: 162 lines of code y: 244 line churn evals/elsuite/cant_do_that_anymore/chess/move_variants.py x: 113 lines of code y: 120 line churn evals/elsuite/cant_do_that_anymore/chess/notation.py x: 61 lines of code y: 106 line churn evals/elsuite/cant_do_that_anymore/chess/pieces.py x: 203 lines of code y: 263 line churn evals/elsuite/cant_do_that_anymore/chess/utils.py x: 75 lines of code y: 107 line churn evals/elsuite/cant_do_that_anymore/defaults.py x: 2 lines of code y: 15 line churn evals/elsuite/cant_do_that_anymore/eval.py x: 170 lines of code y: 201 line churn evals/elsuite/cant_do_that_anymore/scripts/dataset_creation.py x: 235 lines of code y: 312 line churn evals/elsuite/cant_do_that_anymore/scripts/diagonal_dataset_creation.py x: 216 lines of code y: 316 line churn evals/elsuite/cant_do_that_anymore/utils.py x: 178 lines of code y: 250 line churn evals/elsuite/bugged_tools/bugged_tools.py x: 132 lines of code y: 186 line churn evals/elsuite/bugged_tools/eval.py x: 210 lines of code y: 285 line churn evals/elsuite/bugged_tools/tools.py x: 497 lines of code y: 653 line churn evals/elsuite/solver_tools_convo.py x: 181 lines of code y: 240 line churn evals/registry/evals/bugged_tools.yaml x: 28 lines of code y: 31 line churn evals/elsuite/ballots/scripts/make_plots.py x: 233 lines of code y: 299 line churn evals/elsuite/basic/match_with_solvers.py x: 65 lines of code y: 82 line churn evals/elsuite/bluff/solver_player.py x: 82 lines of code y: 105 line churn evals/elsuite/make_me_pay/eval.py x: 126 lines of code y: 221 line churn evals/elsuite/make_me_pay/scripts/make_plots.py x: 102 lines of code y: 227 line churn evals/elsuite/make_me_pay/task_description.py x: 57 lines of code y: 68 line churn evals/elsuite/make_me_say/core.py x: 223 lines of code y: 328 line churn evals/elsuite/text_compression/scripts/dataset/dataset.py x: 63 lines of code y: 69 line churn evals/elsuite/text_compression/scripts/make_plots.py x: 63 lines of code y: 78 line churn evals/prompt/base.py x: 64 lines of code y: 41 line churn evals/registry/evals/make-me-pay.yaml x: 59 lines of code y: 457 line churn evals/registry/evals/schelling_point.yaml x: 66 lines of code y: 71 line churn evals/registry/evals/self_prompting.yaml x: 19 lines of code y: 25 line churn evals/registry/evals/steganography.yaml x: 3 lines of code y: 52 line churn evals/registry/evals/theory_of_mind.yaml x: 48 lines of code y: 56 line churn evals/solvers/nested/fewshot_solver.py x: 91 lines of code y: 112 line churn evals/solvers/postprocessors/base.py x: 6 lines of code y: 17 line churn evals/solvers/prompts/hhh.py x: 99 lines of code y: 302 line churn evals/elsuite/make_me_pay/solvers/lm_con_artist_solver.py x: 64 lines of code y: 87 line churn evals/elsuite/sandbagging/solvers.py x: 152 lines of code y: 224 line churn evals/elsuite/self_prompting/solvers/baselines.py x: 45 lines of code y: 76 line churn evals/registry/solvers/sandbagging.yaml x: 109 lines of code y: 120 line churn evals/record.py x: 450 lines of code y: 270 line churn evals/cli/oaievalset.py x: 111 lines of code y: 69 line churn evals/completion_fns/retrieval.py x: 68 lines of code y: 136 line churn evals/elsuite/ballots/prompts.py x: 44 lines of code y: 152 line churn evals/elsuite/make_me_pay/solvers/prompts.py x: 18 lines of code y: 21 line churn evals/elsuite/make_me_say/defaults.py x: 34 lines of code y: 208 line churn evals/elsuite/sandbagging/defaults.py x: 12 lines of code y: 57 line churn evals/elsuite/sandbagging/sandbagging_eval.py x: 68 lines of code y: 99 line churn evals/elsuite/schelling_point/prompts.py x: 25 lines of code y: 338 line churn evals/elsuite/schelling_point/utils.py x: 61 lines of code y: 108 line churn evals/registry/data/word_association/corpus_tools/validators.py x: 151 lines of code y: 384 line churn scripts/battle_generator.py x: 49 lines of code y: 100 line churn scripts/modelgraded_generator.py x: 185 lines of code y: 545 line churn scripts/pattern_identification_generator.py x: 48 lines of code y: 153 line churn evals/elsuite/mmmu/eval.py x: 159 lines of code y: 184 line churn evals/completion_fns/langchain_llm.py x: 70 lines of code y: 109 line churn evals/registry/evals/mmmu.yaml x: 390 lines of code y: 419 line churn evals/elsuite/make_me_say/utils.py x: 34 lines of code y: 88 line churn evals/elsuite/schelling_point/eval.py x: 73 lines of code y: 334 line churn evals/elsuite/bluff/eval.py x: 164 lines of code y: 208 line churn evals/completion_fns/langchain_math.py x: 21 lines of code y: 57 line churn evals/elsuite/make_me_pay/utils.py x: 47 lines of code y: 175 line churn evals/elsuite/make_me_say/autoeval.py x: 116 lines of code y: 155 line churn evals/elsuite/bluff/bluff/cards.py x: 206 lines of code y: 288 line churn evals/elsuite/bluff/bluff/players.py x: 107 lines of code y: 170 line churn evals/elsuite/bluff/scripts/make_plots.py x: 103 lines of code y: 141 line churn evals/registry/evals/bluff.yaml x: 39 lines of code y: 49 line churn evals/elsuite/sandbagging/mmlu_eval.py x: 60 lines of code y: 73 line churn evals/elsuite/sandbagging/scripts/consistency_plots.py x: 88 lines of code y: 142 line churn evals/elsuite/sandbagging/scripts/utils.py x: 100 lines of code y: 143 line churn evals/elsuite/make_me_pay/makemepay.py x: 222 lines of code y: 395 line churn evals/elsuite/theory_of_mind/scripts/data_generation.py x: 66 lines of code y: 82 line churn evals/elsuite/theory_of_mind/scripts/make_plots.py x: 90 lines of code y: 133 line churn evals/elsuite/self_prompting/eval.py x: 210 lines of code y: 261 line churn evals/elsuite/self_prompting/scripts/dataset/eval_list.py x: 52 lines of code y: 52 line churn evals/task_state.py x: 13 lines of code y: 46 line churn evals/elsuite/basic/json_match.py x: 77 lines of code y: 106 line churn evals/elsuite/modelgraded/classify.py x: 97 lines of code y: 791 line churn evals/elsuite/multiple_choice.py x: 84 lines of code y: 106 line churn evals/elsuite/utils.py x: 150 lines of code y: 154 line churn evals/metrics.py x: 52 lines of code y: 29 line churn evals/registry/data/canto_wu_pronunciation/csv_to_json.py x: 55 lines of code y: 96 line churn evals/registry/data/german-part-of-speech/parsePosDe.py x: 172 lines of code y: 336 line churn evals/registry/data/mapping_to_matricies/data_generator.py x: 30 lines of code y: 68 line churn evals/registry/data/mazes/nxn_maze_eval_generator.py x: 162 lines of code y: 437 line churn evals/registry/data/poker_analysis/poker_analysis_sample_generator.py x: 95 lines of code y: 209 line churn evals/registry/data/simple_physics_engine/wave_function_collapse.py x: 157 lines of code y: 280 line churn evals/registry/data/solve-for-variable/tools/main.py x: 65 lines of code y: 127 line churn evals/registry/data/solve-for-variable/tools/problem.py x: 90 lines of code y: 252 line churn evals/registry/data/solve-for-variable/tools/solve.py x: 231 lines of code y: 731 line churn evals/registry/data/solve-for-variable/tools/tester.py x: 94 lines of code y: 248 line churn evals/registry/data/word_association/corpus_tools/corpus.py x: 58 lines of code y: 113 line churn evals/registry/data/word_association/corpus_tools/logger_config.py x: 8 lines of code y: 17 line churn evals/registry/data/word_association/corpus_tools/processor.py x: 36 lines of code y: 120 line churn evals/registry/data/word_association/corpus_tools/related_words.py x: 64 lines of code y: 199 line churn evals/registry/data/word_association/corpus_tools/sample_generators.py x: 154 lines of code y: 251 line churn evals/base.py x: 51 lines of code y: 89 line churn evals/elsuite/basic/match.py x: 57 lines of code y: 34 line churn evals/elsuite/translate.py x: 66 lines of code y: 16 line churn evals/elsuite/text_compression/eval.py x: 52 lines of code y: 59 line churn evals/elsuite/steganography/prompts.py x: 55 lines of code y: 329 line churn evals/elsuite/ballots/utils.py x: 118 lines of code y: 142 line churn evals/registry/evals/make-me-say.yaml x: 25 lines of code y: 30 line churn evals/elsuite/modelgraded/classify_utils.py x: 145 lines of code y: 351 line churn evals/registry/evals/exams.yaml x: 112 lines of code y: 127 line churn evals/registry/evals/css-selectors.yaml x: 3 lines of code y: 20 line churn evals/registry/evals/logic-container.yaml x: 10 lines of code y: 10 line churn evals/registry/modelgraded/keywords.yaml x: 20 lines of code y: 20 line churn evals/registry/evals/coq-editing.yaml x: 3 lines of code y: 23 line churn evals/registry/data/backgammon/generate_samples.ipynb x: 1349 lines of code y: 1349 line churn evals/registry/modelgraded/onomatopoeia.yaml x: 30 lines of code y: 30 line churn evals/registry/data/nfl-point-combinations/combinations_generator.py x: 25 lines of code y: 40 line churn evals/api.py x: 61 lines of code y: 257 line churn evals/registry/evals/raven-matrices.yaml x: 224 lines of code y: 313 line churn evals/elsuite/basic/fuzzy_match.py x: 49 lines of code y: 45 line churn evals/registry/data/points_on_line/eval_generator.py x: 73 lines of code y: 129 line churn evals/registry/modelgraded/iambic_pentameter.yaml x: 14 lines of code y: 15 line churn evals/utils/misc.py x: 19 lines of code y: 2 line churn evals/registry/evals/ukraine-gec.yaml x: 180 lines of code y: 198 line churn evals/registry/evals/test-basic.yaml x: 32 lines of code y: 14 line churn evals/registry/evals/mmlu.yaml x: 399 lines of code y: 455 line churn evals/elsuite/modelgraded/base.py x: 16 lines of code y: 349 line churn evals/registry/modelgraded/diversity.yaml x: 13 lines of code y: 4 line churn evals/registry/evals/coqa-ex.yaml x: 55 lines of code y: 75 line churn evals/registry/evals/test-modelgraded-battle.yaml x: 36 lines of code y: 20 line churn evals/registry/modelgraded/closedqa.yaml x: 21 lines of code y: 6 line churn evals/utils/snowflake.py x: 100 lines of code y: 10 line churn evals/registry/data/logiqa/logiqa.jsonl x: 3 lines of code y: 2711 line churn evals/registry/evals/naughty_strings.yaml x: 36 lines of code y: 39 line churn evals/registry/modelgraded/humor.yaml x: 17 lines of code y: 54 line churn evals/registry/evals/stock-options.yaml x: 96 lines of code y: 107 line churn evals/registry/evals/test-modelgraded-generated.yaml x: 9 lines of code y: 38 line churn evals/registry/data/pattern_identification/samples.v0.jsonl x: 3 lines of code y: 506 line churn
2711.0
line churn
  min: 1.0
  average: 44.06
  25th percentile: 3.0
  median: 8.0
  75th percentile: 25.0
  max: 2711.0
0 1349.0
lines of code
min: 1.0 | average: 27.81 | 25th percentile: 3.0 | median: 3.0 | 75th percentile: 16.0 | max: 1349.0