openai / evals
File Change Frequency

File change frequency (churn) shows the distribution of file updates (days with at least one commit).

Overview
File Change Frequency Overall
  • There are 1,496 files with 41,910 lines of code.
    • 0 files changed more than 100 times (0 lines of code)
    • 0 files changed 51-100 times (0 lines of code)
    • 1 file changed 21-50 times (242 lines of code)
    • 14 files changed 6-20 times (1,792 lines of code)
    • 1,481 files changed 1-5 times (39,876 lines of code)
0% | 0% | <1% | 4% | 95%
Legend:
101+
51-100
21-50
6-20
1-5

explore: grouped by folders | grouped by update frequency | data
Contributors Count Frequency Overall
  • There are 1,496 files with 41,910 lines of code.
    • 0 files changed by more than 25 contributors (0 lines of code)
    • 3 files changed by 11-25 contributors (559 lines of code)
    • 8 files changed by 6-10 contributors (1,225 lines of code)
    • 134 files changed by 2-5 contributors (8,415 lines of code)
    • 1,351 files changed by 1 contributor (31,711 lines of code)
0% | 1% | 2% | 20% | 75%
Legend:
26+
11-25
6-10
2-5
1

explore: grouped by folders | grouped by contributors count | data
File Change Frequency per File Extension
jsonl, yaml, py, md, txt, sh, json, gitignore, ipynb, gitattributes, html, js, in, ini, toml
File Change Frequency per Extension
The number of recorded file updates
101+
51-100
21-50
6-20
1-5
py0% | 0% | <1% | 5% | 93%
yaml0% | 0% | 0% | <1% | 99%
toml0% | 0% | 0% | 100% | 0%
jsonl0% | 0% | 0% | 0% | 100%
ipynb0% | 0% | 0% | 0% | 100%
html0% | 0% | 0% | 0% | 100%
js0% | 0% | 0% | 0% | 100%
in0% | 0% | 0% | 0% | 100%
File Change Frequency per Logical Decomposition
primary
primary (file change frequency)
The number of recorded file updates
101+
51-100
21-50
6-20
1-5
evals0% | 0% | <1% | 3% | 95%
scripts0% | 0% | 0% | 100% | 0%
ROOT0% | 0% | 0% | 94% | 5%
Most Frequently Changed Files (Top 50)

See data for all files...

File# lines# unitscreatedlast modified# changes
(days)		# contributorsfirst
contributor		latest
contributor
registry.py
in evals
 242 27 2023-03-21 2024-03-28 22 17 343165+rlbayes@users.norepl... oliver.jaffe@hotmail.co.uk
64 - 2023-03-21 2024-05-01 20 19 1520816+andremafei@users.no... erik.t.ritter@gmail.com
classify.py
in evals/elsuite/modelgraded
 97 3 2023-03-16 2023-09-26 19 7 shane@openai.com bomarni@googlemail.com
oaieval.py
in evals/cli
 253 6 2023-03-18 2024-09-30 19 18 120423412+andrew-openai@use... steven@openai.com
record.py
in evals
 450 54 2023-03-16 2024-01-26 13 9 shane@openai.com 140545726+ianmckenzie-oai@u...
api_utils.py
in evals/utils
 15 1 2023-03-18 2024-03-26 10 8 120423412+andrew-openai@use... oliver.jaffe@hotmail.co.uk
utils.py
in evals/elsuite
 150 13 2023-03-22 2023-09-26 10 6 343165+rlbayes@users.norepl... bomarni@googlemail.com
data.py
in evals
 148 21 2023-03-17 2024-04-02 9 10 343165+rlbayes@users.norepl... 150190178+josnyder-2@users....
eval.py
in evals
 170 15 2023-03-20 2024-03-28 9 9 2406911+zhangmarvin@users.n... oliver.jaffe@hotmail.co.uk
includes.py
in evals/elsuite/basic
 48 3 2023-04-11 2023-06-12 7 4 73198383+hwchung27@users.no... 120423412+andrew-openai@use...
test-modelgraded.yaml
in evals/registry/evals
 81 - 2023-03-20 2023-09-18 7 3 343165+rlbayes@users.norepl... 55913678+cholotook@users.no...
utils.py
in evals/elsuite/make_me_say
 34 4 2023-09-19 2023-12-20 6 5 140545726+ianmckenzie-oai@u... inwaves@users.noreply.githu...
48 3 2023-03-15 2024-01-10 6 6 jasonwei@openai.com 140545726+ianmckenzie-oai@u...
49 1 2023-03-16 2024-01-10 6 5 shane@openai.com 140545726+ianmckenzie-oai@u...
185 1 2023-03-16 2024-01-10 6 5 shane@openai.com 140545726+ianmckenzie-oai@u...
test-all.yaml
in evals/registry/eval_sets
 21 - 2023-03-16 2023-04-24 5 3 shane@openai.com 343165+rlbayes@users.norepl...
test-modelgraded-battle.yaml
in evals/registry/evals
 36 - 2023-03-16 2023-04-24 5 3 shane@openai.com 343165+rlbayes@users.norepl...
fuzzy_match.py
in evals/elsuite/basic
 49 3 2023-04-11 2023-06-05 5 4 73198383+hwchung27@users.no... jwang47@users.noreply.githu...
base.py
in evals
 51 1 2023-03-20 2023-09-26 5 4 343165+rlbayes@users.norepl... lukevanseters@gmail.com
match.py
in evals/elsuite/basic
 57 3 2023-04-11 2023-09-26 5 4 73198383+hwchung27@users.no... lukevanseters@gmail.com
api.py
in evals
 61 5 2023-03-28 2023-06-08 5 4 jwang47@users.noreply.githu... 131678108+wingsdrafterwork@...
oaievalset.py
in evals/cli
 111 8 2023-03-21 2024-01-10 5 4 343165+rlbayes@users.norepl... 140545726+ianmckenzie-oai@u...
classify_utils.py
in evals/elsuite/modelgraded
 145 8 2023-04-04 2023-09-18 5 2 343165+rlbayes@users.norepl... 97272807+sohenze@users.nore...
openai.py
in evals/completion_fns
 147 10 2023-04-11 2024-03-26 5 6 jwang47@users.noreply.githu... oliver.jaffe@hotmail.co.uk
best.yaml
in evals/registry/modelgraded
 10 - 2023-03-20 2023-09-18 4 2 343165+rlbayes@users.norepl... 55913678+cholotook@users.no...
base.py
in evals/elsuite/modelgraded
 16 - 2023-04-04 2023-04-27 4 1 343165+rlbayes@users.norepl... 343165+rlbayes@users.norepl...
human_cli_solver.py
in evals/solvers
 29 3 2023-11-09 2024-03-13 4 3 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
utils.py
in evals/solvers
 37 2 2023-11-09 2024-03-28 4 2 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
json_validator.py
in evals/elsuite/basic
 41 4 2023-04-14 2023-06-12 4 2 120423412+andrew-openai@use... 120423412+andrew-openai@use...
coqa-ex.yaml
in evals/registry/evals
 55 - 2023-03-17 2023-04-24 4 3 343165+rlbayes@users.norepl... 343165+rlbayes@users.norepl...
base.py
in evals/prompt
 64 8 2023-03-29 2024-03-13 4 3 343165+rlbayes@users.norepl... oliver.jaffe@hotmail.co.uk
hhh.py
in evals/solvers/prompts
 99 - 2023-11-09 2024-03-13 4 3 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
autoeval.py
in evals/elsuite/make_me_say
 116 3 2023-09-19 2023-12-05 4 4 140545726+ianmckenzie-oai@u... erik.t.ritter@gmail.com
solver.py
in evals/solvers
 125 17 2023-11-09 2024-03-28 4 2 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
validators.py
in evals/registry/data/word_association/corpus_tools
 151 11 2023-07-04 2024-01-10 4 4 douglas.monsky@gmail.com 140545726+ianmckenzie-oai@u...
in
 4 - 2023-03-30 2023-04-12 3 3 jwang47@users.noreply.githu... 120423412+andrew-openai@use...
test-modelgraded-generated.yaml
in evals/registry/evals
 9 - 2023-03-16 2023-03-27 3 2 shane@openai.com 343165+rlbayes@users.norepl...
test-modelgraded.yaml
in evals/registry/eval_sets
 14 - 2023-03-17 2023-04-06 3 2 343165+rlbayes@users.norepl... 343165+rlbayes@users.norepl...
langchain_math.py
in evals/completion_fns
 21 4 2023-04-11 2023-12-05 3 4 jwang47@users.noreply.githu... erik.t.ritter@gmail.com
langchain_llms.yaml
in evals/registry/completion_fns
 24 - 2023-04-11 2023-12-21 3 3 73198383+hwchung27@users.no... lorenzo.pacchiardi@stats.ox...
sql.yaml
in evals/registry/modelgraded
 24 - 2023-04-21 2023-05-30 3 2 mark@haym.me 133797909+jorge-openai@user...
prompts.py
in evals/elsuite/schelling_point
 25 - 2023-09-19 2024-01-10 3 2 140545726+ianmckenzie-oai@u... 140545726+ianmckenzie-oai@u...
ballots.yaml
in evals/registry/evals
 34 - 2023-09-19 2024-03-13 3 3 140545726+ianmckenzie-oai@u... oliver.jaffe@hotmail.co.uk
prompts.py
in evals/elsuite/ballots
 44 - 2023-09-19 2024-01-10 3 2 140545726+ianmckenzie-oai@u... 140545726+ianmckenzie-oai@u...
utils.py
in evals/elsuite/make_me_pay
 47 6 2023-09-19 2023-12-05 3 3 140545726+ianmckenzie-oai@u... erik.t.ritter@gmail.com
eval.py
in evals/elsuite/make_me_say
 48 3 2023-09-19 2024-03-28 3 2 140545726+ianmckenzie-oai@u... oliver.jaffe@hotmail.co.uk
metrics.py
in evals
 52 8 2023-06-02 2023-09-26 3 3 21045365+kjbilton@users.nor... bomarni@googlemail.com
task_description.py
in evals/elsuite/make_me_pay
 57 - 2023-11-15 2024-03-13 3 2 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
make-me-pay.yaml
in evals/registry/evals
 59 - 2023-09-19 2024-03-13 3 2 140545726+ianmckenzie-oai@u... oliver.jaffe@hotmail.co.uk
cot_solver.py
in evals/solvers/nested
 61 7 2024-01-29 2024-03-28 3 2 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
Files With Most Contributors (Top 50)
Based on the number of unique email addresses found in commits.

See data for all files...

File# lines# unitscreatedlast modified# changes
(days)		# contributorsfirst
contributor		latest
contributor
64 - 2023-03-21 2024-05-01 20 19 1520816+andremafei@users.no... erik.t.ritter@gmail.com
oaieval.py
in evals/cli
 253 6 2023-03-18 2024-09-30 19 18 120423412+andrew-openai@use... steven@openai.com
registry.py
in evals
 242 27 2023-03-21 2024-03-28 22 17 343165+rlbayes@users.norepl... oliver.jaffe@hotmail.co.uk
data.py
in evals
 148 21 2023-03-17 2024-04-02 9 10 343165+rlbayes@users.norepl... 150190178+josnyder-2@users....
record.py
in evals
 450 54 2023-03-16 2024-01-26 13 9 shane@openai.com 140545726+ianmckenzie-oai@u...
eval.py
in evals
 170 15 2023-03-20 2024-03-28 9 9 2406911+zhangmarvin@users.n... oliver.jaffe@hotmail.co.uk
api_utils.py
in evals/utils
 15 1 2023-03-18 2024-03-26 10 8 120423412+andrew-openai@use... oliver.jaffe@hotmail.co.uk
classify.py
in evals/elsuite/modelgraded
 97 3 2023-03-16 2023-09-26 19 7 shane@openai.com bomarni@googlemail.com
utils.py
in evals/elsuite
 150 13 2023-03-22 2023-09-26 10 6 343165+rlbayes@users.norepl... bomarni@googlemail.com
48 3 2023-03-15 2024-01-10 6 6 jasonwei@openai.com 140545726+ianmckenzie-oai@u...
openai.py
in evals/completion_fns
 147 10 2023-04-11 2024-03-26 5 6 jwang47@users.noreply.githu... oliver.jaffe@hotmail.co.uk
185 1 2023-03-16 2024-01-10 6 5 shane@openai.com 140545726+ianmckenzie-oai@u...
49 1 2023-03-16 2024-01-10 6 5 shane@openai.com 140545726+ianmckenzie-oai@u...
utils.py
in evals/elsuite/make_me_say
 34 4 2023-09-19 2023-12-20 6 5 140545726+ianmckenzie-oai@u... inwaves@users.noreply.githu...
includes.py
in evals/elsuite/basic
 48 3 2023-04-11 2023-06-12 7 4 73198383+hwchung27@users.no... 120423412+andrew-openai@use...
base.py
in evals
 51 1 2023-03-20 2023-09-26 5 4 343165+rlbayes@users.norepl... lukevanseters@gmail.com
match.py
in evals/elsuite/basic
 57 3 2023-04-11 2023-09-26 5 4 73198383+hwchung27@users.no... lukevanseters@gmail.com
fuzzy_match.py
in evals/elsuite/basic
 49 3 2023-04-11 2023-06-05 5 4 73198383+hwchung27@users.no... jwang47@users.noreply.githu...
oaievalset.py
in evals/cli
 111 8 2023-03-21 2024-01-10 5 4 343165+rlbayes@users.norepl... 140545726+ianmckenzie-oai@u...
api.py
in evals
 61 5 2023-03-28 2023-06-08 5 4 jwang47@users.noreply.githu... 131678108+wingsdrafterwork@...
autoeval.py
in evals/elsuite/make_me_say
 116 3 2023-09-19 2023-12-05 4 4 140545726+ianmckenzie-oai@u... erik.t.ritter@gmail.com
validators.py
in evals/registry/data/word_association/corpus_tools
 151 11 2023-07-04 2024-01-10 4 4 douglas.monsky@gmail.com 140545726+ianmckenzie-oai@u...
langchain_math.py
in evals/completion_fns
 21 4 2023-04-11 2023-12-05 3 4 jwang47@users.noreply.githu... erik.t.ritter@gmail.com
langchain_llm.py
in evals/completion_fns
 70 7 2023-04-11 2024-01-03 3 4 jwang47@users.noreply.githu... z@hyperf.io
test-modelgraded.yaml
in evals/registry/evals
 81 - 2023-03-20 2023-09-18 7 3 343165+rlbayes@users.norepl... 55913678+cholotook@users.no...
test-all.yaml
in evals/registry/eval_sets
 21 - 2023-03-16 2023-04-24 5 3 shane@openai.com 343165+rlbayes@users.norepl...
test-modelgraded-battle.yaml
in evals/registry/evals
 36 - 2023-03-16 2023-04-24 5 3 shane@openai.com 343165+rlbayes@users.norepl...
hhh.py
in evals/solvers/prompts
 99 - 2023-11-09 2024-03-13 4 3 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
human_cli_solver.py
in evals/solvers
 29 3 2023-11-09 2024-03-13 4 3 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
coqa-ex.yaml
in evals/registry/evals
 55 - 2023-03-17 2023-04-24 4 3 343165+rlbayes@users.norepl... 343165+rlbayes@users.norepl...
base.py
in evals/prompt
 64 8 2023-03-29 2024-03-13 4 3 343165+rlbayes@users.norepl... oliver.jaffe@hotmail.co.uk
in
 4 - 2023-03-30 2023-04-12 3 3 jwang47@users.noreply.githu... 120423412+andrew-openai@use...
solvers.py
in evals/elsuite/sandbagging
 152 16 2023-11-15 2024-01-29 3 3 oliver.jaffe@hotmail.co.uk junshern@users.noreply.gith...
strategy_solver.py
in evals/elsuite/bluff
 88 5 2023-11-15 2024-03-28 3 3 33967107+johny-b@users.nore... oliver.jaffe@hotmail.co.uk
utils.py
in evals/elsuite/make_me_pay
 47 6 2023-09-19 2023-12-05 3 3 140545726+ianmckenzie-oai@u... erik.t.ritter@gmail.com
langchain_llms.yaml
in evals/registry/completion_fns
 24 - 2023-04-11 2023-12-21 3 3 73198383+hwchung27@users.no... lorenzo.pacchiardi@stats.ox...
schelling_point.yaml
in evals/registry/evals
 66 - 2023-09-19 2024-03-13 3 3 140545726+ianmckenzie-oai@u... oliver.jaffe@hotmail.co.uk
ballots.yaml
in evals/registry/evals
 34 - 2023-09-19 2024-03-13 3 3 140545726+ianmckenzie-oai@u... oliver.jaffe@hotmail.co.uk
metrics.py
in evals
 52 8 2023-06-02 2023-09-26 3 3 21045365+kjbilton@users.nor... bomarni@googlemail.com
retrieval.py
in evals/completion_fns
 68 6 2023-04-12 2024-01-10 3 3 120423412+andrew-openai@use... 140545726+ianmckenzie-oai@u...
classify_utils.py
in evals/elsuite/modelgraded
 145 8 2023-04-04 2023-09-18 5 2 343165+rlbayes@users.norepl... 97272807+sohenze@users.nore...
solver.py
in evals/solvers
 125 17 2023-11-09 2024-03-28 4 2 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
utils.py
in evals/solvers
 37 2 2023-11-09 2024-03-28 4 2 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
json_validator.py
in evals/elsuite/basic
 41 4 2023-04-14 2023-06-12 4 2 120423412+andrew-openai@use... 120423412+andrew-openai@use...
best.yaml
in evals/registry/modelgraded
 10 - 2023-03-20 2023-09-18 4 2 343165+rlbayes@users.norepl... 55913678+cholotook@users.no...
cot_solver.py
in evals/solvers/nested
 61 7 2024-01-29 2024-03-28 3 2 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
self_consistency_solver.py
in evals/solvers/nested
 118 6 2024-01-29 2024-03-28 3 2 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
eval.py
in evals/elsuite/make_me_pay
 126 3 2023-09-19 2024-03-13 3 2 140545726+ianmckenzie-oai@u... oliver.jaffe@hotmail.co.uk
task_description.py
in evals/elsuite/make_me_pay
 57 - 2023-11-15 2024-03-13 3 2 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
eval.py
in evals/elsuite/ballots
 161 3 2023-09-19 2023-12-20 3 2 140545726+ianmckenzie-oai@u... 129281094+james-aung@users....
Files With Least Contributors (Top 50)
Based on the number of unique email addresses found in commits.

See data for all files...

File# lines# unitscreatedlast modified# changes
(days)		# contributorsfirst
contributor		latest
contributor
14 1
jsonl
match.jsonl
in evals/registry/data/coqa
 3 -
jsonl
samples.jsonl
in evals/registry/data/coqa
 3 -
generate_samples.ipynb
in evals/registry/data/backgammon
 1349 - 2023-06-22 2023-06-22 1 1 bakebrain@gmail.com bakebrain@gmail.com
actions.py
in evals/elsuite/multistep_web_tasks/webarena/browser_env
 1014 50 2024-03-21 2024-03-21 1 1 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
tools.py
in evals/elsuite/bugged_tools
 497 37 2024-03-19 2024-03-19 1 1 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
processors.py
in evals/elsuite/multistep_web_tasks/webarena/browser_env
 495 21 2024-03-21 2024-03-21 1 1 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
make_plots.py
in evals/elsuite/error_recovery/scripts
 446 20 2024-03-19 2024-03-19 1 1 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
session.py
in evals/elsuite/multistep_web_tasks
 416 28 2024-03-21 2024-03-21 1 1 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
mmlu.yaml
in evals/registry/evals
 399 - 2023-05-08 2023-05-08 1 1 jwang47@users.noreply.githu... jwang47@users.noreply.githu...
mmmu.yaml
in evals/registry/evals
 390 - 2023-12-21 2023-12-21 1 1 erik.t.ritter@gmail.com erik.t.ritter@gmail.com
make_plots.py
in evals/elsuite/identifying_variables/scripts
 325 17 2024-03-19 2024-03-19 1 1 giulio.starace@gmail.com giulio.starace@gmail.com
gen_data.py
in evals/elsuite/identifying_variables/scripts
 319 13 2024-03-19 2024-03-19 1 1 giulio.starace@gmail.com giulio.starace@gmail.com
eval.py
in evals/elsuite/skill_acquisition
 313 7 2024-03-19 2024-03-19 1 1 inwaves@users.noreply.githu... inwaves@users.noreply.githu...
plot_experiments.py
in evals/elsuite/hr_ml_agent_bench/scripts
 307 - 2024-03-21 2024-03-21 1 1 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
low_level_actions.py
in evals/elsuite/hr_ml_agent_bench
 304 13 2024-03-21 2024-03-21 1 1 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
environment.py
in evals/elsuite/hr_ml_agent_bench
 283 21 2024-03-21 2024-03-21 1 1 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
constants.py
in evals/elsuite/multistep_web_tasks/webarena/browser_env
 282 - 2024-03-21 2024-03-21 1 1 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
playwright_api.py
in evals/elsuite/multistep_web_tasks/webarena/core
 279 32 2024-03-21 2024-03-21 1 1 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
eval_run.py
in evals/elsuite/multistep_web_tasks/webarena
 277 14 2024-03-21 2024-03-21 1 1 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
evaluators.py
in evals/elsuite/multistep_web_tasks/webarena/evaluation_harness
 273 19 2024-03-21 2024-03-21 1 1 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
make_plots.py
in evals/elsuite/already_said_that/scripts
 263 8 2024-03-19 2024-03-19 1 1 giulio.starace@gmail.com giulio.starace@gmail.com
eval.py
in evals/elsuite/incontext_rl
 246 12 2024-03-19 2024-03-19 1 1 129281094+james-aung@users.... 129281094+james-aung@users....
eval.py
in evals/elsuite/function_deduction
 244 14 2024-03-19 2024-03-19 1 1 129281094+james-aung@users.... 129281094+james-aung@users....
dataset_creation.py
in evals/elsuite/cant_do_that_anymore/scripts
 235 8 2024-03-19 2024-03-19 1 1 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
make_plots.py
in evals/elsuite/track_the_stat/scripts
 235 9 2024-03-19 2024-03-19 1 1 giulio.starace@gmail.com giulio.starace@gmail.com
make_plots.py
in evals/elsuite/ballots/scripts
 233 12 2024-03-13 2024-03-13 1 1 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
plot_experiments.py
in evals/elsuite/incontext_rl/scripts
 233 8 2024-03-19 2024-03-19 1 1 129281094+james-aung@users.... 129281094+james-aung@users....
eval.py
in evals/elsuite/identifying_variables
 227 14 2024-03-19 2024-03-19 1 1 giulio.starace@gmail.com giulio.starace@gmail.com
raven-matrices.yaml
in evals/registry/evals
 224 - 2023-06-08 2023-06-08 1 1 44269117+ggendro@users.nore... 44269117+ggendro@users.nore...
diagonal_dataset_creation.py
in evals/elsuite/cant_do_that_anymore/scripts
 216 6 2024-03-19 2024-03-19 1 1 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
corrset.py
in evals/elsuite/identifying_variables/renderers
 216 11 2024-03-19 2024-03-19 1 1 giulio.starace@gmail.com giulio.starace@gmail.com
eval.py
in evals/elsuite/self_prompting
 210 7 2023-11-15 2023-11-15 1 1 junshern@users.noreply.gith... junshern@users.noreply.gith...
eval.py
in evals/elsuite/bugged_tools
 210 9 2024-03-19 2024-03-19 1 1 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
cards.py
in evals/elsuite/bluff/bluff
 206 37 2023-11-15 2023-11-15 1 1 33967107+johny-b@users.nore... 33967107+johny-b@users.nore...
eval.py
in evals/elsuite/error_recovery
 204 9 2024-03-19 2024-03-19 1 1 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
pieces.py
in evals/elsuite/cant_do_that_anymore/chess
 203 9 2024-03-19 2024-03-19 1 1 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
custom_datasets.py
in evals/elsuite/steganography/scripts/dataset
 197 12 2024-03-13 2024-03-13 1 1 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
make_plots.py
in evals/elsuite/function_deduction/scripts
 195 4 2024-03-19 2024-03-19 1 1 129281094+james-aung@users.... 129281094+james-aung@users....
basic_browser_env.py
in evals/elsuite/multistep_web_tasks/webarena/browser_env
 191 11 2024-03-21 2024-03-21 1 1 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
high_level_actions.py
in evals/elsuite/hr_ml_agent_bench
 191 4 2024-03-21 2024-03-21 1 1 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
utils.py
in evals/elsuite/multistep_web_tasks/webarena/core
 188 7 2024-03-21 2024-03-21 1 1 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
openai_assistants_solver.py
in evals/solvers/providers/openai
 186 11 2024-03-28 2024-03-28 1 1 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
openai_solver.py
in evals/solvers/providers/openai
 181 18 2024-03-28 2024-03-28 1 1 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
solver_tools_convo.py
in evals/elsuite
 181 11 2024-03-19 2024-03-19 1 1 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
ukraine-gec.yaml
in evals/registry/evals
 180 - 2023-06-02 2023-06-02 1 1 roman.mashta@gmail.com roman.mashta@gmail.com
utils.py
in evals/elsuite/cant_do_that_anymore
 178 12 2024-03-19 2024-03-19 1 1 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
strong_solver.py
in evals/elsuite/multistep_web_tasks/solvers/strong_solver
 173 12 2024-03-21 2024-03-21 1 1 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
autoeval.py
in evals/elsuite/hr_ml_agent_bench
 172 2 2024-03-21 2024-03-21 1 1 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
eval.py
in evals/elsuite/cant_do_that_anymore
 170 4 2024-03-19 2024-03-19 1 1 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
Correlations

File Size vs. Number of Changes: 1496 points

evals/cli/oaieval.py x: 253 lines of code y: 19 # changes evals/registry/data/imo_exact_answers/samples.jsonl x: 3 lines of code y: 1 # changes evals/registry/evals/imo_exact_answers.yaml x: 8 lines of code y: 1 # changes pyproject.toml x: 64 lines of code y: 20 # changes evals/data.py x: 148 lines of code y: 9 # changes evals/completion_fns/solver_completion_fn.py x: 47 lines of code y: 1 # changes evals/elsuite/bluff/strategy_solver.py x: 88 lines of code y: 3 # changes evals/elsuite/bugged_tools/task_description.py x: 9 lines of code y: 2 # changes evals/elsuite/function_deduction/prompts.py x: 6 lines of code y: 2 # changes evals/elsuite/function_deduction/solvers.py x: 140 lines of code y: 2 # changes evals/elsuite/hr_ml_agent_bench/solvers/baseline.py x: 90 lines of code y: 2 # changes evals/elsuite/make_me_say/eval.py x: 48 lines of code y: 3 # changes evals/eval.py x: 170 lines of code y: 9 # changes evals/registry.py x: 242 lines of code y: 22 # changes evals/registry/solvers/already_said_that.yaml x: 75 lines of code y: 2 # changes evals/registry/solvers/bluff.yaml x: 80 lines of code y: 2 # changes evals/registry/solvers/cant_do_that_anymore.yaml x: 16 lines of code y: 2 # changes evals/registry/solvers/defaults.yaml x: 294 lines of code y: 3 # changes evals/registry/solvers/error_recovery.yaml x: 33 lines of code y: 2 # changes evals/registry/solvers/function_deduction.yaml x: 174 lines of code y: 2 # changes evals/registry/solvers/hr-ml-agent-bench.yaml x: 37 lines of code y: 2 # changes evals/registry/solvers/incontext_rl.yaml x: 24 lines of code y: 2 # changes evals/registry/solvers/make-me-pay.yaml x: 101 lines of code y: 2 # changes evals/registry/solvers/self_prompting.yaml x: 96 lines of code y: 2 # changes evals/registry/solvers/skill_acquisition.yaml x: 267 lines of code y: 2 # changes evals/registry/solvers/theory_of_mind.yaml x: 394 lines of code y: 3 # changes evals/registry/solvers/together.yaml x: 85 lines of code y: 2 # changes evals/solvers/memory.py x: 50 lines of code y: 1 # changes evals/solvers/nested/cot_solver.py x: 61 lines of code y: 3 # changes evals/solvers/nested/self_consistency_solver.py x: 118 lines of code y: 3 # changes evals/solvers/prompts/cot.py x: 4 lines of code y: 2 # changes evals/solvers/providers/openai/openai_assistants_solver.py x: 186 lines of code y: 1 # changes evals/solvers/providers/openai/openai_solver.py x: 181 lines of code y: 1 # changes evals/solvers/providers/together/together_solver.py x: 68 lines of code y: 1 # changes evals/solvers/solver.py x: 125 lines of code y: 4 # changes evals/solvers/utils.py x: 37 lines of code y: 4 # changes evals/registry/solvers/gemini.yaml x: 15 lines of code y: 1 # changes evals/solvers/providers/google/gemini_solver.py x: 157 lines of code y: 1 # changes evals/completion_fns/openai.py x: 147 lines of code y: 5 # changes evals/utils/api_utils.py x: 15 lines of code y: 10 # changes evals/registry/solvers/anthropic.yaml x: 90 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/__init__.py x: 1 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/actions.py x: 37 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/auto_marking.py x: 53 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/autoeval.py x: 172 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/ant/baselines/human.py x: 36 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/ant/baselines/naive.py x: 28 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/ant/scripts/grade.py x: 43 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/bipedal_walker/baselines/human.py x: 57 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/bipedal_walker/scripts/grade.py x: 31 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/cartpole/scripts/grade.py x: 40 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/cifar10/env/train.py x: 105 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/cifar10/scripts/prepare.py x: 5 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/feedback/scripts/prepare.py x: 24 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/house_price/scripts/grade.py x: 33 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/house_price/scripts/prepare.py x: 20 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/humanoid/scripts/grade.py x: 45 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/ogbn_arxiv/env/train.py x: 126 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/parkinsons_disease/env/train.py x: 124 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/parkinsons_disease/scripts/prepare.py x: 98 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/spaceship_titanic/scripts/grade.py x: 26 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/vectorization/env/train.py x: 118 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/vectorization/scripts/grade.py x: 64 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/vectorization/scripts/human_baseline.py x: 83 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/environment.py x: 283 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/eval.py x: 91 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/high_level_actions.py x: 191 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/low_level_actions.py x: 304 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/scripts/plot_experiments.py x: 307 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/scripts/run_experiments.py x: 60 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/utils.py x: 112 lines of code y: 1 # changes evals/registry/evals/hr-ml-agent-bench.yaml x: 137 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/constants.py x: 70 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/docker/flask-playwright/app.py x: 165 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/docker/homepage/templates/index.html x: 108 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/reproducibility/make_plots.py x: 94 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/session.py x: 416 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/solvers/strong_solver/strong_solver.py x: 173 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/solvers/webarena_solvers/webarena_prompts.py x: 12 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/bash_env/basic_bash_env.py x: 163 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/browser_env/actions.py x: 1014 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/browser_env/auto_login.py x: 100 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/browser_env/browser_utils.py x: 78 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/browser_env/constants.py x: 282 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/browser_env/helper_functions.py x: 129 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/browser_env/processors.py x: 495 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/core/env.py x: 75 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/core/playwright_api.py x: 279 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/core/utils.py x: 188 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/eval_run.py x: 277 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/evaluation_harness/evaluators.py x: 273 lines of code y: 1 # changes evals/registry/solvers/multistep_web_tasks.yaml x: 56 lines of code y: 1 # changes evals/elsuite/incontext_rl/eval.py x: 246 lines of code y: 1 # changes evals/elsuite/incontext_rl/scripts/plot_experiments.py x: 233 lines of code y: 1 # changes evals/elsuite/function_deduction/eval.py x: 244 lines of code y: 1 # changes evals/elsuite/function_deduction/scripts/make_plots.py x: 195 lines of code y: 1 # changes evals/elsuite/identifying_variables/eval.py x: 227 lines of code y: 1 # changes evals/elsuite/identifying_variables/graph_utils.py x: 130 lines of code y: 1 # changes evals/elsuite/identifying_variables/renderers/corrset.py x: 216 lines of code y: 1 # changes evals/elsuite/identifying_variables/scripts/gen_data.py x: 319 lines of code y: 1 # changes evals/elsuite/identifying_variables/scripts/make_plots.py x: 325 lines of code y: 1 # changes evals/elsuite/identifying_variables/structs.py x: 18 lines of code y: 1 # changes evals/elsuite/identifying_variables/utils.py x: 72 lines of code y: 1 # changes evals/elsuite/track_the_stat/scripts/make_plots.py x: 235 lines of code y: 1 # changes evals/elsuite/already_said_that/distractors.py x: 81 lines of code y: 1 # changes evals/elsuite/already_said_that/scripts/make_plots.py x: 263 lines of code y: 1 # changes evals/elsuite/already_said_that/utils.py x: 104 lines of code y: 1 # changes evals/utils/log_utils.py x: 53 lines of code y: 2 # changes evals/elsuite/twenty_questions/eval.py x: 169 lines of code y: 1 # changes evals/elsuite/skill_acquisition/eval.py x: 313 lines of code y: 1 # changes evals/elsuite/skill_acquisition/scraping/scrape_distractor_articles.py x: 76 lines of code y: 1 # changes evals/elsuite/skill_acquisition/solvers.py x: 13 lines of code y: 1 # changes evals/elsuite/skill_acquisition/utils.py x: 115 lines of code y: 1 # changes evals/elsuite/error_recovery/eval.py x: 204 lines of code y: 1 # changes evals/elsuite/error_recovery/scripts/make_plots.py x: 446 lines of code y: 1 # changes evals/elsuite/cant_do_that_anymore/chess/move_variants.py x: 113 lines of code y: 1 # changes evals/elsuite/cant_do_that_anymore/chess/notation.py x: 61 lines of code y: 1 # changes evals/elsuite/cant_do_that_anymore/utils.py x: 178 lines of code y: 1 # changes evals/elsuite/bugged_tools/bugged_tools.py x: 132 lines of code y: 1 # changes evals/elsuite/bugged_tools/eval.py x: 210 lines of code y: 1 # changes evals/elsuite/bugged_tools/tools.py x: 497 lines of code y: 1 # changes evals/elsuite/basic/match_with_solvers.py x: 65 lines of code y: 2 # changes evals/elsuite/bluff/solver_player.py x: 82 lines of code y: 2 # changes evals/elsuite/make_me_pay/eval.py x: 126 lines of code y: 3 # changes evals/elsuite/make_me_pay/task_description.py x: 57 lines of code y: 3 # changes evals/elsuite/make_me_say/core.py x: 223 lines of code y: 3 # changes evals/prompt/base.py x: 64 lines of code y: 4 # changes evals/registry/evals/ballots.yaml x: 34 lines of code y: 3 # changes evals/registry/evals/make-me-pay.yaml x: 59 lines of code y: 3 # changes evals/registry/evals/schelling_point.yaml x: 66 lines of code y: 3 # changes evals/registry/evals/self_prompting.yaml x: 19 lines of code y: 2 # changes evals/registry/evals/theory_of_mind.yaml x: 48 lines of code y: 2 # changes evals/solvers/human_cli_solver.py x: 29 lines of code y: 4 # changes evals/solvers/nested/fewshot_solver.py x: 91 lines of code y: 2 # changes evals/solvers/prompts/hhh.py x: 99 lines of code y: 4 # changes evals/elsuite/sandbagging/solvers.py x: 152 lines of code y: 3 # changes evals/elsuite/self_prompting/solvers/baselines.py x: 45 lines of code y: 2 # changes evals/elsuite/self_prompting/solvers/custom_cot_solver.py x: 57 lines of code y: 2 # changes evals/record.py x: 450 lines of code y: 13 # changes evals/cli/oaievalset.py x: 111 lines of code y: 5 # changes evals/elsuite/ballots/prompts.py x: 44 lines of code y: 3 # changes evals/elsuite/sandbagging/defaults.py x: 12 lines of code y: 2 # changes evals/elsuite/sandbagging/sandbagging_eval.py x: 68 lines of code y: 2 # changes evals/elsuite/schelling_point/prompts.py x: 25 lines of code y: 3 # changes evals/elsuite/schelling_point/utils.py x: 61 lines of code y: 2 # changes evals/elsuite/steganography/eval.py x: 69 lines of code y: 2 # changes evals/registry/data/word_association/corpus_tools/validators.py x: 151 lines of code y: 4 # changes scripts/battle_generator.py x: 49 lines of code y: 6 # changes scripts/modelgraded_generator.py x: 185 lines of code y: 6 # changes scripts/pattern_identification_generator.py x: 48 lines of code y: 6 # changes evals/elsuite/mmmu/eval.py x: 159 lines of code y: 3 # changes evals/completion_fns/langchain_llm.py x: 70 lines of code y: 3 # changes evals/registry/completion_fns/langchain_llms.yaml x: 24 lines of code y: 3 # changes evals/registry/evals/mmmu.yaml x: 390 lines of code y: 1 # changes evals/elsuite/make_me_say/utils.py x: 34 lines of code y: 6 # changes evals/elsuite/ballots/eval.py x: 161 lines of code y: 3 # changes evals/elsuite/schelling_point/eval.py x: 73 lines of code y: 2 # changes evals/elsuite/bluff/eval.py x: 164 lines of code y: 2 # changes evals/completion_fns/langchain_math.py x: 21 lines of code y: 3 # changes evals/elsuite/make_me_say/autoeval.py x: 116 lines of code y: 4 # changes evals/elsuite/bluff/bluff/cards.py x: 206 lines of code y: 1 # changes evals/elsuite/make_me_pay/makemepay.py x: 222 lines of code y: 2 # changes evals/elsuite/lambada.py x: 39 lines of code y: 2 # changes evals/elsuite/modelgraded/classify.py x: 97 lines of code y: 19 # changes evals/elsuite/utils.py x: 150 lines of code y: 10 # changes evals/metrics.py x: 52 lines of code y: 3 # changes evals/registry/data/canto_wu_pronunciation/csv_to_json.py x: 55 lines of code y: 2 # changes evals/registry/data/german-part-of-speech/parsePosDe.py x: 172 lines of code y: 2 # changes evals/registry/data/mapping_to_matricies/data_generator.py x: 30 lines of code y: 2 # changes evals/registry/data/mazes/nxn_maze_eval_generator.py x: 162 lines of code y: 2 # changes evals/registry/data/simple_physics_engine/wave_function_collapse.py x: 157 lines of code y: 2 # changes evals/registry/data/solve-for-variable/tools/solve.py x: 231 lines of code y: 2 # changes evals/registry/data/solve-for-variable/tools/tester.py x: 94 lines of code y: 2 # changes evals/registry/data/word_association/corpus_tools/pipelines.py x: 14 lines of code y: 2 # changes evals/registry/data/word_association/corpus_tools/processor.py x: 36 lines of code y: 2 # changes evals/registry/data/word_association/corpus_tools/sample_generators.py x: 154 lines of code y: 2 # changes evals/base.py x: 51 lines of code y: 5 # changes evals/elsuite/basic/match.py x: 57 lines of code y: 5 # changes evals/registry/evals/test-modelgraded.yaml x: 81 lines of code y: 7 # changes evals/registry/modelgraded/best.yaml x: 10 lines of code y: 4 # changes evals/elsuite/modelgraded/classify_utils.py x: 145 lines of code y: 5 # changes evals/registry/data/backgammon/generate_samples.ipynb x: 1349 lines of code y: 1 # changes evals/elsuite/basic/includes.py x: 48 lines of code y: 7 # changes evals/elsuite/basic/json_validator.py x: 41 lines of code y: 4 # changes evals/api.py x: 61 lines of code y: 5 # changes evals/registry/evals/raven-matrices.yaml x: 224 lines of code y: 1 # changes evals/registry/evals/mmlu.yaml x: 399 lines of code y: 1 # changes evals/elsuite/modelgraded/base.py x: 16 lines of code y: 4 # changes evals/registry/eval_sets/test-all.yaml x: 21 lines of code y: 5 # changes evals/registry/evals/coqa-ex.yaml x: 55 lines of code y: 4 # changes evals/registry/evals/test-modelgraded-battle.yaml x: 36 lines of code y: 5 # changes evals/registry/modelgraded/closedqa.yaml x: 21 lines of code y: 2 # changes MANIFEST.in x: 4 lines of code y: 3 # changes evals/registry/eval_sets/test-modelgraded.yaml x: 14 lines of code y: 3 # changes evals/registry/evals/stock-options.yaml x: 96 lines of code y: 1 # changes evals/registry/evals/test-modelgraded-generated.yaml x: 9 lines of code y: 3 # changes
22.0
# changes
  min: 1.0
  average: 1.25
  25th percentile: 1.0
  median: 1.0
  75th percentile: 1.0
  max: 22.0
0 1349.0
lines of code
min: 1.0 | average: 28.01 | 25th percentile: 3.0 | median: 3.0 | 75th percentile: 16.0 | max: 1349.0

Number of Contributors vs. Number of Changes: 1496 points

evals/cli/oaieval.py x: 18 # contributors y: 19 # changes evals/registry/data/imo_exact_answers/samples.jsonl x: 1 # contributors y: 1 # changes pyproject.toml x: 19 # contributors y: 20 # changes evals/data.py x: 10 # contributors y: 9 # changes evals/elsuite/bluff/strategy_solver.py x: 3 # contributors y: 3 # changes evals/elsuite/bugged_tools/task_description.py x: 1 # contributors y: 2 # changes evals/elsuite/function_deduction/prompts.py x: 2 # contributors y: 2 # changes evals/elsuite/make_me_say/eval.py x: 2 # contributors y: 3 # changes evals/eval.py x: 9 # contributors y: 9 # changes evals/registry.py x: 17 # contributors y: 22 # changes evals/solvers/solver.py x: 2 # contributors y: 4 # changes evals/completion_fns/openai.py x: 6 # contributors y: 5 # changes evals/utils/api_utils.py x: 8 # contributors y: 10 # changes evals/prompt/base.py x: 3 # contributors y: 4 # changes evals/record.py x: 9 # contributors y: 13 # changes evals/cli/oaievalset.py x: 4 # contributors y: 5 # changes evals/registry/data/word_association/corpus_tools/validators.py x: 4 # contributors y: 4 # changes scripts/battle_generator.py x: 5 # contributors y: 6 # changes scripts/pattern_identification_generator.py x: 6 # contributors y: 6 # changes evals/elsuite/mmmu/eval.py x: 1 # contributors y: 3 # changes evals/completion_fns/langchain_llm.py x: 4 # contributors y: 3 # changes evals/elsuite/modelgraded/classify.py x: 7 # contributors y: 19 # changes evals/elsuite/utils.py x: 6 # contributors y: 10 # changes evals/registry/evals/test-modelgraded.yaml x: 3 # contributors y: 7 # changes evals/elsuite/modelgraded/classify_utils.py x: 2 # contributors y: 5 # changes evals/elsuite/basic/includes.py x: 4 # contributors y: 7 # changes evals/elsuite/modelgraded/base.py x: 1 # contributors y: 4 # changes evals/registry/eval_sets/test-all.yaml x: 3 # contributors y: 5 # changes evals/registry/data/logiqa/logiqa.jsonl x: 2 # contributors y: 1 # changes
22.0
# changes
  min: 1.0
  average: 1.25
  25th percentile: 1.0
  median: 1.0
  75th percentile: 1.0
  max: 22.0
0 19.0
# contributors
min: 1.0 | average: 1.19 | 25th percentile: 1.0 | median: 1.0 | 75th percentile: 1.0 | max: 19.0

Number of Contributors vs. File Size: 1496 points

evals/cli/oaieval.py x: 18 # contributors y: 253 lines of code evals/registry/data/imo_exact_answers/samples.jsonl x: 1 # contributors y: 3 lines of code evals/registry/evals/imo_exact_answers.yaml x: 1 # contributors y: 8 lines of code pyproject.toml x: 19 # contributors y: 64 lines of code evals/data.py x: 10 # contributors y: 148 lines of code evals/completion_fns/solver_completion_fn.py x: 1 # contributors y: 47 lines of code evals/elsuite/bluff/strategy_solver.py x: 3 # contributors y: 88 lines of code evals/elsuite/function_deduction/prompts.py x: 2 # contributors y: 6 lines of code evals/elsuite/function_deduction/solvers.py x: 2 # contributors y: 140 lines of code evals/elsuite/hr_ml_agent_bench/solvers/baseline.py x: 2 # contributors y: 90 lines of code evals/elsuite/make_me_say/eval.py x: 2 # contributors y: 48 lines of code evals/eval.py x: 9 # contributors y: 170 lines of code evals/registry.py x: 17 # contributors y: 242 lines of code evals/registry/solvers/already_said_that.yaml x: 2 # contributors y: 75 lines of code evals/registry/solvers/bluff.yaml x: 2 # contributors y: 80 lines of code evals/registry/solvers/cant_do_that_anymore.yaml x: 1 # contributors y: 16 lines of code evals/registry/solvers/defaults.yaml x: 2 # contributors y: 294 lines of code evals/registry/solvers/error_recovery.yaml x: 1 # contributors y: 33 lines of code evals/registry/solvers/function_deduction.yaml x: 2 # contributors y: 174 lines of code evals/registry/solvers/hr-ml-agent-bench.yaml x: 2 # contributors y: 37 lines of code evals/registry/solvers/incontext_rl.yaml x: 2 # contributors y: 24 lines of code evals/registry/solvers/make-me-pay.yaml x: 2 # contributors y: 101 lines of code evals/registry/solvers/self_prompting.yaml x: 2 # contributors y: 96 lines of code evals/registry/solvers/skill_acquisition.yaml x: 2 # contributors y: 267 lines of code evals/registry/solvers/theory_of_mind.yaml x: 2 # contributors y: 394 lines of code evals/solvers/memory.py x: 1 # contributors y: 50 lines of code evals/solvers/nested/cot_solver.py x: 2 # contributors y: 61 lines of code evals/solvers/nested/self_consistency_solver.py x: 2 # contributors y: 118 lines of code evals/solvers/prompts/cot.py x: 2 # contributors y: 4 lines of code evals/solvers/providers/openai/openai_assistants_solver.py x: 1 # contributors y: 186 lines of code evals/solvers/providers/openai/openai_solver.py x: 1 # contributors y: 181 lines of code evals/solvers/providers/together/together_solver.py x: 1 # contributors y: 68 lines of code evals/solvers/solver.py x: 2 # contributors y: 125 lines of code evals/registry/solvers/gemini.yaml x: 1 # contributors y: 15 lines of code evals/solvers/providers/google/gemini_solver.py x: 1 # contributors y: 157 lines of code evals/completion_fns/openai.py x: 6 # contributors y: 147 lines of code evals/utils/api_utils.py x: 8 # contributors y: 15 lines of code evals/registry/solvers/anthropic.yaml x: 1 # contributors y: 90 lines of code evals/elsuite/hr_ml_agent_bench/autoeval.py x: 1 # contributors y: 172 lines of code evals/elsuite/hr_ml_agent_bench/benchmarks/ant/baselines/naive.py x: 1 # contributors y: 28 lines of code evals/elsuite/hr_ml_agent_bench/benchmarks/bipedal_walker/baselines/human.py x: 1 # contributors y: 57 lines of code evals/elsuite/hr_ml_agent_bench/benchmarks/cartpole/scripts/grade.py x: 1 # contributors y: 40 lines of code evals/elsuite/hr_ml_agent_bench/benchmarks/cifar10/env/train.py x: 1 # contributors y: 105 lines of code evals/elsuite/hr_ml_agent_bench/benchmarks/feedback/scripts/prepare.py x: 1 # contributors y: 24 lines of code evals/elsuite/hr_ml_agent_bench/benchmarks/ogbn_arxiv/env/train.py x: 1 # contributors y: 126 lines of code evals/elsuite/hr_ml_agent_bench/benchmarks/parkinsons_disease/scripts/prepare.py x: 1 # contributors y: 98 lines of code evals/elsuite/hr_ml_agent_bench/benchmarks/vectorization/env/train.py x: 1 # contributors y: 118 lines of code evals/elsuite/hr_ml_agent_bench/benchmarks/vectorization/scripts/human_baseline.py x: 1 # contributors y: 83 lines of code evals/elsuite/hr_ml_agent_bench/environment.py x: 1 # contributors y: 283 lines of code evals/elsuite/hr_ml_agent_bench/eval.py x: 1 # contributors y: 91 lines of code evals/elsuite/hr_ml_agent_bench/low_level_actions.py x: 1 # contributors y: 304 lines of code evals/elsuite/hr_ml_agent_bench/scripts/run_experiments.py x: 1 # contributors y: 60 lines of code evals/elsuite/hr_ml_agent_bench/utils.py x: 1 # contributors y: 112 lines of code evals/registry/evals/hr-ml-agent-bench.yaml x: 1 # contributors y: 137 lines of code evals/elsuite/multistep_web_tasks/constants.py x: 1 # contributors y: 70 lines of code evals/elsuite/multistep_web_tasks/docker/flask-playwright/app.py x: 1 # contributors y: 165 lines of code evals/elsuite/multistep_web_tasks/docker/homepage/templates/index.html x: 1 # contributors y: 108 lines of code evals/elsuite/multistep_web_tasks/session.py x: 1 # contributors y: 416 lines of code evals/elsuite/multistep_web_tasks/webarena/bash_env/basic_bash_env.py x: 1 # contributors y: 163 lines of code evals/elsuite/multistep_web_tasks/webarena/browser_env/actions.py x: 1 # contributors y: 1014 lines of code evals/elsuite/multistep_web_tasks/webarena/browser_env/browser_utils.py x: 1 # contributors y: 78 lines of code evals/elsuite/multistep_web_tasks/webarena/browser_env/helper_functions.py x: 1 # contributors y: 129 lines of code evals/elsuite/multistep_web_tasks/webarena/browser_env/processors.py x: 1 # contributors y: 495 lines of code evals/elsuite/multistep_web_tasks/webarena/core/playwright_api.py x: 1 # contributors y: 279 lines of code evals/elsuite/multistep_web_tasks/webarena/evaluation_harness/evaluators.py x: 1 # contributors y: 273 lines of code evals/elsuite/incontext_rl/eval.py x: 1 # contributors y: 246 lines of code evals/elsuite/incontext_rl/scripts/plot_experiments.py x: 1 # contributors y: 233 lines of code evals/elsuite/function_deduction/eval.py x: 1 # contributors y: 244 lines of code evals/elsuite/function_deduction/scripts/make_plots.py x: 1 # contributors y: 195 lines of code evals/elsuite/identifying_variables/eval.py x: 1 # contributors y: 227 lines of code evals/elsuite/identifying_variables/renderers/corrset.py x: 1 # contributors y: 216 lines of code evals/elsuite/identifying_variables/scripts/gen_data.py x: 1 # contributors y: 319 lines of code evals/elsuite/identifying_variables/scripts/make_plots.py x: 1 # contributors y: 325 lines of code evals/elsuite/track_the_stat/scripts/make_plots.py x: 1 # contributors y: 235 lines of code evals/elsuite/already_said_that/scripts/make_plots.py x: 1 # contributors y: 263 lines of code evals/elsuite/skill_acquisition/eval.py x: 1 # contributors y: 313 lines of code evals/elsuite/error_recovery/eval.py x: 1 # contributors y: 204 lines of code evals/elsuite/error_recovery/scripts/make_plots.py x: 1 # contributors y: 446 lines of code evals/elsuite/cant_do_that_anymore/utils.py x: 1 # contributors y: 178 lines of code evals/elsuite/bugged_tools/eval.py x: 1 # contributors y: 210 lines of code evals/elsuite/basic/match_with_solvers.py x: 2 # contributors y: 65 lines of code evals/elsuite/make_me_pay/task_description.py x: 2 # contributors y: 57 lines of code evals/elsuite/make_me_say/core.py x: 2 # contributors y: 223 lines of code evals/elsuite/steganography/scripts/dataset/custom_datasets.py x: 1 # contributors y: 197 lines of code evals/prompt/base.py x: 3 # contributors y: 64 lines of code evals/registry/evals/ballots.yaml x: 3 # contributors y: 34 lines of code evals/registry/evals/self_prompting.yaml x: 2 # contributors y: 19 lines of code evals/solvers/human_cli_solver.py x: 3 # contributors y: 29 lines of code evals/solvers/nested/fewshot_solver.py x: 2 # contributors y: 91 lines of code evals/solvers/prompts/hhh.py x: 3 # contributors y: 99 lines of code evals/elsuite/sandbagging/solvers.py x: 3 # contributors y: 152 lines of code evals/record.py x: 9 # contributors y: 450 lines of code evals/cli/oaievalset.py x: 4 # contributors y: 111 lines of code evals/elsuite/ballots/prompts.py x: 2 # contributors y: 44 lines of code evals/elsuite/sandbagging/defaults.py x: 2 # contributors y: 12 lines of code evals/registry/data/word_association/corpus_tools/validators.py x: 4 # contributors y: 151 lines of code scripts/battle_generator.py x: 5 # contributors y: 49 lines of code scripts/modelgraded_generator.py x: 5 # contributors y: 185 lines of code scripts/pattern_identification_generator.py x: 6 # contributors y: 48 lines of code evals/completion_fns/langchain_llm.py x: 4 # contributors y: 70 lines of code evals/registry/completion_fns/langchain_llms.yaml x: 3 # contributors y: 24 lines of code evals/registry/evals/mmmu.yaml x: 1 # contributors y: 390 lines of code evals/elsuite/make_me_say/utils.py x: 5 # contributors y: 34 lines of code evals/elsuite/ballots/eval.py x: 2 # contributors y: 161 lines of code evals/elsuite/schelling_point/eval.py x: 2 # contributors y: 73 lines of code evals/completion_fns/langchain_math.py x: 4 # contributors y: 21 lines of code evals/elsuite/make_me_pay/utils.py x: 3 # contributors y: 47 lines of code evals/elsuite/make_me_say/autoeval.py x: 4 # contributors y: 116 lines of code evals/elsuite/lambada.py x: 2 # contributors y: 39 lines of code evals/elsuite/modelgraded/classify.py x: 7 # contributors y: 97 lines of code evals/elsuite/utils.py x: 6 # contributors y: 150 lines of code evals/metrics.py x: 3 # contributors y: 52 lines of code evals/registry/data/mapping_to_matricies/data_generator.py x: 2 # contributors y: 30 lines of code evals/registry/data/simple_physics_engine/wave_function_collapse.py x: 2 # contributors y: 157 lines of code evals/registry/data/solve-for-variable/tools/solve.py x: 2 # contributors y: 231 lines of code evals/registry/data/word_association/corpus_tools/sample_generators.py x: 2 # contributors y: 154 lines of code evals/base.py x: 4 # contributors y: 51 lines of code evals/elsuite/basic/match.py x: 4 # contributors y: 57 lines of code evals/registry/evals/test-modelgraded.yaml x: 3 # contributors y: 81 lines of code evals/elsuite/modelgraded/classify_utils.py x: 2 # contributors y: 145 lines of code evals/registry/data/backgammon/generate_samples.ipynb x: 1 # contributors y: 1349 lines of code evals/api.py x: 4 # contributors y: 61 lines of code evals/registry/evals/mmlu.yaml x: 1 # contributors y: 399 lines of code evals/registry/eval_sets/test-all.yaml x: 3 # contributors y: 21 lines of code evals/registry/evals/coqa-ex.yaml x: 3 # contributors y: 55 lines of code MANIFEST.in x: 3 # contributors y: 4 lines of code
1349.0
lines of code
  min: 1.0
  average: 28.01
  25th percentile: 3.0
  median: 3.0
  75th percentile: 16.0
  max: 1349.0
0 19.0
# contributors
min: 1.0 | average: 1.19 | 25th percentile: 1.0 | median: 1.0 | 75th percentile: 1.0 | max: 19.0