openai / evals
File Change Frequency

File change frequency (churn) shows the distribution of file updates (days with at least one commit).

Overview
File Change Frequency Overall
  • There are 1,496 files with 41,910 lines of code.
    • 0 files changed more than 100 times (0 lines of code)
    • 0 files changed 51-100 times (0 lines of code)
    • 1 file changed 21-50 times (242 lines of code)
    • 14 files changed 6-20 times (1,792 lines of code)
    • 1,481 files changed 1-5 times (39,876 lines of code)
0% | 0% | <1% | 4% | 95%
Legend:
101+
51-100
21-50
6-20
1-5

explore: grouped by folders | grouped by update frequency | data
Contributors Count Frequency Overall
  • There are 1,496 files with 41,910 lines of code.
    • 0 files changed by more than 25 contributors (0 lines of code)
    • 3 files changed by 11-25 contributors (559 lines of code)
    • 8 files changed by 6-10 contributors (1,225 lines of code)
    • 134 files changed by 2-5 contributors (8,415 lines of code)
    • 1,351 files changed by 1 contributor (31,711 lines of code)
0% | 1% | 2% | 20% | 75%
Legend:
26+
11-25
6-10
2-5
1

explore: grouped by folders | grouped by contributors count | data
File Change Frequency per File Extension
jsonl, yaml, py, md, txt, sh, json, gitignore, ipynb, gitattributes, html, js, in, ini, toml
File Change Frequency per Extension
The number of recorded file updates
101+
51-100
21-50
6-20
1-5
py0% | 0% | <1% | 5% | 93%
yaml0% | 0% | 0% | <1% | 99%
toml0% | 0% | 0% | 100% | 0%
jsonl0% | 0% | 0% | 0% | 100%
ipynb0% | 0% | 0% | 0% | 100%
html0% | 0% | 0% | 0% | 100%
js0% | 0% | 0% | 0% | 100%
in0% | 0% | 0% | 0% | 100%
File Change Frequency per Logical Decomposition
primary
primary (file change frequency)
The number of recorded file updates
101+
51-100
21-50
6-20
1-5
evals0% | 0% | <1% | 3% | 95%
scripts0% | 0% | 0% | 100% | 0%
ROOT0% | 0% | 0% | 94% | 5%
Most Frequently Changed Files (Top 50)

See data for all files...

File# lines# unitscreatedlast modified# changes
(days)
# contributorsfirst
contributor
latest
contributor
registry.py
in evals
242 27 2023-03-21 2024-03-28 22 17 343165+rlbayes@users.norepl... oliver.jaffe@hotmail.co.uk
64 - 2023-03-21 2024-05-01 20 19 1520816+andremafei@users.no... erik.t.ritter@gmail.com
classify.py
in evals/elsuite/modelgraded
97 3 2023-03-16 2023-09-26 19 7 shane@openai.com bomarni@googlemail.com
oaieval.py
in evals/cli
253 6 2023-03-18 2024-09-30 19 18 120423412+andrew-openai@use... steven@openai.com
record.py
in evals
450 54 2023-03-16 2024-01-26 13 9 shane@openai.com 140545726+ianmckenzie-oai@u...
api_utils.py
in evals/utils
15 1 2023-03-18 2024-03-26 10 8 120423412+andrew-openai@use... oliver.jaffe@hotmail.co.uk
utils.py
in evals/elsuite
150 13 2023-03-22 2023-09-26 10 6 343165+rlbayes@users.norepl... bomarni@googlemail.com
data.py
in evals
148 21 2023-03-17 2024-04-02 9 10 343165+rlbayes@users.norepl... 150190178+josnyder-2@users....
eval.py
in evals
170 15 2023-03-20 2024-03-28 9 9 2406911+zhangmarvin@users.n... oliver.jaffe@hotmail.co.uk
includes.py
in evals/elsuite/basic
48 3 2023-04-11 2023-06-12 7 4 73198383+hwchung27@users.no... 120423412+andrew-openai@use...
test-modelgraded.yaml
in evals/registry/evals
81 - 2023-03-20 2023-09-18 7 3 343165+rlbayes@users.norepl... 55913678+cholotook@users.no...
utils.py
in evals/elsuite/make_me_say
34 4 2023-09-19 2023-12-20 6 5 140545726+ianmckenzie-oai@u... inwaves@users.noreply.githu...
48 3 2023-03-15 2024-01-10 6 6 jasonwei@openai.com 140545726+ianmckenzie-oai@u...
49 1 2023-03-16 2024-01-10 6 5 shane@openai.com 140545726+ianmckenzie-oai@u...
185 1 2023-03-16 2024-01-10 6 5 shane@openai.com 140545726+ianmckenzie-oai@u...
test-all.yaml
in evals/registry/eval_sets
21 - 2023-03-16 2023-04-24 5 3 shane@openai.com 343165+rlbayes@users.norepl...
test-modelgraded-battle.yaml
in evals/registry/evals
36 - 2023-03-16 2023-04-24 5 3 shane@openai.com 343165+rlbayes@users.norepl...
fuzzy_match.py
in evals/elsuite/basic
49 3 2023-04-11 2023-06-05 5 4 73198383+hwchung27@users.no... jwang47@users.noreply.githu...
base.py
in evals
51 1 2023-03-20 2023-09-26 5 4 343165+rlbayes@users.norepl... lukevanseters@gmail.com
match.py
in evals/elsuite/basic
57 3 2023-04-11 2023-09-26 5 4 73198383+hwchung27@users.no... lukevanseters@gmail.com
api.py
in evals
61 5 2023-03-28 2023-06-08 5 4 jwang47@users.noreply.githu... 131678108+wingsdrafterwork@...
oaievalset.py
in evals/cli
111 8 2023-03-21 2024-01-10 5 4 343165+rlbayes@users.norepl... 140545726+ianmckenzie-oai@u...
classify_utils.py
in evals/elsuite/modelgraded
145 8 2023-04-04 2023-09-18 5 2 343165+rlbayes@users.norepl... 97272807+sohenze@users.nore...
openai.py
in evals/completion_fns
147 10 2023-04-11 2024-03-26 5 6 jwang47@users.noreply.githu... oliver.jaffe@hotmail.co.uk
best.yaml
in evals/registry/modelgraded
10 - 2023-03-20 2023-09-18 4 2 343165+rlbayes@users.norepl... 55913678+cholotook@users.no...
base.py
in evals/elsuite/modelgraded
16 - 2023-04-04 2023-04-27 4 1 343165+rlbayes@users.norepl... 343165+rlbayes@users.norepl...
human_cli_solver.py
in evals/solvers
29 3 2023-11-09 2024-03-13 4 3 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
utils.py
in evals/solvers
37 2 2023-11-09 2024-03-28 4 2 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
json_validator.py
in evals/elsuite/basic
41 4 2023-04-14 2023-06-12 4 2 120423412+andrew-openai@use... 120423412+andrew-openai@use...
coqa-ex.yaml
in evals/registry/evals
55 - 2023-03-17 2023-04-24 4 3 343165+rlbayes@users.norepl... 343165+rlbayes@users.norepl...
base.py
in evals/prompt
64 8 2023-03-29 2024-03-13 4 3 343165+rlbayes@users.norepl... oliver.jaffe@hotmail.co.uk
hhh.py
in evals/solvers/prompts
99 - 2023-11-09 2024-03-13 4 3 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
autoeval.py
in evals/elsuite/make_me_say
116 3 2023-09-19 2023-12-05 4 4 140545726+ianmckenzie-oai@u... erik.t.ritter@gmail.com
solver.py
in evals/solvers
125 17 2023-11-09 2024-03-28 4 2 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
validators.py
in evals/registry/data/word_association/corpus_tools
151 11 2023-07-04 2024-01-10 4 4 douglas.monsky@gmail.com 140545726+ianmckenzie-oai@u...
in
4 - 2023-03-30 2023-04-12 3 3 jwang47@users.noreply.githu... 120423412+andrew-openai@use...
test-modelgraded-generated.yaml
in evals/registry/evals
9 - 2023-03-16 2023-03-27 3 2 shane@openai.com 343165+rlbayes@users.norepl...
test-modelgraded.yaml
in evals/registry/eval_sets
14 - 2023-03-17 2023-04-06 3 2 343165+rlbayes@users.norepl... 343165+rlbayes@users.norepl...
langchain_math.py
in evals/completion_fns
21 4 2023-04-11 2023-12-05 3 4 jwang47@users.noreply.githu... erik.t.ritter@gmail.com
langchain_llms.yaml
in evals/registry/completion_fns
24 - 2023-04-11 2023-12-21 3 3 73198383+hwchung27@users.no... lorenzo.pacchiardi@stats.ox...
sql.yaml
in evals/registry/modelgraded
24 - 2023-04-21 2023-05-30 3 2 mark@haym.me 133797909+jorge-openai@user...
prompts.py
in evals/elsuite/schelling_point
25 - 2023-09-19 2024-01-10 3 2 140545726+ianmckenzie-oai@u... 140545726+ianmckenzie-oai@u...
ballots.yaml
in evals/registry/evals
34 - 2023-09-19 2024-03-13 3 3 140545726+ianmckenzie-oai@u... oliver.jaffe@hotmail.co.uk
prompts.py
in evals/elsuite/ballots
44 - 2023-09-19 2024-01-10 3 2 140545726+ianmckenzie-oai@u... 140545726+ianmckenzie-oai@u...
utils.py
in evals/elsuite/make_me_pay
47 6 2023-09-19 2023-12-05 3 3 140545726+ianmckenzie-oai@u... erik.t.ritter@gmail.com
eval.py
in evals/elsuite/make_me_say
48 3 2023-09-19 2024-03-28 3 2 140545726+ianmckenzie-oai@u... oliver.jaffe@hotmail.co.uk
metrics.py
in evals
52 8 2023-06-02 2023-09-26 3 3 21045365+kjbilton@users.nor... bomarni@googlemail.com
task_description.py
in evals/elsuite/make_me_pay
57 - 2023-11-15 2024-03-13 3 2 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
make-me-pay.yaml
in evals/registry/evals
59 - 2023-09-19 2024-03-13 3 2 140545726+ianmckenzie-oai@u... oliver.jaffe@hotmail.co.uk
cot_solver.py
in evals/solvers/nested
61 7 2024-01-29 2024-03-28 3 2 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
Files With Most Contributors (Top 50)
Based on the number of unique email addresses found in commits.

See data for all files...

File# lines# unitscreatedlast modified# changes
(days)
# contributorsfirst
contributor
latest
contributor
64 - 2023-03-21 2024-05-01 20 19 1520816+andremafei@users.no... erik.t.ritter@gmail.com
oaieval.py
in evals/cli
253 6 2023-03-18 2024-09-30 19 18 120423412+andrew-openai@use... steven@openai.com
registry.py
in evals
242 27 2023-03-21 2024-03-28 22 17 343165+rlbayes@users.norepl... oliver.jaffe@hotmail.co.uk
data.py
in evals
148 21 2023-03-17 2024-04-02 9 10 343165+rlbayes@users.norepl... 150190178+josnyder-2@users....
record.py
in evals
450 54 2023-03-16 2024-01-26 13 9 shane@openai.com 140545726+ianmckenzie-oai@u...
eval.py
in evals
170 15 2023-03-20 2024-03-28 9 9 2406911+zhangmarvin@users.n... oliver.jaffe@hotmail.co.uk
api_utils.py
in evals/utils
15 1 2023-03-18 2024-03-26 10 8 120423412+andrew-openai@use... oliver.jaffe@hotmail.co.uk
classify.py
in evals/elsuite/modelgraded
97 3 2023-03-16 2023-09-26 19 7 shane@openai.com bomarni@googlemail.com
utils.py
in evals/elsuite
150 13 2023-03-22 2023-09-26 10 6 343165+rlbayes@users.norepl... bomarni@googlemail.com
48 3 2023-03-15 2024-01-10 6 6 jasonwei@openai.com 140545726+ianmckenzie-oai@u...
openai.py
in evals/completion_fns
147 10 2023-04-11 2024-03-26 5 6 jwang47@users.noreply.githu... oliver.jaffe@hotmail.co.uk
185 1 2023-03-16 2024-01-10 6 5 shane@openai.com 140545726+ianmckenzie-oai@u...
49 1 2023-03-16 2024-01-10 6 5 shane@openai.com 140545726+ianmckenzie-oai@u...
utils.py
in evals/elsuite/make_me_say
34 4 2023-09-19 2023-12-20 6 5 140545726+ianmckenzie-oai@u... inwaves@users.noreply.githu...
includes.py
in evals/elsuite/basic
48 3 2023-04-11 2023-06-12 7 4 73198383+hwchung27@users.no... 120423412+andrew-openai@use...
base.py
in evals
51 1 2023-03-20 2023-09-26 5 4 343165+rlbayes@users.norepl... lukevanseters@gmail.com
match.py
in evals/elsuite/basic
57 3 2023-04-11 2023-09-26 5 4 73198383+hwchung27@users.no... lukevanseters@gmail.com
fuzzy_match.py
in evals/elsuite/basic
49 3 2023-04-11 2023-06-05 5 4 73198383+hwchung27@users.no... jwang47@users.noreply.githu...
oaievalset.py
in evals/cli
111 8 2023-03-21 2024-01-10 5 4 343165+rlbayes@users.norepl... 140545726+ianmckenzie-oai@u...
api.py
in evals
61 5 2023-03-28 2023-06-08 5 4 jwang47@users.noreply.githu... 131678108+wingsdrafterwork@...
autoeval.py
in evals/elsuite/make_me_say
116 3 2023-09-19 2023-12-05 4 4 140545726+ianmckenzie-oai@u... erik.t.ritter@gmail.com
validators.py
in evals/registry/data/word_association/corpus_tools
151 11 2023-07-04 2024-01-10 4 4 douglas.monsky@gmail.com 140545726+ianmckenzie-oai@u...
langchain_math.py
in evals/completion_fns
21 4 2023-04-11 2023-12-05 3 4 jwang47@users.noreply.githu... erik.t.ritter@gmail.com
langchain_llm.py
in evals/completion_fns
70 7 2023-04-11 2024-01-03 3 4 jwang47@users.noreply.githu... z@hyperf.io
test-modelgraded.yaml
in evals/registry/evals
81 - 2023-03-20 2023-09-18 7 3 343165+rlbayes@users.norepl... 55913678+cholotook@users.no...
test-all.yaml
in evals/registry/eval_sets
21 - 2023-03-16 2023-04-24 5 3 shane@openai.com 343165+rlbayes@users.norepl...
test-modelgraded-battle.yaml
in evals/registry/evals
36 - 2023-03-16 2023-04-24 5 3 shane@openai.com 343165+rlbayes@users.norepl...
hhh.py
in evals/solvers/prompts
99 - 2023-11-09 2024-03-13 4 3 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
human_cli_solver.py
in evals/solvers
29 3 2023-11-09 2024-03-13 4 3 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
coqa-ex.yaml
in evals/registry/evals
55 - 2023-03-17 2023-04-24 4 3 343165+rlbayes@users.norepl... 343165+rlbayes@users.norepl...
base.py
in evals/prompt
64 8 2023-03-29 2024-03-13 4 3 343165+rlbayes@users.norepl... oliver.jaffe@hotmail.co.uk
in
4 - 2023-03-30 2023-04-12 3 3 jwang47@users.noreply.githu... 120423412+andrew-openai@use...
solvers.py
in evals/elsuite/sandbagging
152 16 2023-11-15 2024-01-29 3 3 oliver.jaffe@hotmail.co.uk junshern@users.noreply.gith...
strategy_solver.py
in evals/elsuite/bluff
88 5 2023-11-15 2024-03-28 3 3 33967107+johny-b@users.nore... oliver.jaffe@hotmail.co.uk
utils.py
in evals/elsuite/make_me_pay
47 6 2023-09-19 2023-12-05 3 3 140545726+ianmckenzie-oai@u... erik.t.ritter@gmail.com
langchain_llms.yaml
in evals/registry/completion_fns
24 - 2023-04-11 2023-12-21 3 3 73198383+hwchung27@users.no... lorenzo.pacchiardi@stats.ox...
schelling_point.yaml
in evals/registry/evals
66 - 2023-09-19 2024-03-13 3 3 140545726+ianmckenzie-oai@u... oliver.jaffe@hotmail.co.uk
ballots.yaml
in evals/registry/evals
34 - 2023-09-19 2024-03-13 3 3 140545726+ianmckenzie-oai@u... oliver.jaffe@hotmail.co.uk
metrics.py
in evals
52 8 2023-06-02 2023-09-26 3 3 21045365+kjbilton@users.nor... bomarni@googlemail.com
retrieval.py
in evals/completion_fns
68 6 2023-04-12 2024-01-10 3 3 120423412+andrew-openai@use... 140545726+ianmckenzie-oai@u...
classify_utils.py
in evals/elsuite/modelgraded
145 8 2023-04-04 2023-09-18 5 2 343165+rlbayes@users.norepl... 97272807+sohenze@users.nore...
solver.py
in evals/solvers
125 17 2023-11-09 2024-03-28 4 2 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
utils.py
in evals/solvers
37 2 2023-11-09 2024-03-28 4 2 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
json_validator.py
in evals/elsuite/basic
41 4 2023-04-14 2023-06-12 4 2 120423412+andrew-openai@use... 120423412+andrew-openai@use...
best.yaml
in evals/registry/modelgraded
10 - 2023-03-20 2023-09-18 4 2 343165+rlbayes@users.norepl... 55913678+cholotook@users.no...
cot_solver.py
in evals/solvers/nested
61 7 2024-01-29 2024-03-28 3 2 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
self_consistency_solver.py
in evals/solvers/nested
118 6 2024-01-29 2024-03-28 3 2 junshern@users.noreply.gith... oliver.jaffe@hotmail.co.uk
eval.py
in evals/elsuite/make_me_pay
126 3 2023-09-19 2024-03-13 3 2 140545726+ianmckenzie-oai@u... oliver.jaffe@hotmail.co.uk
task_description.py
in evals/elsuite/make_me_pay
57 - 2023-11-15 2024-03-13 3 2 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
eval.py
in evals/elsuite/ballots
161 3 2023-09-19 2023-12-20 3 2 140545726+ianmckenzie-oai@u... 129281094+james-aung@users....
Files With Least Contributors (Top 50)
Based on the number of unique email addresses found in commits.

See data for all files...

File# lines# unitscreatedlast modified# changes
(days)
# contributorsfirst
contributor
latest
contributor
14 1
jsonl
match.jsonl
in evals/registry/data/coqa
3 -
jsonl
samples.jsonl
in evals/registry/data/coqa
3 -
generate_samples.ipynb
in evals/registry/data/backgammon
1349 - 2023-06-22 2023-06-22 1 1 bakebrain@gmail.com bakebrain@gmail.com
actions.py
in evals/elsuite/multistep_web_tasks/webarena/browser_env
1014 50 2024-03-21 2024-03-21 1 1 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
tools.py
in evals/elsuite/bugged_tools
497 37 2024-03-19 2024-03-19 1 1 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
processors.py
in evals/elsuite/multistep_web_tasks/webarena/browser_env
495 21 2024-03-21 2024-03-21 1 1 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
make_plots.py
in evals/elsuite/error_recovery/scripts
446 20 2024-03-19 2024-03-19 1 1 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
session.py
in evals/elsuite/multistep_web_tasks
416 28 2024-03-21 2024-03-21 1 1 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
mmlu.yaml
in evals/registry/evals
399 - 2023-05-08 2023-05-08 1 1 jwang47@users.noreply.githu... jwang47@users.noreply.githu...
mmmu.yaml
in evals/registry/evals
390 - 2023-12-21 2023-12-21 1 1 erik.t.ritter@gmail.com erik.t.ritter@gmail.com
make_plots.py
in evals/elsuite/identifying_variables/scripts
325 17 2024-03-19 2024-03-19 1 1 giulio.starace@gmail.com giulio.starace@gmail.com
gen_data.py
in evals/elsuite/identifying_variables/scripts
319 13 2024-03-19 2024-03-19 1 1 giulio.starace@gmail.com giulio.starace@gmail.com
eval.py
in evals/elsuite/skill_acquisition
313 7 2024-03-19 2024-03-19 1 1 inwaves@users.noreply.githu... inwaves@users.noreply.githu...
plot_experiments.py
in evals/elsuite/hr_ml_agent_bench/scripts
307 - 2024-03-21 2024-03-21 1 1 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
low_level_actions.py
in evals/elsuite/hr_ml_agent_bench
304 13 2024-03-21 2024-03-21 1 1 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
environment.py
in evals/elsuite/hr_ml_agent_bench
283 21 2024-03-21 2024-03-21 1 1 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
constants.py
in evals/elsuite/multistep_web_tasks/webarena/browser_env
282 - 2024-03-21 2024-03-21 1 1 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
playwright_api.py
in evals/elsuite/multistep_web_tasks/webarena/core
279 32 2024-03-21 2024-03-21 1 1 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
eval_run.py
in evals/elsuite/multistep_web_tasks/webarena
277 14 2024-03-21 2024-03-21 1 1 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
evaluators.py
in evals/elsuite/multistep_web_tasks/webarena/evaluation_harness
273 19 2024-03-21 2024-03-21 1 1 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
make_plots.py
in evals/elsuite/already_said_that/scripts
263 8 2024-03-19 2024-03-19 1 1 giulio.starace@gmail.com giulio.starace@gmail.com
eval.py
in evals/elsuite/incontext_rl
246 12 2024-03-19 2024-03-19 1 1 129281094+james-aung@users.... 129281094+james-aung@users....
eval.py
in evals/elsuite/function_deduction
244 14 2024-03-19 2024-03-19 1 1 129281094+james-aung@users.... 129281094+james-aung@users....
dataset_creation.py
in evals/elsuite/cant_do_that_anymore/scripts
235 8 2024-03-19 2024-03-19 1 1 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
make_plots.py
in evals/elsuite/track_the_stat/scripts
235 9 2024-03-19 2024-03-19 1 1 giulio.starace@gmail.com giulio.starace@gmail.com
make_plots.py
in evals/elsuite/ballots/scripts
233 12 2024-03-13 2024-03-13 1 1 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
plot_experiments.py
in evals/elsuite/incontext_rl/scripts
233 8 2024-03-19 2024-03-19 1 1 129281094+james-aung@users.... 129281094+james-aung@users....
eval.py
in evals/elsuite/identifying_variables
227 14 2024-03-19 2024-03-19 1 1 giulio.starace@gmail.com giulio.starace@gmail.com
raven-matrices.yaml
in evals/registry/evals
224 - 2023-06-08 2023-06-08 1 1 44269117+ggendro@users.nore... 44269117+ggendro@users.nore...
diagonal_dataset_creation.py
in evals/elsuite/cant_do_that_anymore/scripts
216 6 2024-03-19 2024-03-19 1 1 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
corrset.py
in evals/elsuite/identifying_variables/renderers
216 11 2024-03-19 2024-03-19 1 1 giulio.starace@gmail.com giulio.starace@gmail.com
eval.py
in evals/elsuite/self_prompting
210 7 2023-11-15 2023-11-15 1 1 junshern@users.noreply.gith... junshern@users.noreply.gith...
eval.py
in evals/elsuite/bugged_tools
210 9 2024-03-19 2024-03-19 1 1 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
cards.py
in evals/elsuite/bluff/bluff
206 37 2023-11-15 2023-11-15 1 1 33967107+johny-b@users.nore... 33967107+johny-b@users.nore...
eval.py
in evals/elsuite/error_recovery
204 9 2024-03-19 2024-03-19 1 1 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
pieces.py
in evals/elsuite/cant_do_that_anymore/chess
203 9 2024-03-19 2024-03-19 1 1 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
custom_datasets.py
in evals/elsuite/steganography/scripts/dataset
197 12 2024-03-13 2024-03-13 1 1 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
make_plots.py
in evals/elsuite/function_deduction/scripts
195 4 2024-03-19 2024-03-19 1 1 129281094+james-aung@users.... 129281094+james-aung@users....
basic_browser_env.py
in evals/elsuite/multistep_web_tasks/webarena/browser_env
191 11 2024-03-21 2024-03-21 1 1 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
high_level_actions.py
in evals/elsuite/hr_ml_agent_bench
191 4 2024-03-21 2024-03-21 1 1 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
utils.py
in evals/elsuite/multistep_web_tasks/webarena/core
188 7 2024-03-21 2024-03-21 1 1 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
openai_assistants_solver.py
in evals/solvers/providers/openai
186 11 2024-03-28 2024-03-28 1 1 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
openai_solver.py
in evals/solvers/providers/openai
181 18 2024-03-28 2024-03-28 1 1 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
solver_tools_convo.py
in evals/elsuite
181 11 2024-03-19 2024-03-19 1 1 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
ukraine-gec.yaml
in evals/registry/evals
180 - 2023-06-02 2023-06-02 1 1 roman.mashta@gmail.com roman.mashta@gmail.com
utils.py
in evals/elsuite/cant_do_that_anymore
178 12 2024-03-19 2024-03-19 1 1 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
strong_solver.py
in evals/elsuite/multistep_web_tasks/solvers/strong_solver
173 12 2024-03-21 2024-03-21 1 1 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
autoeval.py
in evals/elsuite/hr_ml_agent_bench
172 2 2024-03-21 2024-03-21 1 1 danesherbs@users.noreply.gi... danesherbs@users.noreply.gi...
eval.py
in evals/elsuite/cant_do_that_anymore
170 4 2024-03-19 2024-03-19 1 1 oliver.jaffe@hotmail.co.uk oliver.jaffe@hotmail.co.uk
Correlations

File Size vs. Number of Changes: 1496 points

evals/cli/oaieval.py x: 253 lines of code y: 19 # changes evals/registry/data/imo_exact_answers/samples.jsonl x: 3 lines of code y: 1 # changes evals/registry/evals/imo_exact_answers.yaml x: 8 lines of code y: 1 # changes pyproject.toml x: 64 lines of code y: 20 # changes evals/data.py x: 148 lines of code y: 9 # changes evals/completion_fns/solver_completion_fn.py x: 47 lines of code y: 1 # changes evals/elsuite/bluff/strategy_solver.py x: 88 lines of code y: 3 # changes evals/elsuite/bugged_tools/task_description.py x: 9 lines of code y: 2 # changes evals/elsuite/function_deduction/prompts.py x: 6 lines of code y: 2 # changes evals/elsuite/function_deduction/solvers.py x: 140 lines of code y: 2 # changes evals/elsuite/hr_ml_agent_bench/solvers/baseline.py x: 90 lines of code y: 2 # changes evals/elsuite/make_me_say/eval.py x: 48 lines of code y: 3 # changes evals/eval.py x: 170 lines of code y: 9 # changes evals/registry.py x: 242 lines of code y: 22 # changes evals/registry/solvers/already_said_that.yaml x: 75 lines of code y: 2 # changes evals/registry/solvers/bluff.yaml x: 80 lines of code y: 2 # changes evals/registry/solvers/cant_do_that_anymore.yaml x: 16 lines of code y: 2 # changes evals/registry/solvers/defaults.yaml x: 294 lines of code y: 3 # changes evals/registry/solvers/error_recovery.yaml x: 33 lines of code y: 2 # changes evals/registry/solvers/function_deduction.yaml x: 174 lines of code y: 2 # changes evals/registry/solvers/hr-ml-agent-bench.yaml x: 37 lines of code y: 2 # changes evals/registry/solvers/incontext_rl.yaml x: 24 lines of code y: 2 # changes evals/registry/solvers/make-me-pay.yaml x: 101 lines of code y: 2 # changes evals/registry/solvers/self_prompting.yaml x: 96 lines of code y: 2 # changes evals/registry/solvers/skill_acquisition.yaml x: 267 lines of code y: 2 # changes evals/registry/solvers/theory_of_mind.yaml x: 394 lines of code y: 3 # changes evals/registry/solvers/together.yaml x: 85 lines of code y: 2 # changes evals/solvers/memory.py x: 50 lines of code y: 1 # changes evals/solvers/nested/cot_solver.py x: 61 lines of code y: 3 # changes evals/solvers/nested/self_consistency_solver.py x: 118 lines of code y: 3 # changes evals/solvers/prompts/cot.py x: 4 lines of code y: 2 # changes evals/solvers/providers/openai/openai_assistants_solver.py x: 186 lines of code y: 1 # changes evals/solvers/providers/openai/openai_solver.py x: 181 lines of code y: 1 # changes evals/solvers/providers/together/together_solver.py x: 68 lines of code y: 1 # changes evals/solvers/solver.py x: 125 lines of code y: 4 # changes evals/solvers/utils.py x: 37 lines of code y: 4 # changes evals/registry/solvers/gemini.yaml x: 15 lines of code y: 1 # changes evals/solvers/providers/google/gemini_solver.py x: 157 lines of code y: 1 # changes evals/completion_fns/openai.py x: 147 lines of code y: 5 # changes evals/utils/api_utils.py x: 15 lines of code y: 10 # changes evals/registry/solvers/anthropic.yaml x: 90 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/__init__.py x: 1 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/actions.py x: 37 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/auto_marking.py x: 53 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/autoeval.py x: 172 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/ant/baselines/human.py x: 36 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/ant/baselines/naive.py x: 28 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/ant/scripts/grade.py x: 43 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/bipedal_walker/baselines/human.py x: 57 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/bipedal_walker/scripts/grade.py x: 31 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/cartpole/scripts/grade.py x: 40 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/cifar10/env/train.py x: 105 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/cifar10/scripts/prepare.py x: 5 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/feedback/scripts/prepare.py x: 24 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/house_price/scripts/grade.py x: 33 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/house_price/scripts/prepare.py x: 20 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/humanoid/scripts/grade.py x: 45 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/ogbn_arxiv/env/train.py x: 126 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/parkinsons_disease/env/train.py x: 124 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/parkinsons_disease/scripts/prepare.py x: 98 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/spaceship_titanic/scripts/grade.py x: 26 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/vectorization/env/train.py x: 118 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/vectorization/scripts/grade.py x: 64 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/benchmarks/vectorization/scripts/human_baseline.py x: 83 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/environment.py x: 283 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/eval.py x: 91 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/high_level_actions.py x: 191 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/low_level_actions.py x: 304 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/scripts/plot_experiments.py x: 307 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/scripts/run_experiments.py x: 60 lines of code y: 1 # changes evals/elsuite/hr_ml_agent_bench/utils.py x: 112 lines of code y: 1 # changes evals/registry/evals/hr-ml-agent-bench.yaml x: 137 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/constants.py x: 70 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/docker/flask-playwright/app.py x: 165 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/docker/homepage/templates/index.html x: 108 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/reproducibility/make_plots.py x: 94 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/session.py x: 416 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/solvers/strong_solver/strong_solver.py x: 173 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/solvers/webarena_solvers/webarena_prompts.py x: 12 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/bash_env/basic_bash_env.py x: 163 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/browser_env/actions.py x: 1014 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/browser_env/auto_login.py x: 100 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/browser_env/browser_utils.py x: 78 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/browser_env/constants.py x: 282 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/browser_env/helper_functions.py x: 129 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/browser_env/processors.py x: 495 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/core/env.py x: 75 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/core/playwright_api.py x: 279 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/core/utils.py x: 188 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/eval_run.py x: 277 lines of code y: 1 # changes evals/elsuite/multistep_web_tasks/webarena/evaluation_harness/evaluators.py x: 273 lines of code y: 1 # changes evals/registry/solvers/multistep_web_tasks.yaml x: 56 lines of code y: 1 # changes evals/elsuite/incontext_rl/eval.py x: 246 lines of code y: 1 # changes evals/elsuite/incontext_rl/scripts/plot_experiments.py x: 233 lines of code y: 1 # changes evals/elsuite/function_deduction/eval.py x: 244 lines of code y: 1 # changes evals/elsuite/function_deduction/scripts/make_plots.py x: 195 lines of code y: 1 # changes evals/elsuite/identifying_variables/eval.py x: 227 lines of code y: 1 # changes evals/elsuite/identifying_variables/graph_utils.py x: 130 lines of code y: 1 # changes evals/elsuite/identifying_variables/renderers/corrset.py x: 216 lines of code y: 1 # changes evals/elsuite/identifying_variables/scripts/gen_data.py x: 319 lines of code y: 1 # changes evals/elsuite/identifying_variables/scripts/make_plots.py x: 325 lines of code y: 1 # changes evals/elsuite/identifying_variables/structs.py x: 18 lines of code y: 1 # changes evals/elsuite/identifying_variables/utils.py x: 72 lines of code y: 1 # changes evals/elsuite/track_the_stat/scripts/make_plots.py x: 235 lines of code y: 1 # changes evals/elsuite/already_said_that/distractors.py x: 81 lines of code y: 1 # changes evals/elsuite/already_said_that/scripts/make_plots.py x: 263 lines of code y: 1 # changes evals/elsuite/already_said_that/utils.py x: 104 lines of code y: 1 # changes evals/utils/log_utils.py x: 53 lines of code y: 2 # changes evals/elsuite/twenty_questions/eval.py x: 169 lines of code y: 1 # changes evals/elsuite/skill_acquisition/eval.py x: 313 lines of code y: 1 # changes evals/elsuite/skill_acquisition/scraping/scrape_distractor_articles.py x: 76 lines of code y: 1 # changes evals/elsuite/skill_acquisition/solvers.py x: 13 lines of code y: 1 # changes evals/elsuite/skill_acquisition/utils.py x: 115 lines of code y: 1 # changes evals/elsuite/error_recovery/eval.py x: 204 lines of code y: 1 # changes evals/elsuite/error_recovery/scripts/make_plots.py x: 446 lines of code y: 1 # changes evals/elsuite/cant_do_that_anymore/chess/move_variants.py x: 113 lines of code y: 1 # changes evals/elsuite/cant_do_that_anymore/chess/notation.py x: 61 lines of code y: 1 # changes evals/elsuite/cant_do_that_anymore/utils.py x: 178 lines of code y: 1 # changes evals/elsuite/bugged_tools/bugged_tools.py x: 132 lines of code y: 1 # changes evals/elsuite/bugged_tools/eval.py x: 210 lines of code y: 1 # changes evals/elsuite/bugged_tools/tools.py x: 497 lines of code y: 1 # changes evals/elsuite/basic/match_with_solvers.py x: 65 lines of code y: 2 # changes evals/elsuite/bluff/solver_player.py x: 82 lines of code y: 2 # changes evals/elsuite/make_me_pay/eval.py x: 126 lines of code y: 3 # changes evals/elsuite/make_me_pay/task_description.py x: 57 lines of code y: 3 # changes evals/elsuite/make_me_say/core.py x: 223 lines of code y: 3 # changes evals/prompt/base.py x: 64 lines of code y: 4 # changes evals/registry/evals/ballots.yaml x: 34 lines of code y: 3 # changes evals/registry/evals/make-me-pay.yaml x: 59 lines of code y: 3 # changes evals/registry/evals/schelling_point.yaml x: 66 lines of code y: 3 # changes evals/registry/evals/self_prompting.yaml x: 19 lines of code y: 2 # changes evals/registry/evals/theory_of_mind.yaml x: 48 lines of code y: 2 # changes evals/solvers/human_cli_solver.py x: 29 lines of code y: 4 # changes evals/solvers/nested/fewshot_solver.py x: 91 lines of code y: 2 # changes evals/solvers/prompts/hhh.py x: 99 lines of code y: 4 # changes evals/elsuite/sandbagging/solvers.py x: 152 lines of code y: 3 # changes evals/elsuite/self_prompting/solvers/baselines.py x: 45 lines of code y: 2 # changes evals/elsuite/self_prompting/solvers/custom_cot_solver.py x: 57 lines of code y: 2 # changes evals/record.py x: 450 lines of code y: 13 # changes evals/cli/oaievalset.py x: 111 lines of code y: 5 # changes evals/elsuite/ballots/prompts.py x: 44 lines of code y: 3 # changes evals/elsuite/sandbagging/defaults.py x: 12 lines of code y: 2 # changes evals/elsuite/sandbagging/sandbagging_eval.py x: 68 lines of code y: 2 # changes evals/elsuite/schelling_point/prompts.py x: 25 lines of code y: 3 # changes evals/elsuite/schelling_point/utils.py x: 61 lines of code y: 2 # changes evals/elsuite/steganography/eval.py x: 69 lines of code y: 2 # changes evals/registry/data/word_association/corpus_tools/validators.py x: 151 lines of code y: 4 # changes scripts/battle_generator.py x: 49 lines of code y: 6 # changes scripts/modelgraded_generator.py x: 185 lines of code y: 6 # changes scripts/pattern_identification_generator.py x: 48 lines of code y: 6 # changes evals/elsuite/mmmu/eval.py x: 159 lines of code y: 3 # changes evals/completion_fns/langchain_llm.py x: 70 lines of code y: 3 # changes evals/registry/completion_fns/langchain_llms.yaml x: 24 lines of code y: 3 # changes evals/registry/evals/mmmu.yaml x: 390 lines of code y: 1 # changes evals/elsuite/make_me_say/utils.py x: 34 lines of code y: 6 # changes evals/elsuite/ballots/eval.py x: 161 lines of code y: 3 # changes evals/elsuite/schelling_point/eval.py x: 73 lines of code y: 2 # changes evals/elsuite/bluff/eval.py x: 164 lines of code y: 2 # changes evals/completion_fns/langchain_math.py x: 21 lines of code y: 3 # changes evals/elsuite/make_me_say/autoeval.py x: 116 lines of code y: 4 # changes evals/elsuite/bluff/bluff/cards.py x: 206 lines of code y: 1 # changes evals/elsuite/make_me_pay/makemepay.py x: 222 lines of code y: 2 # changes evals/elsuite/lambada.py x: 39 lines of code y: 2 # changes evals/elsuite/modelgraded/classify.py x: 97 lines of code y: 19 # changes evals/elsuite/utils.py x: 150 lines of code y: 10 # changes evals/metrics.py x: 52 lines of code y: 3 # changes evals/registry/data/canto_wu_pronunciation/csv_to_json.py x: 55 lines of code y: 2 # changes evals/registry/data/german-part-of-speech/parsePosDe.py x: 172 lines of code y: 2 # changes evals/registry/data/mapping_to_matricies/data_generator.py x: 30 lines of code y: 2 # changes evals/registry/data/mazes/nxn_maze_eval_generator.py x: 162 lines of code y: 2 # changes evals/registry/data/simple_physics_engine/wave_function_collapse.py x: 157 lines of code y: 2 # changes evals/registry/data/solve-for-variable/tools/solve.py x: 231 lines of code y: 2 # changes evals/registry/data/solve-for-variable/tools/tester.py x: 94 lines of code y: 2 # changes evals/registry/data/word_association/corpus_tools/pipelines.py x: 14 lines of code y: 2 # changes evals/registry/data/word_association/corpus_tools/processor.py x: 36 lines of code y: 2 # changes evals/registry/data/word_association/corpus_tools/sample_generators.py x: 154 lines of code y: 2 # changes evals/base.py x: 51 lines of code y: 5 # changes evals/elsuite/basic/match.py x: 57 lines of code y: 5 # changes evals/registry/evals/test-modelgraded.yaml x: 81 lines of code y: 7 # changes evals/registry/modelgraded/best.yaml x: 10 lines of code y: 4 # changes evals/elsuite/modelgraded/classify_utils.py x: 145 lines of code y: 5 # changes evals/registry/data/backgammon/generate_samples.ipynb x: 1349 lines of code y: 1 # changes evals/elsuite/basic/includes.py x: 48 lines of code y: 7 # changes evals/elsuite/basic/json_validator.py x: 41 lines of code y: 4 # changes evals/api.py x: 61 lines of code y: 5 # changes evals/registry/evals/raven-matrices.yaml x: 224 lines of code y: 1 # changes evals/registry/evals/mmlu.yaml x: 399 lines of code y: 1 # changes evals/elsuite/modelgraded/base.py x: 16 lines of code y: 4 # changes evals/registry/eval_sets/test-all.yaml x: 21 lines of code y: 5 # changes evals/registry/evals/coqa-ex.yaml x: 55 lines of code y: 4 # changes evals/registry/evals/test-modelgraded-battle.yaml x: 36 lines of code y: 5 # changes evals/registry/modelgraded/closedqa.yaml x: 21 lines of code y: 2 # changes MANIFEST.in x: 4 lines of code y: 3 # changes evals/registry/eval_sets/test-modelgraded.yaml x: 14 lines of code y: 3 # changes evals/registry/evals/stock-options.yaml x: 96 lines of code y: 1 # changes evals/registry/evals/test-modelgraded-generated.yaml x: 9 lines of code y: 3 # changes
22.0
# changes
  min: 1.0
  average: 1.25
  25th percentile: 1.0
  median: 1.0
  75th percentile: 1.0
  max: 22.0
0 1349.0
lines of code
min: 1.0 | average: 28.01 | 25th percentile: 3.0 | median: 3.0 | 75th percentile: 16.0 | max: 1349.0

Number of Contributors vs. Number of Changes: 1496 points

evals/cli/oaieval.py x: 18 # contributors y: 19 # changes evals/registry/data/imo_exact_answers/samples.jsonl x: 1 # contributors y: 1 # changes pyproject.toml x: 19 # contributors y: 20 # changes evals/data.py x: 10 # contributors y: 9 # changes evals/elsuite/bluff/strategy_solver.py x: 3 # contributors y: 3 # changes evals/elsuite/bugged_tools/task_description.py x: 1 # contributors y: 2 # changes evals/elsuite/function_deduction/prompts.py x: 2 # contributors y: 2 # changes evals/elsuite/make_me_say/eval.py x: 2 # contributors y: 3 # changes evals/eval.py x: 9 # contributors y: 9 # changes evals/registry.py x: 17 # contributors y: 22 # changes evals/solvers/solver.py x: 2 # contributors y: 4 # changes evals/completion_fns/openai.py x: 6 # contributors y: 5 # changes evals/utils/api_utils.py x: 8 # contributors y: 10 # changes evals/prompt/base.py x: 3 # contributors y: 4 # changes evals/record.py x: 9 # contributors y: 13 # changes evals/cli/oaievalset.py x: 4 # contributors y: 5 # changes evals/registry/data/word_association/corpus_tools/validators.py x: 4 # contributors y: 4 # changes scripts/battle_generator.py x: 5 # contributors y: 6 # changes scripts/pattern_identification_generator.py x: 6 # contributors y: 6 # changes evals/elsuite/mmmu/eval.py x: 1 # contributors y: 3 # changes evals/completion_fns/langchain_llm.py x: 4 # contributors y: 3 # changes evals/elsuite/modelgraded/classify.py x: 7 # contributors y: 19 # changes evals/elsuite/utils.py x: 6 # contributors y: 10 # changes evals/registry/evals/test-modelgraded.yaml x: 3 # contributors y: 7 # changes evals/elsuite/modelgraded/classify_utils.py x: 2 # contributors y: 5 # changes evals/elsuite/basic/includes.py x: 4 # contributors y: 7 # changes evals/elsuite/modelgraded/base.py x: 1 # contributors y: 4 # changes evals/registry/eval_sets/test-all.yaml x: 3 # contributors y: 5 # changes evals/registry/data/logiqa/logiqa.jsonl x: 2 # contributors y: 1 # changes
22.0
# changes
  min: 1.0
  average: 1.25
  25th percentile: 1.0
  median: 1.0
  75th percentile: 1.0
  max: 22.0
0 19.0
# contributors
min: 1.0 | average: 1.19 | 25th percentile: 1.0 | median: 1.0 | 75th percentile: 1.0 | max: 19.0

Number of Contributors vs. File Size: 1496 points

evals/cli/oaieval.py x: 18 # contributors y: 253 lines of code evals/registry/data/imo_exact_answers/samples.jsonl x: 1 # contributors y: 3 lines of code evals/registry/evals/imo_exact_answers.yaml x: 1 # contributors y: 8 lines of code pyproject.toml x: 19 # contributors y: 64 lines of code evals/data.py x: 10 # contributors y: 148 lines of code evals/completion_fns/solver_completion_fn.py x: 1 # contributors y: 47 lines of code evals/elsuite/bluff/strategy_solver.py x: 3 # contributors y: 88 lines of code evals/elsuite/function_deduction/prompts.py x: 2 # contributors y: 6 lines of code evals/elsuite/function_deduction/solvers.py x: 2 # contributors y: 140 lines of code evals/elsuite/hr_ml_agent_bench/solvers/baseline.py x: 2 # contributors y: 90 lines of code evals/elsuite/make_me_say/eval.py x: 2 # contributors y: 48 lines of code evals/eval.py x: 9 # contributors y: 170 lines of code evals/registry.py x: 17 # contributors y: 242 lines of code evals/registry/solvers/already_said_that.yaml x: 2 # contributors y: 75 lines of code evals/registry/solvers/bluff.yaml x: 2 # contributors y: 80 lines of code evals/registry/solvers/cant_do_that_anymore.yaml x: 1 # contributors y: 16 lines of code evals/registry/solvers/defaults.yaml x: 2 # contributors y: 294 lines of code evals/registry/solvers/error_recovery.yaml x: 1 # contributors y: 33 lines of code evals/registry/solvers/function_deduction.yaml x: 2 # contributors y: 174 lines of code evals/registry/solvers/hr-ml-agent-bench.yaml x: 2 # contributors y: 37 lines of code evals/registry/solvers/incontext_rl.yaml x: 2 # contributors y: 24 lines of code evals/registry/solvers/make-me-pay.yaml x: 2 # contributors y: 101 lines of code evals/registry/solvers/self_prompting.yaml x: 2 # contributors y: 96 lines of code evals/registry/solvers/skill_acquisition.yaml x: 2 # contributors y: 267 lines of code evals/registry/solvers/theory_of_mind.yaml x: 2 # contributors y: 394 lines of code evals/solvers/memory.py x: 1 # contributors y: 50 lines of code evals/solvers/nested/cot_solver.py x: 2 # contributors y: 61 lines of code evals/solvers/nested/self_consistency_solver.py x: 2 # contributors y: 118 lines of code evals/solvers/prompts/cot.py x: 2 # contributors y: 4 lines of code evals/solvers/providers/openai/openai_assistants_solver.py x: 1 # contributors y: 186 lines of code evals/solvers/providers/openai/openai_solver.py x: 1 # contributors y: 181 lines of code evals/solvers/providers/together/together_solver.py x: 1 # contributors y: 68 lines of code evals/solvers/solver.py x: 2 # contributors y: 125 lines of code evals/registry/solvers/gemini.yaml x: 1 # contributors y: 15 lines of code evals/solvers/providers/google/gemini_solver.py x: 1 # contributors y: 157 lines of code evals/completion_fns/openai.py x: 6 # contributors y: 147 lines of code evals/utils/api_utils.py x: 8 # contributors y: 15 lines of code evals/registry/solvers/anthropic.yaml x: 1 # contributors y: 90 lines of code evals/elsuite/hr_ml_agent_bench/autoeval.py x: 1 # contributors y: 172 lines of code evals/elsuite/hr_ml_agent_bench/benchmarks/ant/baselines/naive.py x: 1 # contributors y: 28 lines of code evals/elsuite/hr_ml_agent_bench/benchmarks/bipedal_walker/baselines/human.py x: 1 # contributors y: 57 lines of code evals/elsuite/hr_ml_agent_bench/benchmarks/cartpole/scripts/grade.py x: 1 # contributors y: 40 lines of code evals/elsuite/hr_ml_agent_bench/benchmarks/cifar10/env/train.py x: 1 # contributors y: 105 lines of code evals/elsuite/hr_ml_agent_bench/benchmarks/feedback/scripts/prepare.py x: 1 # contributors y: 24 lines of code evals/elsuite/hr_ml_agent_bench/benchmarks/ogbn_arxiv/env/train.py x: 1 # contributors y: 126 lines of code evals/elsuite/hr_ml_agent_bench/benchmarks/parkinsons_disease/scripts/prepare.py x: 1 # contributors y: 98 lines of code evals/elsuite/hr_ml_agent_bench/benchmarks/vectorization/env/train.py x: 1 # contributors y: 118 lines of code evals/elsuite/hr_ml_agent_bench/benchmarks/vectorization/scripts/human_baseline.py x: 1 # contributors y: 83 lines of code evals/elsuite/hr_ml_agent_bench/environment.py x: 1 # contributors y: 283 lines of code evals/elsuite/hr_ml_agent_bench/eval.py x: 1 # contributors y: 91 lines of code evals/elsuite/hr_ml_agent_bench/low_level_actions.py x: 1 # contributors y: 304 lines of code evals/elsuite/hr_ml_agent_bench/scripts/run_experiments.py x: 1 # contributors y: 60 lines of code evals/elsuite/hr_ml_agent_bench/utils.py x: 1 # contributors y: 112 lines of code evals/registry/evals/hr-ml-agent-bench.yaml x: 1 # contributors y: 137 lines of code evals/elsuite/multistep_web_tasks/constants.py x: 1 # contributors y: 70 lines of code evals/elsuite/multistep_web_tasks/docker/flask-playwright/app.py x: 1 # contributors y: 165 lines of code evals/elsuite/multistep_web_tasks/docker/homepage/templates/index.html x: 1 # contributors y: 108 lines of code evals/elsuite/multistep_web_tasks/session.py x: 1 # contributors y: 416 lines of code evals/elsuite/multistep_web_tasks/webarena/bash_env/basic_bash_env.py x: 1 # contributors y: 163 lines of code evals/elsuite/multistep_web_tasks/webarena/browser_env/actions.py x: 1 # contributors y: 1014 lines of code evals/elsuite/multistep_web_tasks/webarena/browser_env/browser_utils.py x: 1 # contributors y: 78 lines of code evals/elsuite/multistep_web_tasks/webarena/browser_env/helper_functions.py x: 1 # contributors y: 129 lines of code evals/elsuite/multistep_web_tasks/webarena/browser_env/processors.py x: 1 # contributors y: 495 lines of code evals/elsuite/multistep_web_tasks/webarena/core/playwright_api.py x: 1 # contributors y: 279 lines of code evals/elsuite/multistep_web_tasks/webarena/evaluation_harness/evaluators.py x: 1 # contributors y: 273 lines of code evals/elsuite/incontext_rl/eval.py x: 1 # contributors y: 246 lines of code evals/elsuite/incontext_rl/scripts/plot_experiments.py x: 1 # contributors y: 233 lines of code evals/elsuite/function_deduction/eval.py x: 1 # contributors y: 244 lines of code evals/elsuite/function_deduction/scripts/make_plots.py x: 1 # contributors y: 195 lines of code evals/elsuite/identifying_variables/eval.py x: 1 # contributors y: 227 lines of code evals/elsuite/identifying_variables/renderers/corrset.py x: 1 # contributors y: 216 lines of code evals/elsuite/identifying_variables/scripts/gen_data.py x: 1 # contributors y: 319 lines of code evals/elsuite/identifying_variables/scripts/make_plots.py x: 1 # contributors y: 325 lines of code evals/elsuite/track_the_stat/scripts/make_plots.py x: 1 # contributors y: 235 lines of code evals/elsuite/already_said_that/scripts/make_plots.py x: 1 # contributors y: 263 lines of code evals/elsuite/skill_acquisition/eval.py x: 1 # contributors y: 313 lines of code evals/elsuite/error_recovery/eval.py x: 1 # contributors y: 204 lines of code evals/elsuite/error_recovery/scripts/make_plots.py x: 1 # contributors y: 446 lines of code evals/elsuite/cant_do_that_anymore/utils.py x: 1 # contributors y: 178 lines of code evals/elsuite/bugged_tools/eval.py x: 1 # contributors y: 210 lines of code evals/elsuite/basic/match_with_solvers.py x: 2 # contributors y: 65 lines of code evals/elsuite/make_me_pay/task_description.py x: 2 # contributors y: 57 lines of code evals/elsuite/make_me_say/core.py x: 2 # contributors y: 223 lines of code evals/elsuite/steganography/scripts/dataset/custom_datasets.py x: 1 # contributors y: 197 lines of code evals/prompt/base.py x: 3 # contributors y: 64 lines of code evals/registry/evals/ballots.yaml x: 3 # contributors y: 34 lines of code evals/registry/evals/self_prompting.yaml x: 2 # contributors y: 19 lines of code evals/solvers/human_cli_solver.py x: 3 # contributors y: 29 lines of code evals/solvers/nested/fewshot_solver.py x: 2 # contributors y: 91 lines of code evals/solvers/prompts/hhh.py x: 3 # contributors y: 99 lines of code evals/elsuite/sandbagging/solvers.py x: 3 # contributors y: 152 lines of code evals/record.py x: 9 # contributors y: 450 lines of code evals/cli/oaievalset.py x: 4 # contributors y: 111 lines of code evals/elsuite/ballots/prompts.py x: 2 # contributors y: 44 lines of code evals/elsuite/sandbagging/defaults.py x: 2 # contributors y: 12 lines of code evals/registry/data/word_association/corpus_tools/validators.py x: 4 # contributors y: 151 lines of code scripts/battle_generator.py x: 5 # contributors y: 49 lines of code scripts/modelgraded_generator.py x: 5 # contributors y: 185 lines of code scripts/pattern_identification_generator.py x: 6 # contributors y: 48 lines of code evals/completion_fns/langchain_llm.py x: 4 # contributors y: 70 lines of code evals/registry/completion_fns/langchain_llms.yaml x: 3 # contributors y: 24 lines of code evals/registry/evals/mmmu.yaml x: 1 # contributors y: 390 lines of code evals/elsuite/make_me_say/utils.py x: 5 # contributors y: 34 lines of code evals/elsuite/ballots/eval.py x: 2 # contributors y: 161 lines of code evals/elsuite/schelling_point/eval.py x: 2 # contributors y: 73 lines of code evals/completion_fns/langchain_math.py x: 4 # contributors y: 21 lines of code evals/elsuite/make_me_pay/utils.py x: 3 # contributors y: 47 lines of code evals/elsuite/make_me_say/autoeval.py x: 4 # contributors y: 116 lines of code evals/elsuite/lambada.py x: 2 # contributors y: 39 lines of code evals/elsuite/modelgraded/classify.py x: 7 # contributors y: 97 lines of code evals/elsuite/utils.py x: 6 # contributors y: 150 lines of code evals/metrics.py x: 3 # contributors y: 52 lines of code evals/registry/data/mapping_to_matricies/data_generator.py x: 2 # contributors y: 30 lines of code evals/registry/data/simple_physics_engine/wave_function_collapse.py x: 2 # contributors y: 157 lines of code evals/registry/data/solve-for-variable/tools/solve.py x: 2 # contributors y: 231 lines of code evals/registry/data/word_association/corpus_tools/sample_generators.py x: 2 # contributors y: 154 lines of code evals/base.py x: 4 # contributors y: 51 lines of code evals/elsuite/basic/match.py x: 4 # contributors y: 57 lines of code evals/registry/evals/test-modelgraded.yaml x: 3 # contributors y: 81 lines of code evals/elsuite/modelgraded/classify_utils.py x: 2 # contributors y: 145 lines of code evals/registry/data/backgammon/generate_samples.ipynb x: 1 # contributors y: 1349 lines of code evals/api.py x: 4 # contributors y: 61 lines of code evals/registry/evals/mmlu.yaml x: 1 # contributors y: 399 lines of code evals/registry/eval_sets/test-all.yaml x: 3 # contributors y: 21 lines of code evals/registry/evals/coqa-ex.yaml x: 3 # contributors y: 55 lines of code MANIFEST.in x: 3 # contributors y: 4 lines of code
1349.0
lines of code
  min: 1.0
  average: 28.01
  25th percentile: 3.0
  median: 3.0
  75th percentile: 16.0
  max: 1349.0
0 19.0
# contributors
min: 1.0 | average: 1.19 | 25th percentile: 1.0 | median: 1.0 | 75th percentile: 1.0 | max: 19.0