huggingface / open-r1 Duplication

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

14% duplication:

3,424 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
500 duplicated lines

32 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Duplication Between Components (50+ lines)

Download: SVG DOT (open online Graphviz editor)

From Component --> To Component	Duplicated Lines	File Pairs	Details
scripts (6%) --> src (1%)	75	3 file pairs	details...

Open 2D force graph... Open 3D force graph...

Show more details on duplication between components...

Longest Duplicates

The list of 32 longest duplicates.

See data for all 32 duplicates...

Size	#	Folders	Files	Lines	Code
35	x 2	src/open_r1 src/open_r1	grpo.py sft.py	121:178 (38%) 109:166 (46%)	view
31	x 2	recipes/Qwen2.5-1.5B-Instruct/grpo recipes/Qwen2.5-1.5B-Instruct/grpo	config_demo_code.yaml config_demo_code_ioi.yaml	10:42 (63%) 10:42 (58%)	view
20	x 2	src/open_r1 src/open_r1	rewards.py rewards.py	159:179 (4%) 240:260 (4%)	view
16	x 2	scripts/pass_rate_filtering src/open_r1	compute_pass_rate.py sft.py	48:69 (13%) 55:75 (21%)	view
14	x 2	src/open_r1 src/open_r1	grpo.py sft.py	32:52 (15%) 52:71 (18%)	view
13	x 2	recipes/accelerate_configs recipes/accelerate_configs	zero2.yaml zero3.yaml	9:21 (61%) 10:22 (59%)	view
13	x 2	src/open_r1 src/open_r1	configs.py configs.py	155:169 (5%) 194:208 (5%)	view
13	x 2	scripts/pass_rate_filtering src/open_r1	compute_pass_rate.py grpo.py	48:65 (10%) 35:52 (14%)	view
12	x 2	src/open_r1 src/open_r1	grpo.py sft.py	59:79 (13%) 73:91 (15%)	view
11	x 2	recipes/accelerate_configs recipes/accelerate_configs	ddp.yaml zero3.yaml	6:16 (68%) 12:22 (50%)	view
11	x 2	recipes/accelerate_configs recipes/accelerate_configs	fsdp.yaml zero3.yaml	17:27 (40%) 12:22 (50%)	view
11	x 2	recipes/accelerate_configs recipes/accelerate_configs	ddp.yaml fsdp.yaml	6:16 (68%) 17:27 (40%)	view
11	x 2	recipes/accelerate_configs recipes/accelerate_configs	ddp.yaml zero2.yaml	6:16 (68%) 11:21 (52%)	view
11	x 2	recipes/accelerate_configs recipes/accelerate_configs	fsdp.yaml zero2.yaml	17:27 (40%) 11:21 (52%)	view
9	x 2	src/open_r1 src/open_r1	rewards.py rewards.py	51:59 (2%) 162:170 (2%)	view
9	x 2	src/open_r1 src/open_r1	configs.py configs.py	130:138 (4%) 175:183 (4%)	view
9	x 2	src/open_r1 src/open_r1	rewards.py rewards.py	51:59 (2%) 243:251 (2%)	view
8	x 2	src/open_r1 src/open_r1	rewards.py rewards.py	61:69 (1%) 253:260 (1%)	view
8	x 2	scripts/pass_rate_filtering src/open_r1	compute_pass_rate.py grpo.py	83:94 (6%) 92:103 (8%)	view
8	x 2	scripts scripts	e2b_router.py morph_router.py	130:152 (14%) 135:157 (10%)	view
8	x 2	recipes/OlympicCoder-32B/sft recipes/OlympicCoder-7B/sft	config_v00.00.yaml config_v00.00.yaml	25:32 (18%) 24:31 (20%)	view
8	x 2	src/open_r1 src/open_r1	rewards.py rewards.py	61:69 (1%) 172:179 (1%)	view
7	x 2	scripts src/open_r1/utils	morph_router.py code_providers.py	130:136 (8%) 330:336 (3%)	view
7	x 2	recipes/Qwen2.5-1.5B-Instruct/grpo recipes/Qwen2.5-1.5B-Instruct/grpo	config_demo.yaml config_demo_code_ioi.yaml	13:19 (14%) 14:20 (13%)	view
7	x 2	scripts scripts	e2b_router.py morph_router.py	96:104 (12%) 79:87 (8%)	view
7	x 2	recipes/Qwen2.5-1.5B-Instruct/grpo recipes/Qwen2.5-1.5B-Instruct/grpo	config_demo.yaml config_demo_code.yaml	13:19 (14%) 14:20 (14%)	view
7	x 2	src/open_r1 src/open_r1	rewards.py rewards.py	150:156 (1%) 232:238 (1%)	view
6	x 2	recipes/accelerate_configs recipes/accelerate_configs	zero2.yaml zero3.yaml	1:6 (28%) 1:6 (27%)	view
6	x 2	src/open_r1 src/open_r1	rewards.py rewards.py	42:47 (1%) 229:235 (1%)	view
6	x 2	recipes/Qwen2.5-1.5B-Instruct/grpo recipes/Qwen2.5-1.5B-Instruct/grpo	config_demo_code.yaml config_demo_code_ioi.yaml	49:54 (12%) 43:48 (11%)	view
6	x 2	src/open_r1 src/open_r1	rewards.py rewards.py	486:491 (1%) 512:517 (1%)	view
6	x 2	src/open_r1 src/open_r1	rewards.py rewards.py	410:415 (1%) 466:471 (1%)	view