huggingface / alignment-handbook Duplication

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

49% duplication:

2,264 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
1,128 duplicated lines

194 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Longest Duplicates

The list of 50 longest duplicates.

See data for all 194 duplicates...

Size	#	Folders	Files	Lines	Code
43	x 2	scripts scripts	run_cpt.py run_sft.py	147:209 (34%) 172:233 (30%)	view
37	x 2	scripts scripts	run_cpt.py run_sft.py	34:89 (30%) 36:91 (26%)	view
36	x 2	recipes/constitutional-ai/sft recipes/constitutional-ai/sft	config_anthropic.yaml config_grok.yaml	11:48 (83%) 11:48 (83%)	view
22	x 2	scripts scripts	run_dpo.py run_orpo.py	48:83 (12%) 47:82 (12%)	view
17	x 2	recipes/smollm2/dpo recipes/smollm2/dpo	config.yaml config_smol.yaml	3:24 (45%) 3:24 (45%)	view
16	x 2	scripts scripts	run_dpo.py run_sft.py	226:248 (9%) 198:220 (11%)	view
16	x 2	scripts scripts	run_cpt.py run_dpo.py	174:196 (13%) 226:248 (9%)	view
14	x 2	scripts scripts	run_dpo.py run_orpo.py	110:127 (8%) 166:183 (8%)	view
14	x 2	scripts scripts	run_dpo.py run_sft.py	141:156 (8%) 108:123 (9%)	view
14	x 2	scripts scripts	run_dpo.py run_orpo.py	204:226 (8%) 211:233 (8%)	view
14	x 2	scripts scripts	run_cpt.py run_sft.py	121:135 (11%) 107:121 (9%)	view
13	x 2	scripts scripts	run_cpt.py run_dpo.py	122:135 (10%) 141:154 (7%)	view
12	x 2	scripts scripts	run_cpt.py run_orpo.py	175:189 (9%) 236:250 (6%)	view
12	x 2	scripts scripts	run_orpo.py run_sft.py	236:250 (6%) 199:213 (8%)	view
12	x 2	scripts scripts	run_dpo.py run_orpo.py	227:241 (6%) 236:250 (6%)	view
11	x 2	recipes/smollm2/sft recipes/smollm2/sft	config.yaml config_smol.yaml	15:27 (26%) 12:24 (27%)	view
11	x 2	recipes/smollm2/sft recipes/smollm2/sft	config.yaml config_smol.yaml	39:49 (26%) 36:46 (27%)	view
11	x 2	recipes/smollm2/sft recipes/zephyr-7b-gemma/sft	config.yaml config_full.yaml	39:49 (26%) 38:48 (25%)	view
11	x 2	recipes/smollm/sft recipes/smollm2/sft	config.yaml config.yaml	43:53 (23%) 39:49 (26%)	view
11	x 2	recipes/smollm/sft recipes/smollm2/sft	config.yaml config_smol.yaml	43:53 (23%) 36:46 (27%)	view
11	x 2	scripts scripts	run_dpo.py run_orpo.py	247:261 (6%) 256:270 (6%)	view
11	x 2	recipes/zephyr-7b-beta/dpo recipes/zephyr-7b-beta/dpo	config_full.yaml config_qlora.yaml	7:19 (31%) 23:35 (22%)	view
11	x 2	recipes/accelerate_configs recipes/accelerate_configs	fsdp.yaml multi_gpu.yaml	16:26 (42%) 6:16 (68%)	view
11	x 2	recipes/smollm2/sft recipes/zephyr-7b-gemma/sft	config_smol.yaml config_full.yaml	36:46 (27%) 38:48 (25%)	view
11	x 2	recipes/gpt2-nl/sft recipes/zephyr-7b-beta/sft	config_full.yaml config_full.yaml	10:22 (27%) 11:23 (26%)	view
11	x 2	scripts scripts	run_orpo.py run_sft.py	30:45 (6%) 34:49 (7%)	view
11	x 2	recipes/accelerate_configs recipes/accelerate_configs	deepspeed_zero3.yaml fsdp.yaml	12:22 (50%) 16:26 (42%)	view
11	x 2	recipes/accelerate_configs recipes/accelerate_configs	deepspeed_zero3.yaml multi_gpu.yaml	12:22 (50%) 6:16 (68%)	view
11	x 2	recipes/smollm/sft recipes/zephyr-7b-gemma/sft	config.yaml config_full.yaml	43:53 (23%) 38:48 (25%)	view
10	x 2	scripts scripts	run_dpo.py run_orpo.py	27:36 (5%) 28:37 (5%)	view
10	x 2	recipes/pref_align_scan/dpo recipes/zephyr-7b-beta/dpo	config_zephyr.yaml config_full.yaml	2:15 (29%) 2:16 (28%)	view
10	x 2	recipes/zephyr-7b-beta/sft recipes/zephyr-7b-beta/sft	config_full.yaml config_qlora.yaml	8:19 (24%) 23:34 (18%)	view
10	x 2	recipes/starchat2-15b/dpo recipes/zephyr-7b-gemma/dpo	config_v0.1.yaml config_full.yaml	13:24 (27%) 12:23 (27%)	view
9	x 2	recipes/smollm2/dpo recipes/starchat2-15b/dpo	config_smol.yaml config_v0.1.yaml	35:43 (24%) 35:43 (24%)	view
9	x 2	recipes/zephyr-7b-beta/dpo recipes/zephyr-7b-beta/sft	config_qlora.yaml config_qlora.yaml	11:19 (18%) 12:20 (16%)	view
9	x 2	recipes/smollm2/dpo recipes/starchat2-15b/dpo	config.yaml config_v0.1.yaml	35:43 (24%) 35:43 (24%)	view
9	x 2	recipes/smollm2/dpo recipes/zephyr-7b-gemma/dpo	config.yaml config_full.yaml	35:43 (24%) 34:42 (25%)	view
9	x 2	recipes/starchat2-15b/dpo recipes/zephyr-7b-gemma/dpo	config_v0.1.yaml config_full.yaml	35:43 (24%) 34:42 (25%)	view
9	x 2	recipes/gpt2-nl/cpt recipes/gpt2-nl/sft	config_full.yaml config_full.yaml	37:45 (22%) 37:45 (22%)	view
9	x 2	recipes/starchat2-15b/sft recipes/zephyr-7b-gemma/sft	config_v0.1.yaml config_full.yaml	29:37 (20%) 28:36 (20%)	view
9	x 2	recipes/constitutional-ai/sft recipes/zephyr-7b-beta/sft	config_anthropic.yaml config_full.yaml	27:35 (20%) 25:33 (21%)	view
9	x 2	scripts scripts	run_cpt.py run_orpo.py	34:47 (7%) 32:45 (5%)	view
9	x 2	recipes/constitutional-ai/sft recipes/zephyr-7b-beta/sft	config_anthropic.yaml config_full.yaml	40:48 (20%) 38:46 (21%)	view
9	x 2	recipes/smollm2/dpo recipes/smollm2/dpo	config.yaml config_smol.yaml	35:43 (24%) 35:43 (24%)	view
9	x 2	recipes/constitutional-ai/sft recipes/zephyr-7b-beta/sft	config_grok.yaml config_full.yaml	40:48 (20%) 38:46 (21%)	view
9	x 2	recipes/constitutional-ai/sft recipes/zephyr-7b-beta/sft	config_grok.yaml config_full.yaml	27:35 (20%) 25:33 (21%)	view
9	x 2	recipes/smollm2/dpo recipes/zephyr-7b-gemma/dpo	config_smol.yaml config_full.yaml	35:43 (24%) 34:42 (25%)	view
8	x 2	scripts scripts	run_orpo.py run_sft.py	107:114 (4%) 114:121 (5%)	view
8	x 2	recipes/starchat2-15b/sft recipes/zephyr-7b-gemma/sft	config_v0.1.yaml config_full.yaml	42:49 (18%) 41:48 (18%)	view
8	x 2	recipes/gpt2-nl/dpo recipes/zephyr-7b-beta/dpo	config_full.yaml config_full.yaml	35:42 (21%) 34:41 (22%)	view