azure / torchtune-azureml

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

88% duplication:

3,238 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
2,869 duplicated lines

943 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Longest Duplicates

The list of 50 longest duplicates.

See data for all 943 duplicates...

Size	#	Folders	Files	Lines	Code
108	x 2	ROOT ROOT	training_dpo.ipynb training_sft.ipynb	232:339 (19%) 230:337 (19%)	view
101	x 2	ROOT ROOT	training_kd.ipynb training_sft.ipynb	12:112 (17%) 12:112 (18%)	view
95	x 2	ROOT ROOT	training_dpo.ipynb training_kd.ipynb	251:345 (17%) 258:352 (16%)	view
94	x 2	ROOT ROOT	training_dpo.ipynb training_sft.ipynb	43:136 (16%) 39:132 (16%)	view
90	x 2	ROOT ROOT	training_kd.ipynb training_sft.ipynb	397:486 (15%) 384:473 (16%)	view
89	x 2	ROOT ROOT	training_kd.ipynb training_sft.ipynb	258:346 (15%) 249:337 (15%)	view
84	x 2	ROOT ROOT	training_kd.ipynb training_sft.ipynb	121:204 (14%) 110:193 (15%)	view
74	x 2	ROOT ROOT	training_dpo.ipynb training_kd.ipynb	43:116 (13%) 39:112 (12%)	view
74	x 2	scripts scripts	launcher_distributed_kd.py launcher_single_kd.py	218:339 (30%) 160:281 (37%)	view
72	x 2	scripts scripts	launcher_distributed_kd.py launcher_single_kd.py	347:452 (29%) 282:390 (36%)	view
64	x 2	ROOT ROOT	training_dpo.ipynb training_kd.ipynb	167:230 (11%) 174:237 (11%)	view
58	x 2	scripts scripts	lora_finetune_dpo_phi3.yaml lora_finetune_dpo_phi4.yaml	41:125 (76%) 48:133 (71%)	view
56	x 2	scripts scripts	launcher_single.py launcher_single_kd.py	103:243 (29%) 104:245 (28%)	view
52	x 2	ROOT ROOT	training_kd.ipynb training_sft.ipynb	520:571 (9%) 508:559 (9%)	view
52	x 2	ROOT ROOT	training_dpo.ipynb training_sft.ipynb	437:488 (9%) 440:491 (9%)	view
51	x 2	scripts scripts	launcher_distributed.py launcher_distributed_kd.py	188:301 (20%) 190:303 (20%)	view
50	x 2	scripts scripts	launcher_distributed.py launcher_single.py	379:451 (20%) 309:383 (26%)	view
50	x 2	ROOT ROOT	training_kd.ipynb training_sft.ipynb	207:256 (8%) 198:247 (8%)	view
49	x 2	scripts scripts	launcher_distributed.py launcher_distributed_kd.py	18:122 (19%) 18:122 (19%)	view
49	x 2	scripts scripts	launcher_single.py launcher_single_kd.py	265:341 (25%) 271:347 (24%)	view
46	x 2	scripts scripts	launcher_distributed.py launcher_single.py	216:304 (18%) 159:246 (24%)	view
43	x 2	scripts scripts	launcher_distributed.py launcher_single_kd.py	216:301 (17%) 160:245 (21%)	view
43	x 2	scripts scripts	launcher_distributed_kd.py launcher_single.py	218:303 (17%) 159:243 (22%)	view
43	x 2	scripts scripts	launcher_distributed_kd.py launcher_single.py	347:411 (17%) 276:341 (22%)	view
42	x 2	ROOT ROOT	training_dpo.ipynb training_kd.ipynb	514:555 (7%) 530:571 (7%)	view
42	x 2	ROOT ROOT	training_dpo.ipynb training_sft.ipynb	514:555 (7%) 518:559 (7%)	view
37	x 2	scripts scripts	launcher_distributed_kd.py launcher_single_kd.py	44:126 (15%) 16:98 (18%)	view
36	x 2	scripts scripts	launcher_distributed.py launcher_single.py	44:125 (14%) 16:97 (18%)	view
35	x 2	ROOT ROOT	training_dpo.ipynb training_kd.ipynb	382:416 (6%) 392:426 (6%)	view
34	x 2	scripts scripts	launcher_distributed.py launcher_distributed_kd.py	324:373 (13%) 329:378 (13%)	view
34	x 2	ROOT ROOT	training_dpo.ipynb training_kd.ipynb	437:470 (6%) 453:486 (5%)	view
33	x 2	scripts scripts	launcher_distributed_kd.py launcher_single.py	44:122 (13%) 16:94 (17%)	view
33	x 2	scripts scripts	launcher_distributed.py launcher_single_kd.py	44:122 (13%) 16:94 (16%)	view
33	x 2	scripts scripts	launcher_single.py launcher_single_kd.py	16:94 (17%) 16:94 (16%)	view
31	x 2	ROOT ROOT	training_dpo.ipynb training_sft.ipynb	167:197 (5%) 163:193 (5%)	view
31	x 2	ROOT ROOT	training_dpo.ipynb training_sft.ipynb	200:230 (5%) 198:228 (5%)	view
30	x 2	scripts scripts	lora_finetune_dpo_phi4.yaml lora_finetune_phi4.yaml	19:57 (37%) 19:56 (37%)	view
30	x 2	ROOT ROOT	training_dpo.ipynb training_sft.ipynb	387:416 (5%) 384:413 (5%)	view
28	x 2	ROOT ROOT	training_kd.ipynb training_sft.ipynb	488:515 (4%) 475:502 (5%)	view
27	x 2	scripts scripts	launcher_distributed.py launcher_distributed_kd.py	413:451 (10%) 414:452 (10%)	view
27	x 2	scripts scripts	launcher_distributed.py launcher_single_kd.py	413:451 (10%) 350:390 (13%)	view
27	x 2	scripts scripts	launcher_distributed_kd.py launcher_single.py	414:452 (10%) 343:383 (14%)	view
27	x 2	scripts scripts	launcher_single.py launcher_single_kd.py	343:383 (14%) 350:390 (13%)	view
25	x 2	scripts scripts	lora_finetune_dpo_phi4.yaml lora_finetune_phi3.yaml	89:133 (30%) 78:122 (32%)	view
25	x 2	scripts scripts	lora_finetune_dpo_phi3.yaml lora_finetune_phi3.yaml	81:125 (32%) 78:122 (32%)	view
24	x 2	scripts scripts	lora_finetune_phi3.yaml lora_finetune_phi4.yaml	67:101 (31%) 74:109 (29%)	view
24	x 2	scripts scripts	launcher_distributed.py launcher_single.py	306:334 (9%) 247:275 (12%)	view
23	x 2	ROOT ROOT	training_dpo.ipynb training_kd.ipynb	114:136 (4%) 121:143 (4%)	view
22	x 2	scripts scripts	launcher_distributed.py launcher_single_kd.py	379:411 (8%) 315:347 (11%)	view
22	x 2	scripts scripts	launcher_distributed.py launcher_distributed_kd.py	379:411 (8%) 379:411 (8%)	view