azure / slm-innovator-lab

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

65% duplication:

15,803 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
10,327 duplicated lines

15,218 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Duplication Between Components (50+ lines)

Download: SVG DOT (open online Graphviz editor)

From Component --> To Component	Duplicated Lines	File Pairs	Details
1_synthetic-qa-generation (21%) --> 2_slm-fine-tuning-mlstudio (17%)	2391	64 file pairs	details...
1_synthetic-qa-generation (12%) --> 3_llmops-aifoundry (26%)	1456	32 file pairs	details...
0_lab_preparation (134%) --> 1_synthetic-qa-generation (11%)	1295	17 file pairs	details...
2_slm-fine-tuning-mlstudio (7%) --> 3_llmops-aifoundry (18%)	988	32 file pairs	details...
0_lab_preparation (91%) --> 2_slm-fine-tuning-mlstudio (6%)	884	16 file pairs	details...
0_lab_preparation (65%) --> 3_llmops-aifoundry (11%)	637	8 file pairs	details...

Open 2D force graph... Open 3D force graph...

Show more details on duplication between components...

Longest Duplicates

The list of 50 longest duplicates.

See data for all 15,218 duplicates...

Size	#	Folders	Files	Lines	Code
275	x 2	2_slm-fine-tuning-mlstudio/phi 2_slm-fine-tuning-mlstudio/phi	1_training_custom_phi.ipynb 1_training_mlflow_phi.ipynb	38:312 (31%) 30:304 (31%)	view
169	x 2	2_slm-fine-tuning-mlstudio/phi 2_slm-fine-tuning-mlstudio/phi	2_serving_basic_phi.ipynb 2_serving_llm_phi.ipynb	9:177 (18%) 14:182 (19%)	view
133	x 2	3_llmops-aifoundry/3_4_operationalizing 3_llmops-aifoundry/3_4_operationalizing	contentsafety_with_code_en.ipynb contentsafety_with_code_ja.ipynb	331:463 (23%) 351:483 (22%)	view
105	x 2	2_slm-fine-tuning-mlstudio/phi 2_slm-fine-tuning-mlstudio/phi	1_training_custom_phi.ipynb 1_training_mlflow_phi.ipynb	418:522 (12%) 406:510 (12%)	view
102	x 2	2_slm-fine-tuning-mlstudio/phi 2_slm-fine-tuning-mlstudio/phi	1_training_custom_phi.ipynb 1_training_mlflow_phi.ipynb	665:766 (11%) 657:758 (11%)	view
97	x 2	3_llmops-aifoundry/3_4_operationalizing 3_llmops-aifoundry/3_4_operationalizing	contentsafety_with_code_en.ipynb contentsafety_with_code_ja.ipynb	80:176 (17%) 100:196 (16%)	view
84	x 2	2_slm-fine-tuning-mlstudio/phi 2_slm-fine-tuning-mlstudio/phi	2_serving_basic_phi.ipynb 2_serving_llm_phi.ipynb	462:545 (9%) 333:416 (9%)	view
72	x 2	2_slm-fine-tuning-mlstudio/florence2-VQA 2_slm-fine-tuning-mlstudio/phi	1_training_mlflow_florence2.ipynb 1_training_custom_phi.ipynb	47:118 (11%) 57:128 (8%)	view
72	x 2	2_slm-fine-tuning-mlstudio/florence2-VQA 2_slm-fine-tuning-mlstudio/phi	1_training_mlflow_florence2.ipynb 1_training_mlflow_phi.ipynb	47:118 (11%) 49:120 (8%)	view
66	x 2	3_llmops-aifoundry/3_4_operationalizing 3_llmops-aifoundry/3_4_operationalizing	contentsafety_with_code_en.ipynb contentsafety_with_code_ja.ipynb	234:299 (11%) 254:319 (11%)	view
59	x 2	3_llmops-aifoundry/3_4_operationalizing 3_llmops-aifoundry/3_4_operationalizing	contentsafety_with_code_en.ipynb contentsafety_with_code_ja.ipynb	481:539 (10%) 501:559 (10%)	view
54	x 2	3_llmops-aifoundry/3_4_operationalizing 3_llmops-aifoundry/3_4_operationalizing	contentsafety_with_code_en.ipynb contentsafety_with_code_ja.ipynb	179:232 (9%) 199:252 (9%)	view
53	x 2	2_slm-fine-tuning-mlstudio/florence2-VQA 2_slm-fine-tuning-mlstudio/phi	1_training_mlflow_florence2.ipynb 1_training_custom_phi.ipynb	468:520 (8%) 673:725 (6%)	view
53	x 2	2_slm-fine-tuning-mlstudio/florence2-VQA 2_slm-fine-tuning-mlstudio/phi	1_training_mlflow_florence2.ipynb 1_training_mlflow_phi.ipynb	468:520 (8%) 665:717 (6%)	view
53	x 2	2_slm-fine-tuning-mlstudio/phi 2_slm-fine-tuning-mlstudio/phi	2_serving_basic_phi.ipynb 2_serving_llm_phi.ipynb	198:250 (5%) 207:259 (6%)	view
50	x 2	2_slm-fine-tuning-mlstudio/phi 2_slm-fine-tuning-mlstudio/phi	1_training_custom_phi.ipynb 1_training_mlflow_phi.ipynb	584:633 (5%) 572:621 (5%)	view
50	x 2	1_synthetic-qa-generation/seed 1_synthetic-qa-generation/seed	make_qa_csv.ipynb make_qa_multimodal_pdf_oss.ipynb	122:171 (21%) 459:508 (7%)	view
48	x 2	1_synthetic-qa-generation/seed 1_synthetic-qa-generation/seed	make_qa_multimodal_pdf_oss.ipynb make_qa_only_image_multiple_pdf.ipynb	18:65 (7%) 8:55 (16%)	view
47	x 2	3_llmops-aifoundry/3_2_prototyping/flow-template 3_llmops-aifoundry/3_3_optimizing/flow-template	chat-serverless.flow.dag.yaml chat-serverless.flow.dag.yaml	1:47 (100%) 1:47 (100%)	view
47	x 2	3_llmops-aifoundry/3_4_operationalizing 3_llmops-aifoundry/3_4_operationalizing	contentsafety_with_code_en.ipynb contentsafety_with_code_ja.ipynb	37:83 (8%) 37:83 (8%)	view
43	x 2	2_slm-fine-tuning-mlstudio/phi 2_slm-fine-tuning-mlstudio/phi	2_serving_basic_phi.ipynb 3_optimization_olive.ipynb	170:212 (4%) 181:223 (7%)	view
43	x 2	2_slm-fine-tuning-mlstudio/phi 2_slm-fine-tuning-mlstudio/phi	1_training_custom_phi.ipynb 2_serving_basic_phi.ipynb	726:768 (4%) 170:212 (4%)	view
43	x 2	1_synthetic-qa-generation/evolve-instruct 1_synthetic-qa-generation/reasoningplaning	evolve.py evolve.py	29:78 (12%) 236:285 (6%)	view
43	x 2	2_slm-fine-tuning-mlstudio/phi 2_slm-fine-tuning-mlstudio/phi	1_training_custom_phi.ipynb 3_optimization_olive.ipynb	726:768 (4%) 181:223 (7%)	view
43	x 2	1_synthetic-qa-generation/auto-evolve-instruct 1_synthetic-qa-generation/glan-instruct	trial.ipynb glan_tutorial.ipynb	6:48 (10%) 24:66 (5%)	view
41	x 2	2_slm-fine-tuning-mlstudio/phi 2_slm-fine-tuning-mlstudio/phi	1_training_custom_phi.ipynb 1_training_mlflow_phi.ipynb	764:804 (4%) 763:803 (4%)	view
41	x 2	2_slm-fine-tuning-mlstudio/phi 2_slm-fine-tuning-mlstudio/phi	1_training_mlflow_phi.ipynb 2_serving_basic_phi.ipynb	718:758 (4%) 170:210 (4%)	view
41	x 2	2_slm-fine-tuning-mlstudio/phi 2_slm-fine-tuning-mlstudio/phi	1_training_mlflow_phi.ipynb 3_optimization_olive.ipynb	718:758 (4%) 181:221 (6%)	view
40	x 2	1_synthetic-qa-generation/seed 1_synthetic-qa-generation/seed	make_qa_multimodal_pdf_docai.ipynb make_qa_multimodal_pdf_oss.ipynb	460:499 (4%) 319:358 (5%)	view
40	x 2	2_slm-fine-tuning-mlstudio/phi/src_train 2_slm-fine-tuning-mlstudio/phi/src_train	train.py train_mlflow.py	121:172 (16%) 144:195 (16%)	view
39	x 2	2_slm-fine-tuning-mlstudio/florence2-VQA 2_slm-fine-tuning-mlstudio/phi	1_training_mlflow_florence2.ipynb 1_training_mlflow_phi.ipynb	171:209 (6%) 234:272 (4%)	view
39	x 2	2_slm-fine-tuning-mlstudio/florence2-VQA 2_slm-fine-tuning-mlstudio/phi	1_training_mlflow_florence2.ipynb 1_training_custom_phi.ipynb	171:209 (6%) 242:280 (4%)	view
38	x 2	3_llmops-aifoundry/3_2_prototyping 3_llmops-aifoundry/3_3_optimizing	promptflow_with_code.ipynb promptflow_with_evaluation_code.ipynb	26:63 (7%) 23:60 (11%)	view
38	x 2	2_slm-fine-tuning-mlstudio/florence2-VQA 2_slm-fine-tuning-mlstudio/phi	1_training_mlflow_florence2.ipynb 1_training_mlflow_phi.ipynb	8:45 (5%) 8:45 (4%)	view
38	x 2	3_llmops-aifoundry/3_3_optimizing/evaluation 3_llmops-aifoundry/3_3_optimizing/flow-template	flow.dag.yaml evaluation.flow.dag.yaml	1:38 (69%) 1:38 (69%)	view
37	x 2	1_synthetic-qa-generation/seed 1_synthetic-qa-generation/seed	make_qa_multimodal_pdf_oss.ipynb make_qa_only_image_pdf.ipynb	400:436 (5%) 160:196 (8%)	view
36	x 2	2_slm-fine-tuning-mlstudio/phi/src_train 2_slm-fine-tuning-mlstudio/phi/src_train	train.py train_mlflow.py	24:70 (14%) 49:95 (14%)	view
36	x 2	2_slm-fine-tuning-mlstudio/phi 2_slm-fine-tuning-mlstudio/phi	1_training_custom_phi.ipynb 1_training_mlflow_phi.ipynb	828:863 (4%) 831:866 (4%)	view
36	x 2	2_slm-fine-tuning-mlstudio/florence2-VQA 2_slm-fine-tuning-mlstudio/phi	1_training_mlflow_florence2.ipynb 1_training_mlflow_phi.ipynb	412:447 (5%) 582:617 (4%)	view
36	x 2	2_slm-fine-tuning-mlstudio/florence2-VQA 2_slm-fine-tuning-mlstudio/phi	1_training_mlflow_florence2.ipynb 1_training_custom_phi.ipynb	412:447 (5%) 594:629 (4%)	view
36	x 2	1_synthetic-qa-generation/seed 1_synthetic-qa-generation/seed	make_qa_multimodal_pdf_oss.ipynb make_qa_only_image_pdf.ipynb	549:584 (5%) 296:331 (8%)	view
35	x 2	3_llmops-aifoundry/3_4_operationalizing 3_llmops-aifoundry/3_4_operationalizing	contentsafety_with_code_en.ipynb contentsafety_with_code_ja.ipynb	1:35 (6%) 1:35 (6%)	view
35	x 2	2_slm-fine-tuning-mlstudio/florence2-VQA 2_slm-fine-tuning-mlstudio/phi	2_serving_florence2.ipynb 2_serving_llm_phi.ipynb	777:811 (4%) 837:871 (4%)	view
35	x 2	2_slm-fine-tuning-mlstudio/phi 2_slm-fine-tuning-mlstudio/phi	1_training_custom_phi.ipynb 1_training_mlflow_phi.ipynb	548:582 (4%) 536:570 (4%)	view
35	x 2	2_slm-fine-tuning-mlstudio/phi/src_train 2_slm-fine-tuning-mlstudio/phi/src_train	train.py train_mlflow.py	292:337 (14%) 312:357 (14%)	view
33	x 2	1_synthetic-qa-generation/seed 1_synthetic-qa-generation/seed	make_qa_multimodal_pdf_docai.ipynb make_qa_multimodal_pdf_oss.ipynb	555:587 (3%) 400:432 (4%)	view
33	x 2	1_synthetic-qa-generation/seed 1_synthetic-qa-generation/seed	make_qa_multimodal_pdf_docai.ipynb make_qa_only_image_pdf.ipynb	555:587 (3%) 160:192 (7%)	view
32	x 2	2_slm-fine-tuning-mlstudio/phi 2_slm-fine-tuning-mlstudio/phi	1_training_custom_phi.ipynb 2_serving_basic_phi.ipynb	458:489 (3%) 329:360 (3%)	view
32	x 2	2_slm-fine-tuning-mlstudio/phi 2_slm-fine-tuning-mlstudio/phi	1_training_mlflow_phi.ipynb 2_serving_basic_phi.ipynb	446:477 (3%) 329:360 (3%)	view
32	x 2	1_synthetic-qa-generation/evolve-instruct 1_synthetic-qa-generation/reasoningplaning	evolve.py evolve.py	312:351 (9%) 840:879 (4%)	view