huggingface / cookbook Duplication

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

84% duplication:

54,816 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
46,427 duplicated lines

138,136 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Longest Duplicates

The list of 50 longest duplicates.

See data for all 138,136 duplicates...

Size	#	Folders	Files	Lines	Code
5154	x 2	notebooks/en notebooks/zh-CN	rag_with_hugging_face_gemma_elasticse... rag_with_hugging_face_gemma_elasticse...	1165:6318 (81%) 1166:6319 (81%)	view
2066	x 2	notebooks/en notebooks/zh-CN	labelling_feedback_setfit.ipynb labelling_feedback_setfit.ipynb	611:2676 (77%) 798:2863 (72%)	view
308	x 2	notebooks/en notebooks/zh-CN	rag_with_hugging_face_gemma_elasticse... rag_with_hugging_face_gemma_elasticse...	798:1105 (4%) 799:1106 (4%)	view
278	x 2	notebooks/en notebooks/zh-CN	phoenix_observability_on_hf_spaces.ipynb phoenix_observability_on_hf_spaces.ipynb	303:580 (45%) 298:575 (46%)	view
175	x 2	notebooks/tr notebooks/zh-CN	fine_tuning_code_llm_on_single_gpu.ipynb fine_tuning_code_llm_on_single_gpu.ipynb	730:904 (15%) 723:897 (15%)	view
171	x 2	notebooks/en notebooks/zh-CN	llm_gateway_pii_detection.ipynb llm_gateway_pii_detection.ipynb	291:461 (30%) 293:463 (30%)	view
137	x 2	notebooks/en notebooks/zh-CN	structured_generation.ipynb structured_generation.ipynb	87:223 (25%) 87:223 (25%)	view
130	x 2	notebooks/en notebooks/zh-CN	fine_tuning_code_llm_on_single_gpu.ipynb fine_tuning_code_llm_on_single_gpu.ipynb	400:529 (11%) 387:516 (11%)	view
126	x 2	notebooks/en notebooks/zh-CN	enterprise_cookbook_argilla.ipynb enterprise_cookbook_argilla.ipynb	258:383 (15%) 260:385 (15%)	view
123	x 2	notebooks/en notebooks/zh-CN	rag_with_hugging_face_gemma_elasticse... rag_with_hugging_face_gemma_elasticse...	139:261 (1%) 141:263 (1%)	view
119	x 2	notebooks/en notebooks/zh-CN	rag_with_hugging_face_gemma_elasticse... rag_with_hugging_face_gemma_elasticse...	467:585 (1%) 470:588 (1%)	view
118	x 2	notebooks/en notebooks/tr	fine_tuning_code_llm_on_single_gpu.ipynb fine_tuning_code_llm_on_single_gpu.ipynb	779:896 (10%) 753:870 (10%)	view
118	x 2	notebooks/en notebooks/zh-CN	fine_tuning_code_llm_on_single_gpu.ipynb fine_tuning_code_llm_on_single_gpu.ipynb	779:896 (10%) 746:863 (10%)	view
101	x 2	notebooks/en notebooks/zh-CN	enterprise_cookbook_argilla.ipynb enterprise_cookbook_argilla.ipynb	541:641 (12%) 546:646 (12%)	view
100	x 2	notebooks/en notebooks/zh-CN	fine_tuning_llm_to_generate_persian_p... fine_tuning_llm_to_generate_persian_p...	465:564 (14%) 471:570 (13%)	view
97	x 2	notebooks/en notebooks/zh-CN	llm_gateway_pii_detection.ipynb llm_gateway_pii_detection.ipynb	182:278 (17%) 183:279 (17%)	view
91	x 2	notebooks/en notebooks/zh-CN	fine_tuning_code_llm_on_single_gpu.ipynb fine_tuning_code_llm_on_single_gpu.ipynb	289:379 (8%) 276:366 (8%)	view
87	x 2	notebooks/en notebooks/zh-CN	rag_with_hugging_face_gemma_elasticse... rag_with_hugging_face_gemma_elasticse...	587:673 (1%) 590:676 (1%)	view
87	x 2	notebooks/en notebooks/zh-CN	automatic_embedding_tei_inference_end... automatic_embedding_tei_inference_end...	510:596 (10%) 496:582 (10%)	view
87	x 2	notebooks/en notebooks/zh-CN	rag_with_knowledge_graphs_neo4j.ipynb rag_with_knowledge_graphs_neo4j.ipynb	558:644 (13%) 558:644 (13%)	view
83	x 2	notebooks/en notebooks/zh-CN	analyzing_art_with_hf_and_fiftyone.ipynb analyzing_art_with_hf_and_fiftyone.ipynb	367:449 (9%) 369:451 (9%)	view
82	x 2	notebooks/fr notebooks/zh-CN	structured_generation.ipynb structured_generation.ipynb	199:280 (13%) 142:223 (15%)	view
82	x 2	notebooks/en notebooks/zh-CN	automatic_embedding_tei_inference_end... automatic_embedding_tei_inference_end...	230:311 (9%) 215:296 (9%)	view
82	x 2	notebooks/en notebooks/fr	structured_generation.ipynb structured_generation.ipynb	142:223 (15%) 199:280 (13%)	view
79	x 2	notebooks/en notebooks/zh-CN	fine_tuning_llm_to_generate_persian_p... fine_tuning_llm_to_generate_persian_p...	357:435 (11%) 362:440 (11%)	view
76	x 2	notebooks/en notebooks/zh-CN	llm_gateway_pii_detection.ipynb llm_gateway_pii_detection.ipynb	464:539 (13%) 467:542 (13%)	view
75	x 2	notebooks/en notebooks/zh-CN	semantic_reranking_elasticsearch.ipynb semantic_reranking_elasticsearch.ipynb	474:548 (12%) 472:546 (12%)	view
75	x 2	notebooks/en notebooks/zh-CN	code_search.ipynb code_search.ipynb	588:662 (9%) 583:657 (9%)	view
74	x 2	notebooks/en notebooks/zh-CN	rag_with_hugging_face_gemma_elasticse... rag_with_hugging_face_gemma_elasticse...	265:338 (1%) 267:340 (1%)	view
74	x 2	notebooks/en notebooks/zh-CN	fine_tuning_llm_to_generate_persian_p... fine_tuning_llm_to_generate_persian_p...	282:355 (10%) 285:358 (10%)	view
74	x 2	notebooks/en notebooks/zh-CN	enterprise_cookbook_argilla.ipynb enterprise_cookbook_argilla.ipynb	136:209 (9%) 137:210 (9%)	view
69	x 2	notebooks/en notebooks/zh-CN	rag_with_sql_reranker.ipynb rag_with_sql_reranker.ipynb	712:780 (8%) 710:778 (8%)	view
68	x 2	notebooks/en notebooks/zh-CN	automatic_embedding_tei_inference_end... automatic_embedding_tei_inference_end...	651:718 (8%) 638:705 (8%)	view
67	x 2	notebooks/en notebooks/zh-CN	rag_with_hugging_face_gemma_elasticse... rag_with_hugging_face_gemma_elasticse...	681:747 (1%) 682:748 (1%)	view
66	x 2	notebooks/en notebooks/zh-CN	semantic_reranking_elasticsearch.ipynb semantic_reranking_elasticsearch.ipynb	394:459 (11%) 392:457 (11%)	view
61	x 2	notebooks/en notebooks/zh-CN	code_search.ipynb code_search.ipynb	679:739 (7%) 670:730 (8%)	view
61	x 2	notebooks/tr notebooks/zh-CN	agents.ipynb agents.ipynb	451:511 (9%) 652:712 (7%)	view
60	x 2	notebooks/en notebooks/zh-CN	rag_with_knowledge_graphs_neo4j.ipynb rag_with_knowledge_graphs_neo4j.ipynb	484:543 (9%) 484:543 (9%)	view
59	x 2	notebooks/en notebooks/zh-CN	phoenix_observability_on_hf_spaces.ipynb phoenix_observability_on_hf_spaces.ipynb	241:299 (9%) 236:294 (9%)	view
59	x 2	notebooks/tr notebooks/zh-CN	fine_tuning_code_llm_on_single_gpu.ipynb fine_tuning_code_llm_on_single_gpu.ipynb	414:472 (5%) 407:465 (5%)	view
59	x 2	notebooks/en notebooks/zh-CN	clean_dataset_judges_distilabel.ipynb clean_dataset_judges_distilabel.ipynb	385:443 (9%) 384:442 (9%)	view
59	x 2	notebooks/en notebooks/tr	fine_tuning_code_llm_on_single_gpu.ipynb fine_tuning_code_llm_on_single_gpu.ipynb	420:478 (5%) 414:472 (5%)	view
58	x 2	notebooks/en notebooks/fr	structured_generation.ipynb structured_generation.ipynb	225:282 (10%) 282:339 (9%)	view
58	x 2	notebooks/tr notebooks/zh-CN	fine_tuning_code_llm_on_single_gpu.ipynb fine_tuning_code_llm_on_single_gpu.ipynb	474:531 (5%) 467:524 (5%)	view
58	x 2	notebooks/en notebooks/zh-CN	code_search.ipynb code_search.ipynb	232:289 (7%) 231:288 (7%)	view
58	x 2	notebooks/en notebooks/zh-CN	structured_generation.ipynb structured_generation.ipynb	225:282 (10%) 225:282 (10%)	view
58	x 2	notebooks/fr notebooks/zh-CN	structured_generation.ipynb structured_generation.ipynb	282:339 (9%) 225:282 (10%)	view
57	x 2	notebooks/en notebooks/zh-CN	search_and_learn.ipynb labelling_feedback_setfit.ipynb	8151:8207 (<1%) 2807:2863 (1%)	view
57	x 2	notebooks/en notebooks/zh-CN	rag_with_knowledge_graphs_neo4j.ipynb rag_with_knowledge_graphs_neo4j.ipynb	409:465 (8%) 409:465 (8%)	view
57	x 2	notebooks/en notebooks/en	labelling_feedback_setfit.ipynb search_and_learn.ipynb	2620:2676 (2%) 8151:8207 (<1%)	view