huggingface / blog Duplication

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

67% duplication:

37,090 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
24,986 duplicated lines

38,292 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Longest Duplicates

The list of 50 longest duplicates.

See data for all 38,292 duplicates...

Size	#	Folders	Files	Lines	Code
57	x 2	notebooks notebooks	01_how_to_train.ipynb 10_tf_serving.ipynb	460:516 (3%) 457:513 (5%)	view
51	x 2	notebooks notebooks	HuggingFace_int8_demo.ipynb unified-tool-calling.ipynb	4059:4109 (<1%) 3238:3288 (1%)	view
51	x 2	notebooks notebooks	HuggingFace_int8_demo.ipynb unified-tool-calling.ipynb	1188:1238 (<1%) 4057:4107 (1%)	view
51	x 2	notebooks notebooks	HuggingFace_int8_demo.ipynb unified-tool-calling.ipynb	2281:2331 (<1%) 450:500 (1%)	view
51	x 2	notebooks notebooks	HuggingFace_int8_demo.ipynb unified-tool-calling.ipynb	4059:4109 (<1%) 3186:3236 (1%)	view
51	x 2	notebooks notebooks	HuggingFace_int8_demo.ipynb unified-tool-calling.ipynb	2281:2331 (<1%) 502:552 (1%)	view
51	x 2	notebooks notebooks	HuggingFace_int8_demo.ipynb unified-tool-calling.ipynb	4059:4109 (<1%) 3373:3423 (1%)	view
51	x 2	notebooks notebooks	HuggingFace_int8_demo.ipynb unified-tool-calling.ipynb	2281:2331 (<1%) 569:619 (1%)	view
51	x 2	notebooks notebooks	HuggingFace_int8_demo.ipynb unified-tool-calling.ipynb	4059:4109 (<1%) 3305:3355 (1%)	view
51	x 2	notebooks notebooks	HuggingFace_int8_demo.ipynb unified-tool-calling.ipynb	2281:2331 (<1%) 637:687 (1%)	view
51	x 2	notebooks notebooks	graphml-classification.ipynb graphml-classification.ipynb	4796:4846 (<1%) 4967:5017 (<1%)	view
51	x 2	notebooks notebooks	graphml-classification.ipynb graphml-classification.ipynb	4796:4846 (<1%) 4915:4965 (<1%)	view
51	x 2	notebooks notebooks	HuggingFace_int8_demo.ipynb unified-tool-calling.ipynb	4059:4109 (<1%) 3580:3630 (1%)	view
51	x 2	notebooks notebooks	HuggingFace_int8_demo.ipynb unified-tool-calling.ipynb	2281:2331 (<1%) 792:842 (1%)	view
51	x 2	notebooks notebooks	HuggingFace_int8_demo.ipynb unified-tool-calling.ipynb	4059:4109 (<1%) 3528:3578 (1%)	view
51	x 2	notebooks notebooks	56_fine_tune_segformer.ipynb 56_fine_tune_segformer.ipynb	2246:2296 (<1%) 4046:4096 (<1%)	view
51	x 2	notebooks notebooks	56_fine_tune_segformer.ipynb 56_fine_tune_segformer.ipynb	2246:2296 (<1%) 4098:4148 (<1%)	view
51	x 2	notebooks notebooks	HuggingFace_int8_demo.ipynb HuggingFace_int8_demo.ipynb	2623:2673 (<1%) 5017:5067 (<1%)	view
51	x 2	notebooks notebooks	HuggingFace_int8_demo.ipynb HuggingFace_int8_demo.ipynb	2623:2673 (<1%) 5085:5135 (<1%)	view
51	x 2	notebooks notebooks	graphml-classification.ipynb graphml-classification.ipynb	4796:4846 (<1%) 4863:4913 (<1%)	view
51	x 2	notebooks notebooks	HuggingFace_int8_demo.ipynb unified-tool-calling.ipynb	2281:2331 (<1%) 844:894 (1%)	view
51	x 2	notebooks notebooks	graphml-classification.ipynb graphml-classification.ipynb	3859:3909 (<1%) 5962:6012 (<1%)	view
51	x 2	notebooks notebooks	HuggingFace_int8_demo.ipynb unified-tool-calling.ipynb	4059:4109 (<1%) 3647:3697 (1%)	view
51	x 2	notebooks notebooks	HuggingFace_int8_demo.ipynb unified-tool-calling.ipynb	2281:2331 (<1%) 911:961 (1%)	view
51	x 2	notebooks notebooks	HuggingFace_int8_demo.ipynb unified-tool-calling.ipynb	2281:2331 (<1%) 979:1029 (1%)	view
51	x 2	notebooks notebooks	HuggingFace_int8_demo.ipynb unified-tool-calling.ipynb	4059:4109 (<1%) 3715:3765 (1%)	view
51	x 2	notebooks notebooks	HuggingFace_int8_demo.ipynb unified-tool-calling.ipynb	4059:4109 (<1%) 3870:3920 (1%)	view
51	x 2	notebooks notebooks	HuggingFace_int8_demo.ipynb unified-tool-calling.ipynb	4059:4109 (<1%) 3989:4039 (1%)	view
51	x 2	notebooks notebooks	HuggingFace_int8_demo.ipynb unified-tool-calling.ipynb	4059:4109 (<1%) 3922:3972 (1%)	view
51	x 2	notebooks notebooks	graphml-classification.ipynb graphml-classification.ipynb	1187:1237 (<1%) 2550:2600 (<1%)	view
51	x 2	notebooks notebooks	56_fine_tune_segformer.ipynb 56_fine_tune_segformer.ipynb	2246:2296 (<1%) 4556:4606 (<1%)	view
51	x 2	notebooks notebooks	graphml-classification.ipynb graphml-classification.ipynb	4639:4689 (<1%) 4863:4913 (<1%)	view
51	x 2	notebooks notebooks	56_fine_tune_segformer.ipynb 80_getting_started_with_embeddings.ipynb	2600:2650 (<1%) 1137:1187 (2%)	view
51	x 2	notebooks notebooks	graphml-classification.ipynb graphml-classification.ipynb	5730:5780 (<1%) 5834:5884 (<1%)	view
51	x 2	notebooks notebooks	56_fine_tune_segformer.ipynb 56_fine_tune_segformer.ipynb	2246:2296 (<1%) 4653:4703 (<1%)	view
51	x 2	notebooks notebooks	graphml-classification.ipynb graphml-classification.ipynb	4639:4689 (<1%) 4967:5017 (<1%)	view
51	x 2	notebooks notebooks	graphml-classification.ipynb graphml-classification.ipynb	1187:1237 (<1%) 2696:2746 (<1%)	view
51	x 2	notebooks notebooks	56_fine_tune_segformer.ipynb 80_getting_started_with_embeddings.ipynb	2600:2650 (<1%) 1033:1083 (2%)	view
51	x 2	notebooks notebooks	graphml-classification.ipynb graphml-classification.ipynb	4639:4689 (<1%) 4915:4965 (<1%)	view
51	x 2	notebooks notebooks	56_fine_tune_segformer.ipynb 80_getting_started_with_embeddings.ipynb	2600:2650 (<1%) 1085:1135 (2%)	view
51	x 2	notebooks notebooks	56_fine_tune_segformer.ipynb 56_fine_tune_segformer.ipynb	2246:2296 (<1%) 4729:4779 (<1%)	view
51	x 2	notebooks notebooks	graphml-classification.ipynb graphml-classification.ipynb	1187:1237 (<1%) 2644:2694 (<1%)	view
51	x 2	notebooks notebooks	56_fine_tune_segformer.ipynb 56_fine_tune_segformer.ipynb	2246:2296 (<1%) 4781:4831 (<1%)	view
51	x 2	notebooks notebooks	graphml-classification.ipynb graphml-classification.ipynb	1187:1237 (<1%) 2321:2371 (<1%)	view
51	x 2	notebooks notebooks	HuggingFace_int8_demo.ipynb HuggingFace_int8_demo.ipynb	2214:2264 (<1%) 3991:4041 (<1%)	view
51	x 2	notebooks notebooks	graphml-classification.ipynb graphml-classification.ipynb	1187:1237 (<1%) 2388:2438 (<1%)	view
51	x 2	notebooks notebooks	HuggingFace_int8_demo.ipynb unified-tool-calling.ipynb	2504:2554 (<1%) 4057:4107 (1%)	view
51	x 2	notebooks notebooks	HuggingFace_int8_demo.ipynb HuggingFace_int8_demo.ipynb	2214:2264 (<1%) 3924:3974 (<1%)	view
51	x 2	notebooks notebooks	56_fine_tune_segformer.ipynb 56_fine_tune_segformer.ipynb	2246:2296 (<1%) 4833:4883 (<1%)	view
51	x 2	notebooks notebooks	56_fine_tune_segformer.ipynb 56_fine_tune_segformer.ipynb	2246:2296 (<1%) 4885:4935 (<1%)	view