huggingface / data-is-better-together Duplication

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

48% duplication:

8,029 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
3,920 duplicated lines

2,654 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Duplication Between Components (50+ lines)

Download: SVG DOT (open online Graphviz editor)

From Component --> To Component	Duplicated Lines	File Pairs	Details
community-efforts (9%) --> cookbook-efforts (14%)	917	8 file pairs	details...

Open 2D force graph... Open 3D force graph...

Show more details on duplication between components...

Longest Duplicates

The list of 50 longest duplicates.

See data for all 2,654 duplicates...

Size	#	Folders	Files	Lines	Code
236	x 2	community-efforts/image_preferences community-efforts/image_preferences	01_synthetic_data_generation_images.py 01_synthetic_data_generation_total.py	23:296 (61%) 23:296 (57%)	view
168	x 2	community-efforts/image_preferences community-efforts/image_preferences	01_synthetic_data_generation_prompts.py 01_synthetic_data_generation_total.py	20:218 (67%) 113:311 (41%)	view
160	x 2	community-efforts/image_preferences community-efforts/image_preferences	01_synthetic_data_generation_images.py 01_synthetic_data_generation_prompts.py	113:296 (41%) 20:203 (64%)	view
79	x 2	community-efforts/image_preferences community-efforts/image_preferences	03_upload_to_argilla.ipynb 03_upload_to_argilla.ipynb	348:426 (5%) 680:758 (5%)	view
62	x 2	cookbook-efforts/domain-...ic-datasets/project_app cookbook-efforts/domain-...ic-datasets/project_app	pipeline.yaml pipeline.yaml	90:151 (11%) 402:463 (11%)	view
61	x 2	cookbook-efforts/domain-...ic-datasets/project_app cookbook-efforts/domain-...ic-datasets/project_app	pipeline.yaml pipeline.yaml	90:150 (11%) 266:326 (11%)	view
61	x 2	cookbook-efforts/domain-...ic-datasets/project_app cookbook-efforts/domain-...ic-datasets/project_app	pipeline.yaml pipeline.yaml	266:326 (11%) 402:462 (11%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	494:544 (3%) 951:1001 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	599:649 (3%) 1113:1163 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	599:649 (3%) 1196:1246 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	1061:1111 (3%) 1248:1298 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	1061:1111 (3%) 1196:1246 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	599:649 (3%) 1061:1111 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	761:811 (3%) 951:1001 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	1061:1111 (3%) 1113:1163 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	494:544 (3%) 761:811 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	494:544 (3%) 709:759 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	709:759 (3%) 761:811 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	494:544 (3%) 599:649 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	761:811 (3%) 1351:1401 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	599:649 (3%) 951:1001 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	1248:1298 (3%) 1351:1401 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	709:759 (3%) 951:1001 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	709:759 (3%) 1061:1111 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	599:649 (3%) 709:759 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	599:649 (3%) 761:811 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	599:649 (3%) 1351:1401 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	709:759 (3%) 1113:1163 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	709:759 (3%) 1196:1246 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	709:759 (3%) 1248:1298 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	599:649 (3%) 1248:1298 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	1113:1163 (3%) 1248:1298 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	494:544 (3%) 1351:1401 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	709:759 (3%) 1351:1401 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	1113:1163 (3%) 1351:1401 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	761:811 (3%) 1061:1111 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	494:544 (3%) 1248:1298 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	761:811 (3%) 1113:1163 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	761:811 (3%) 1196:1246 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	494:544 (3%) 1113:1163 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	494:544 (3%) 1196:1246 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	1113:1163 (3%) 1196:1246 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	761:811 (3%) 1248:1298 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	951:1001 (3%) 1061:1111 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	494:544 (3%) 1061:1111 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	951:1001 (3%) 1196:1246 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	1196:1246 (3%) 1248:1298 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	951:1001 (3%) 1113:1163 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	951:1001 (3%) 1248:1298 (3%)	view
51	x 2	community-efforts/prompt_translation community-efforts/prompt_translation	02_upload_prompt_translation_data.ipynb 02_upload_prompt_translation_data.ipynb	1196:1246 (3%) 1351:1401 (3%)	view

Duplicated Units

The list of top 1 duplicated units.

See data for all 1 unit duplicate

Size	#	Folders	Files	Lines	Code
30	x 2	community-efforts/image_preferences community-efforts/image_preferences	01_synthetic_data_generation_total.py 01_synthetic_data_generation_images.py	0:0 0:0	view