huggingface / datasets Duplication

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

16% duplication:

19,238 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
3,218 duplicated lines

324 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Longest Duplicates

The list of 50 longest duplicates.

See data for all 324 duplicates...

Size	#	Folders	Files	Lines	Code
137	x 2	src/datasets src/datasets	dataset_dict.py dataset_dict.py	1774:1920 (13%) 2590:2736 (13%)	view
56	x 2	src/datasets src/datasets	builder.py builder.py	1440:1506 (4%) 1695:1761 (4%)	view
42	x 2	src/datasets src/datasets	dataset_dict.py dataset_dict.py	1619:1756 (4%) 2445:2572 (4%)	view
38	x 2	src/datasets/features src/datasets/features	audio.py pdf.py	240:290 (26%) 207:257 (30%)	view
38	x 2	src/datasets src/datasets	dataset_dict.py iterable_dataset.py	1882:1920 (3%) 4225:4263 (1%)	view
38	x 2	src/datasets src/datasets	dataset_dict.py iterable_dataset.py	2698:2736 (3%) 4225:4263 (1%)	view
36	x 2	src/datasets/features src/datasets/features	image.py video.py	196:253 (15%) 224:281 (21%)	view
28	x 2	src/datasets/features src/datasets/features	audio.py image.py	250:290 (19%) 253:293 (12%)	view
28	x 2	src/datasets/features src/datasets/features	image.py pdf.py	253:293 (12%) 217:257 (22%)	view
25	x 2	src/datasets src/datasets	builder.py builder.py	1409:1437 (2%) 1666:1694 (2%)	view
25	x 2	src/datasets/features src/datasets/features	pdf.py video.py	173:216 (19%) 228:271 (14%)	view
25	x 2	src/datasets/download src/datasets	streaming_download_manager.py streaming.py	15:44 (25%) 7:35 (33%)	view
25	x 2	src/datasets/features src/datasets/features	image.py pdf.py	200:243 (10%) 173:216 (19%)	view
24	x 2	src/datasets/io src/datasets/io	csv.py text.py	39:66 (21%) 33:60 (50%)	view
23	x 2	src/datasets/packaged_modules/text src/datasets/packaged_modules/xml	text.py xml.py	31:62 (30%) 27:58 (58%)	view
22	x 2	src/datasets/io src/datasets/io	csv.py text.py	16:37 (19%) 10:31 (45%)	view
22	x 2	src/datasets/io src/datasets/io	parquet.py text.py	18:39 (24%) 10:31 (45%)	view
22	x 2	src/datasets src/datasets	combine.py combine.py	124:145 (27%) 190:211 (27%)	view
22	x 2	src/datasets/io src/datasets/io	csv.py parquet.py	16:37 (19%) 18:39 (24%)	view
21	x 2	src/datasets/io src/datasets/io	json.py text.py	45:69 (15%) 36:60 (43%)	view
21	x 2	src/datasets/io src/datasets/io	csv.py json.py	42:66 (18%) 45:69 (15%)	view
21	x 2	src/datasets/io src/datasets/io	csv.py parquet.py	42:66 (18%) 46:70 (23%)	view
21	x 2	src/datasets/utils src/datasets/utils	_dill.py _dill.py	273:293 (6%) 442:462 (6%)	view
21	x 2	src/datasets/io src/datasets/io	json.py parquet.py	45:69 (15%) 46:70 (23%)	view
21	x 2	src/datasets/io src/datasets/io	parquet.py text.py	46:70 (23%) 36:60 (43%)	view
20	x 2	src/datasets src/datasets	iterable_dataset.py iterable_dataset.py	1589:1612 (<1%) 1713:1736 (<1%)	view
18	x 2	src/datasets/packaged_modules/folder_based_builder src/datasets/packaged_modules/json	folder_based_builder.py json.py	314:333 (5%) 135:154 (13%)	view
18	x 2	src/datasets src/datasets	iterable_dataset.py iterable_dataset.py	323:341 (<1%) 395:413 (<1%)	view
18	x 2	src/datasets/packaged_modules/csv src/datasets/packaged_modules/text	csv.py text.py	148:168 (11%) 31:55 (24%)	view
18	x 2	src/datasets/utils src/datasets/utils	_dill.py _dill.py	279:296 (5%) 430:447 (5%)	view
18	x 2	src/datasets/packaged_modules/csv src/datasets/packaged_modules/xml	csv.py xml.py	148:168 (11%) 27:51 (46%)	view
17	x 2	src/datasets/utils src/datasets/utils	_dill.py _dill.py	361:377 (5%) 385:401 (5%)	view
15	x 2	src/datasets src/datasets	dataset_dict.py iterable_dataset.py	2659:2674 (1%) 4192:4207 (<1%)	view
15	x 2	src/datasets/packaged_modules/json src/datasets/packaged_modules/text	json.py text.py	71:88 (11%) 32:53 (20%)	view
15	x 2	src/datasets/packaged_modules/json src/datasets/packaged_modules/xml	json.py xml.py	71:88 (11%) 28:49 (38%)	view
15	x 2	src/datasets src/datasets	dataset_dict.py iterable_dataset.py	1843:1858 (1%) 4192:4207 (<1%)	view
15	x 2	src/datasets/utils src/datasets/utils	_dill.py _dill.py	430:444 (4%) 448:462 (4%)	view
15	x 2	src/datasets src/datasets	builder.py builder.py	1588:1603 (1%) 1840:1855 (1%)	view
15	x 2	src/datasets/packaged_modules/csv src/datasets/packaged_modules/json	csv.py json.py	149:166 (9%) 71:88 (11%)	view
14	x 2	src/datasets/utils src/datasets/utils	_dill.py _dill.py	361:374 (4%) 407:420 (4%)	view
14	x 2	src/datasets src/datasets	iterable_dataset.py iterable_dataset.py	2912:2925 (<1%) 2992:3005 (<1%)	view
14	x 2	src/datasets/packaged_modules/arrow src/datasets/packaged_modules/parquet	arrow.py parquet.py	28:45 (28%) 42:59 (17%)	view
14	x 2	src/datasets src/datasets	dataset_dict.py iterable_dataset.py	2679:2693 (1%) 4209:4223 (<1%)	view
14	x 2	src/datasets/utils src/datasets/utils	_dill.py _dill.py	297:310 (4%) 431:444 (4%)	view
14	x 2	src/datasets/utils src/datasets/utils	_dill.py _dill.py	297:310 (4%) 449:462 (4%)	view
14	x 2	src/datasets/utils src/datasets/utils	_dill.py _dill.py	258:271 (4%) 427:440 (4%)	view
14	x 2	src/datasets src/datasets	dataset_dict.py iterable_dataset.py	1863:1877 (1%) 4209:4223 (<1%)	view
14	x 2	src/datasets/utils src/datasets/utils	_dill.py _dill.py	385:398 (4%) 407:420 (4%)	view
14	x 2	src/datasets/utils src/datasets/utils	_dill.py _dill.py	280:293 (4%) 297:310 (4%)	view
13	x 2	src/datasets/utils src/datasets/utils	_dill.py _dill.py	259:271 (4%) 409:421 (4%)	view

Duplicated Units

The list of top 1 duplicated units.

See data for all 1 unit duplicate

Size	#	Folders	Files	Lines	Code
19	x 2	src/datasets/io src/datasets/io	text.py parquet.py	0:0 0:0	view