huggingface / datatrove

Duplication

Places in code with 6 or more lines that are exactly the same.

Longest Duplicates

The list of 50 longest duplicates.

See data for all 61 duplicates...

Size	#	Folders	Files	Lines	Code
51	x 2	src/datatrove/tools/fast_mh3/src src/datatrove/tools/fast_mh3/src	local_union_find.rs s3_union_find.rs	298:367 (21%) 490:559 (11%)	view
33	x 2	src/datatrove/pipeline/readers src/datatrove/pipeline/readers	csv.py jsonl.py	34:66 (71%) 35:67 (64%)	view
25	x 2	src/datatrove/pipeline/readers src/datatrove/pipeline/readers	ipc.py jsonl.py	40:64 (42%) 42:66 (49%)	view
25	x 2	src/datatrove/pipeline/readers src/datatrove/pipeline/readers	jsonl.py parquet.py	42:66 (49%) 43:67 (47%)	view
25	x 2	src/datatrove/pipeline/readers src/datatrove/pipeline/readers	ipc.py parquet.py	40:64 (42%) 43:67 (47%)	view
25	x 2	src/datatrove/pipeline/readers src/datatrove/pipeline/readers	csv.py ipc.py	41:65 (54%) 40:64 (42%)	view
25	x 2	src/datatrove/pipeline/readers src/datatrove/pipeline/readers	csv.py parquet.py	41:65 (54%) 43:67 (47%)	view
23	x 2	src/datatrove/tools/fast_mh3/src src/datatrove/tools/fast_mh3/src	local_union_find.rs s3_union_find.rs	167:192 (9%) 352:378 (5%)	view
22	x 2	src/datatrove/executor src/datatrove/executor	base.py ray.py	85:111 (25%) 228:254 (15%)	view
20	x 2	src/datatrove/tools/fast_mh3/src src/datatrove/tools/fast_mh3/src	s3_union_find.rs s3_union_find.rs	373:394 (4%) 456:477 (4%)	view
20	x 2	src/datatrove/pipeline/dedup src/datatrove/pipeline/dedup	sentence_dedup.py url_dedup.py	246:267 (5%) 248:269 (7%)	view
18	x 2	src/datatrove/tools/fast_mh3/src src/datatrove/tools/fast_mh3/src	local_union_find.rs s3_union_find.rs	111:131 (7%) 283:303 (4%)	view
17	x 2	src/datatrove/tools/fast_mh3/src src/datatrove/tools/fast_mh3/src	local_union_find.rs s3_union_find.rs	223:247 (7%) 413:437 (3%)	view
17	x 2	src/datatrove/pipeline/readers src/datatrove/pipeline/readers	csv.py warc.py	34:50 (36%) 38:54 (19%)	view
17	x 2	src/datatrove/pipeline/readers src/datatrove/pipeline/readers	jsonl.py warc.py	35:51 (33%) 38:54 (19%)	view
15	x 2	src/datatrove/pipeline/readers src/datatrove/pipeline/readers	ipc.py warc.py	50:64 (25%) 56:70 (17%)	view
15	x 2	src/datatrove/pipeline/readers src/datatrove/pipeline/readers	jsonl.py warc.py	52:66 (29%) 56:70 (17%)	view
15	x 2	src/datatrove/pipeline/readers src/datatrove/pipeline/readers	csv.py warc.py	51:65 (32%) 56:70 (17%)	view
15	x 2	src/datatrove/pipeline/readers src/datatrove/pipeline/readers	parquet.py warc.py	53:67 (28%) 56:70 (17%)	view
14	x 2	src/datatrove/tools/fast_mh3/src src/datatrove/tools/fast_mh3/src	local_union_find.rs s3_union_find.rs	42:59 (5%) 83:100 (3%)	view
12	x 2	src/datatrove/pipeline/readers src/datatrove/pipeline/readers	base.py csv.py	139:150 (9%) 39:50 (26%)	view
12	x 2	src/datatrove/tools/fast_mh3/src src/datatrove/tools/fast_mh3/src	local_union_find.rs local_union_find.rs	305:320 (5%) 324:339 (5%)	view
12	x 2	src/datatrove/pipeline/readers src/datatrove/pipeline/readers	base.py jsonl.py	139:150 (9%) 40:51 (23%)	view
12	x 2	src/datatrove/pipeline/readers src/datatrove/pipeline/readers	base.py warc.py	139:150 (9%) 43:54 (13%)	view
12	x 2	src/datatrove/tools/fast_mh3/src src/datatrove/tools/fast_mh3/src	local_union_find.rs s3_union_find.rs	305:320 (5%) 516:531 (2%)	view
12	x 2	src/datatrove/tools/fast_mh3/src src/datatrove/tools/fast_mh3/src	local_union_find.rs s3_union_find.rs	324:339 (5%) 497:512 (2%)	view
12	x 2	src/datatrove/tools/fast_mh3/src src/datatrove/tools/fast_mh3/src	s3_union_find.rs s3_union_find.rs	497:512 (2%) 516:531 (2%)	view
11	x 2	src/datatrove/pipeline/dedup src/datatrove/pipeline/dedup	sentence_dedup.py url_dedup.py	218:230 (3%) 214:226 (3%)	view
11	x 2	src/datatrove/pipeline/dedup src/datatrove/pipeline/dedup	sentence_dedup.py url_dedup.py	146:170 (3%) 145:158 (3%)	view
10	x 2	src/datatrove/pipeline/readers src/datatrove/pipeline/readers	base.py parquet.py	141:150 (7%) 43:52 (18%)	view
10	x 2	src/datatrove/pipeline/dedup src/datatrove/pipeline/dedup	sentence_dedup.py url_dedup.py	235:244 (2%) 237:246 (3%)	view
10	x 2	src/datatrove/pipeline/readers src/datatrove/pipeline/readers	parquet.py warc.py	43:52 (18%) 45:54 (11%)	view
10	x 2	src/datatrove/pipeline/readers src/datatrove/pipeline/readers	base.py ipc.py	141:150 (7%) 40:49 (16%)	view
10	x 2	src/datatrove/pipeline/stats src/datatrove/pipeline/stats	paragraph_stats.py sentence_stats.py	38:47 (22%) 38:47 (25%)	view
10	x 2	src/datatrove/tools/fast_mh3/src src/datatrove/tools/fast_mh3/src	local_union_find.rs s3_union_find.rs	88:102 (4%) 258:272 (2%)	view
10	x 2	src/datatrove/pipeline/readers src/datatrove/pipeline/readers	ipc.py warc.py	40:49 (16%) 45:54 (11%)	view
10	x 2	src/datatrove/tools/fast_mh3/src src/datatrove/tools/fast_mh3/src	local_union_find.rs s3_union_find.rs	22:38 (4%) 40:56 (2%)	view
9	x 2	src/datatrove/pipeline/readers src/datatrove/pipeline/readers	ipc.py parquet.py	88:96 (15%) 79:87 (16%)	view
9	x 2	src/datatrove/pipeline/dedup src/datatrove/pipeline/dedup	sentence_dedup.py url_dedup.py	494:503 (2%) 406:416 (3%)	view
9	x 2	src/datatrove/pipeline/tokens src/datatrove/pipeline/tokens	merger.py merger.py	169:177 (6%) 186:194 (6%)	view
9	x 2	src/datatrove/tools/fast_mh3/src src/datatrove/tools/fast_mh3/src	local_union_find.rs s3_union_find.rs	259:268 (3%) 452:461 (2%)	view
8	x 2	src/datatrove/pipeline/stats src/datatrove/pipeline/stats	paragraph_stats.py word_stats.py	40:47 (17%) 47:54 (16%)	view
8	x 2	src/datatrove/pipeline/stats src/datatrove/pipeline/stats	sentence_stats.py word_stats.py	40:47 (20%) 47:54 (16%)	view
7	x 2	src/datatrove/pipeline/dedup src/datatrove/pipeline/dedup	sentence_dedup.py url_dedup.py	472:478 (2%) 384:390 (2%)	view
7	x 2	src/datatrove/pipeline/dedup src/datatrove/pipeline/dedup	sentence_dedup.py url_dedup.py	386:394 (2%) 321:329 (2%)	view
7	x 2	src/datatrove/pipeline/readers src/datatrove/pipeline/readers	ipc.py parquet.py	31:38 (11%) 33:40 (13%)	view
7	x 2	src/datatrove/pipeline/tokens src/datatrove/pipeline/tokens	tokenizer.py tokenizer.py	202:209 (2%) 235:241 (2%)	view
6	x 2	src/datatrove/pipeline/readers src/datatrove/pipeline/readers	base.py ipc.py	135:140 (4%) 33:38 (10%)	view
6	x 2	src/datatrove/tools/fast_mh3/src src/datatrove/tools/fast_mh3/src	local_union_find.rs local_union_find.rs	187:192 (2%) 263:268 (2%)	view
6	x 2	src/datatrove/tools/fast_mh3/src src/datatrove/tools/fast_mh3/src	local_union_find.rs s3_union_find.rs	263:268 (2%) 373:378 (1%)	view

Duplicated Units

The list of top 1 duplicated units.

See data for all 1 unit duplicate

Size	#	Folders	Files	Lines	Code
6	x 2	src/datatrove/tools/fast_mh3/src src/datatrove/tools/fast_mh3/src	s3_union_find.rs local_union_find.rs	95:102  54:61	view