huggingface / fineweb-2 Duplication

huggingface / fineweb-2

Places in code with 6 or more lines that are exactly the same.

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Duplication Overall

24% duplication:

1,803 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
441 duplicated lines

18 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Longest Duplicates

The list of 18 longest duplicates.

See data for all 18 duplicates...

Size	#	Folders	Files	Lines	Code
54	x 2	misc/reference_datasets/monolingual/ar misc/reference_datasets/monolingual/fr	download_arabicweb24.py download_croissant.py	8:67 (79%) 7:66 (78%)	view
34	x 2	misc/reference_datasets/monolingual/zh misc/reference_datasets/multilingual	download_mapcc.py part jsons.py	46:89 (26%) 56:98 (32%)	view
31	x 2	misc/reference_datasets/monolingual/zh misc/reference_datasets/multilingual	download_mapcc.py part jsons.py	10:44 (24%) 9:43 (29%)	view
15	x 2	misc/reference_datasets/multilingual misc/reference_datasets/multilingual	copy_raw_data.py part jsons.py	96:114 (15%) 115:132 (14%)	view
14	x 2	misc/reference_datasets/monolingual/zh misc/reference_datasets/multilingual	download_mapcc.py part jsons.py	90:103 (10%) 100:113 (13%)	view
12	x 2	misc/reference_datasets/monolingual/ar misc/reference_datasets/multilingual	download_arabicweb24.py copy_raw_data.py	33:44 (17%) 49:60 (12%)	view
12	x 2	misc/reference_datasets/monolingual/fr misc/reference_datasets/multilingual	download_croissant.py copy_raw_data.py	32:43 (17%) 49:60 (12%)	view
11	x 2	misc/precision_filtering misc/precision_filtering	count_common.py wordlist_gen.py	7:27 (27%) 66:86 (21%)	view
10	x 2	misc/reference_datasets/monolingual/hi misc/reference_datasets/monolingual/te	download_sangraha.py download_sangraha.py	11:20 (55%) 11:20 (31%)	view
9	x 2	misc/precision_filtering misc/precision_filtering	slurm_count_word.py wordlist_score.py	37:45 (14%) 41:49 (23%)	view
9	x 2	ablations/evaluation ablations/evaluation	launch_evals.py launch_random_evals.py	49:66 (7%) 11:28 (20%)	view
8	x 2	misc/precision_filtering misc/precision_filtering	run_precision_filtering.py wordlist_score.py	18:25 (8%) 42:49 (21%)	view
8	x 2	misc/precision_filtering misc/precision_filtering	run_precision_filtering.py slurm_count_word.py	18:25 (8%) 38:45 (12%)	view
7	x 2	misc/reference_datasets/monolingual/ar misc/reference_datasets/monolingual/ar	download_101b_arabicwords.py download_arabicweb24.py	13:19 (46%) 75:81 (10%)	view
7	x 2	ablations/evaluation ablations/evaluation	launch_evals.py launch_random_evals.py	189:269 (5%) 58:106 (15%)	view
7	x 2	misc/precision_filtering misc/precision_filtering	slurm_count_word.py wordlist_gen.py	4:17 (10%) 66:77 (13%)	view
7	x 2	misc/precision_filtering misc/precision_filtering	count_common.py slurm_count_word.py	7:18 (17%) 4:17 (10%)	view
6	x 2	misc/reference_datasets/monolingual/te misc/reference_datasets/monolingual/th	download_sangraha.py download_sea_commoncrawl.py	36:41 (18%) 14:19 (46%)	view

Duplicated Units

The list of top 1 duplicated units.

See data for all 1 unit duplicate

Size	#	Folders	Files	Lines	Code
15	x 2	misc/reference_datasets/monolingual/ar misc/reference_datasets/monolingual/fr	download_arabicweb24.py download_croissant.py	0:0 0:0	view