huggingface / OBELICS Duplication

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

17% duplication:

7,414 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
1,307 duplicated lines

533 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Duplication Between Components (50+ lines)

Download: SVG DOT (open online Graphviz editor)

From Component --> To Component	Duplicated Lines	File Pairs	Details
build_obelics (14%) --> obelics (8%)	801	143 file pairs	details...

Open 2D force graph... Open 3D force graph...

Show more details on duplication between components...

Longest Duplicates

The list of 50 longest duplicates.

See data for all 533 duplicates...

Size	#	Folders	Files	Lines	Code
111	x 2	build_obelics obelics/callers	05_filtering_web_docs.py filter_web_documents.py	113:228 (51%) 93:208 (59%)	view
24	x 2	build_obelics build_obelics	02_bis_extract_html_get_image_urls_ne... 02_extract_html_get_image_urls.py	14:41 (18%) 13:40 (17%)	view
21	x 2	build_obelics obelics/callers	02_bis_extract_html_get_image_urls_ne... extract_web_documents.py	96:117 (15%) 125:146 (12%)	view
21	x 2	build_obelics build_obelics	02_bis_extract_html_get_image_urls_ne... 02_extract_html_get_image_urls.py	68:91 (15%) 75:98 (15%)	view
19	x 2	build_obelics obelics/callers	01_download_warc.py download_warc.py	66:86 (22%) 48:68 (32%)	view
18	x 2	build_obelics build_obelics	10_final_cleaning.py 13_final_processing.py	61:79 (17%) 190:208 (6%)	view
14	x 2	obelics/visualization obelics/visualization	web_document_and_filtering_visualizat... web_document_visualization.py	684:697 (2%) 35:48 (25%)	view
13	x 2	build_obelics build_obelics	02_bis_extract_html_get_image_urls_ne... 02_extract_html_get_image_urls.py	125:138 (9%) 125:138 (9%)	view
12	x 2	build_obelics build_obelics	02_bis_extract_html_get_image_urls_ne... 02_extract_html_get_image_urls.py	45:58 (9%) 50:63 (8%)	view
12	x 2	build_obelics build_obelics	06_01_create_set_image_urls_in_webdoc... 06_03_remove_image_duplicates.py	23:34 (14%) 26:37 (10%)	view
12	x 2	build_obelics build_obelics	08_02_urldedup.py 11_03_set_img_urls_dedup.py	11:27 (15%) 13:29 (13%)	view
12	x 2	build_obelics obelics/callers	02_extract_html_get_image_urls.py extract_html.py	50:63 (8%) 31:44 (25%)	view
12	x 2	build_obelics obelics/visualization	02_extract_html_get_image_urls.py global_visualization.py	101:112 (8%) 51:62 (3%)	view
12	x 2	build_obelics obelics/callers	02_bis_extract_html_get_image_urls_ne... extract_html.py	45:58 (9%) 31:44 (25%)	view
11	x 2	build_obelics build_obelics	07_01_nsfw_image_filtering.py 12_02_remove_opt_out_images.py	20:34 (6%) 12:26 (16%)	view
11	x 2	build_obelics obelics/callers	02_extract_html_get_image_urls.py download_warc.py	51:63 (7%) 32:44 (18%)	view
11	x 2	build_obelics obelics/callers	06_01_create_set_image_urls_in_webdoc... download_warc.py	39:51 (13%) 32:44 (18%)	view
11	x 2	build_obelics build_obelics	07_03_nsfw_image_removal.py 12_02_remove_opt_out_images.py	12:26 (15%) 12:26 (16%)	view
11	x 2	build_obelics obelics/callers	06_01_create_set_image_urls_in_webdoc... extract_html.py	39:51 (13%) 32:44 (22%)	view
11	x 2	build_obelics build_obelics	10_final_cleaning.py 13_final_processing.py	124:135 (10%) 315:326 (4%)	view
11	x 2	build_obelics build_obelics	09_07_merge_web_docs_texts_only_and_r... 10_final_cleaning.py	11:26 (18%) 14:29 (10%)	view
11	x 2	build_obelics build_obelics	02_extract_html_get_image_urls.py 06_01_create_set_image_urls_in_webdoc...	51:63 (7%) 39:51 (13%)	view
11	x 2	build_obelics build_obelics	02_bis_extract_html_get_image_urls_ne... 06_01_create_set_image_urls_in_webdoc...	46:58 (8%) 39:51 (13%)	view
11	x 2	obelics/processors obelics/visualization	dom_tree_simplificator.py global_visualization.py	15:25 (5%) 51:61 (3%)	view
11	x 2	build_obelics obelics/callers	02_bis_extract_html_get_image_urls_ne... download_warc.py	46:58 (8%) 32:44 (18%)	view
11	x 2	build_obelics obelics/processors	02_extract_html_get_image_urls.py dom_tree_simplificator.py	101:111 (7%) 15:25 (5%)	view
11	x 2	build_obelics build_obelics	07_01_nsfw_image_filtering.py 07_03_nsfw_image_removal.py	20:34 (6%) 12:26 (15%)	view
11	x 2	obelics/callers obelics/callers	download_warc.py extract_html.py	32:44 (18%) 32:44 (22%)	view
10	x 2	build_obelics obelics/callers	02_bis_extract_html_get_image_urls_ne... extract_web_documents.py	13:26 (7%) 11:24 (5%)	view
10	x 2	build_obelics build_obelics	06_01_create_set_image_urls_in_webdoc... 06_03_remove_image_duplicates.py	59:69 (12%) 102:112 (8%)	view
10	x 2	build_obelics build_obelics	09_01_create_web_docs_texts_only.py 10_final_cleaning.py	9:21 (22%) 19:31 (9%)	view
10	x 2	build_obelics build_obelics	05_filtering_web_docs.py 06_03_remove_image_duplicates.py	23:36 (4%) 11:24 (8%)	view
10	x 2	build_obelics build_obelics	11_01_create_set_img_urls.py 13_final_processing.py	30:41 (22%) 266:277 (3%)	view
10	x 2	build_obelics obelics/callers	01_download_warc.py download_warc.py	11:22 (12%) 10:21 (17%)	view
10	x 2	build_obelics build_obelics	01_download_warc.py 02_extract_html_get_image_urls.py	43:55 (12%) 56:68 (7%)	view
10	x 2	build_obelics obelics/callers	05_filtering_web_docs.py filter_web_documents.py	54:63 (4%) 47:56 (5%)	view
9	x 2	build_obelics obelics/callers	04_merge_web_docs_with_images.py extract_html.py	59:67 (6%) 32:40 (18%)	view
9	x 2	build_obelics build_obelics	09_07_merge_web_docs_texts_only_and_r... 12_02_remove_opt_out_images.py	11:22 (15%) 12:22 (13%)	view
9	x 2	build_obelics build_obelics	06_03_remove_image_duplicates.py 09_07_merge_web_docs_texts_only_and_r...	11:21 (7%) 11:22 (15%)	view
9	x 2	build_obelics obelics/callers	03_dl_images_create_dataset.py extract_web_documents.py	53:61 (5%) 66:74 (5%)	view
9	x 2	build_obelics build_obelics	09_07_merge_web_docs_texts_only_and_r... 11_03_set_img_urls_dedup.py	11:22 (15%) 13:23 (10%)	view
9	x 2	build_obelics build_obelics	10_final_cleaning.py 11_03_set_img_urls_dedup.py	14:25 (8%) 13:23 (10%)	view
9	x 2	build_obelics obelics/callers	04_merge_web_docs_with_images.py download_warc.py	59:67 (6%) 32:40 (15%)	view
9	x 2	build_obelics build_obelics	12_02_remove_opt_out_images.py 13_final_processing.py	12:22 (13%) 18:29 (3%)	view
9	x 2	build_obelics build_obelics	05_filtering_web_docs.py 06_01_create_set_image_urls_in_webdoc...	90:101 (4%) 45:56 (10%)	view
9	x 2	build_obelics build_obelics	07_03_nsfw_image_removal.py 08_02_urldedup.py	12:22 (12%) 11:21 (11%)	view
9	x 2	build_obelics build_obelics	06_03_remove_image_duplicates.py 08_02_urldedup.py	11:21 (7%) 11:21 (11%)	view
9	x 2	build_obelics build_obelics	02_extract_html_get_image_urls.py 04_merge_web_docs_with_images.py	16:26 (6%) 18:28 (6%)	view
9	x 2	build_obelics build_obelics	05_filtering_web_docs.py 13_final_processing.py	23:33 (4%) 18:29 (3%)	view
9	x 2	build_obelics build_obelics	06_03_remove_image_duplicates.py 11_03_set_img_urls_dedup.py	11:21 (7%) 13:23 (10%)	view