huggingface / cosmopedia Duplication

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

9% duplication:

2,446 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
241 duplicated lines

35 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Longest Duplicates

The list of 35 longest duplicates.

See data for all 35 duplicates...

Size	#	Folders	Files	Lines	Code
24	x 2	prompts/stanford prompts/stanford	1_scraper.ipynb 2_generate_course_outlines.ipynb	376:399 (6%) 264:287 (8%)	view
16	x 2	prompts/khanacademy/khan_dl prompts/khanacademy/khan_dl	khan_dl.py khan_dl.py	91:106 (5%) 247:262 (5%)	view
13	x 2	prompts/khanacademy/khan_dl prompts/khanacademy/khan_dl	khan_dl.py khan_dl.py	92:104 (4%) 175:187 (4%)	view
13	x 2	prompts/stories prompts/stories	build_openhermes_stories_prompts.py build_ultrachat_stories_prompts.py	57:69 (32%) 75:87 (29%)	view
13	x 2	prompts/khanacademy/khan_dl prompts/khanacademy/khan_dl	khan_dl.py khan_dl.py	175:187 (4%) 248:260 (4%)	view
11	x 2	prompts/auto_math_text prompts/web_samples	build_science_prompts.py build_web_prompts.py	47:57 (30%) 73:83 (24%)	view
11	x 2	prompts/stories prompts/web_samples	build_openhermes_stories_prompts.py build_web_prompts.py	57:67 (27%) 73:83 (24%)	view
11	x 2	prompts/stories prompts/web_samples	build_ultrachat_stories_prompts.py build_web_prompts.py	75:85 (25%) 73:83 (24%)	view
11	x 2	prompts/auto_math_text prompts/stories	build_science_prompts.py build_openhermes_stories_prompts.py	47:57 (30%) 57:67 (27%)	view
11	x 2	prompts/auto_math_text prompts/stories	build_science_prompts.py build_ultrachat_stories_prompts.py	47:57 (30%) 75:85 (25%)	view
10	x 2	prompts/khanacademy/khan_dl prompts/khanacademy/khan_dl	khan_dl.py khan_dl.py	66:75 (3%) 435:444 (3%)	view
7	x 2	prompts/openstax prompts/openstax	build_openstax_prompts.py build_openstax_prompts.py	123:129 (3%) 151:157 (3%)	view
6	x 2	prompts/stanford prompts/stanford	2_generate_course_outlines.ipynb 2_generate_course_outlines.ipynb	27:32 (2%) 102:107 (2%)	view
6	x 2	prompts/stanford prompts/stanford	2_generate_course_outlines.ipynb 2_generate_course_outlines.ipynb	127:132 (2%) 215:220 (2%)	view
6	x 2	prompts/stories prompts/stories	build_openhermes_stories_prompts.py build_ultrachat_stories_prompts.py	21:37 (15%) 39:55 (13%)	view
6	x 2	prompts/stanford prompts/stanford	2_generate_course_outlines.ipynb 2_generate_course_outlines.ipynb	71:76 (2%) 215:220 (2%)	view
6	x 2	prompts/stanford prompts/stanford	2_generate_course_outlines.ipynb 2_generate_course_outlines.ipynb	102:107 (2%) 215:220 (2%)	view
6	x 2	prompts/stanford prompts/stanford	2_generate_course_outlines.ipynb 2_generate_course_outlines.ipynb	90:95 (2%) 252:257 (2%)	view
6	x 2	prompts/stanford prompts/stanford	2_generate_course_outlines.ipynb 2_generate_course_outlines.ipynb	27:32 (2%) 215:220 (2%)	view
6	x 2	prompts/stanford prompts/stanford	2_generate_course_outlines.ipynb 2_generate_course_outlines.ipynb	71:76 (2%) 127:132 (2%)	view
6	x 2	prompts/stanford prompts/stanford	2_generate_course_outlines.ipynb 2_generate_course_outlines.ipynb	71:76 (2%) 90:95 (2%)	view
6	x 2	prompts/stanford prompts/stanford	2_generate_course_outlines.ipynb 2_generate_course_outlines.ipynb	90:95 (2%) 127:132 (2%)	view
6	x 2	prompts/stanford prompts/stanford	2_generate_course_outlines.ipynb 2_generate_course_outlines.ipynb	127:132 (2%) 252:257 (2%)	view
6	x 2	prompts/stanford prompts/stanford	2_generate_course_outlines.ipynb 2_generate_course_outlines.ipynb	27:32 (2%) 71:76 (2%)	view
6	x 2	prompts/stanford prompts/stanford	2_generate_course_outlines.ipynb 2_generate_course_outlines.ipynb	71:76 (2%) 252:257 (2%)	view
6	x 2	prompts/stanford prompts/stanford	2_generate_course_outlines.ipynb 2_generate_course_outlines.ipynb	90:95 (2%) 215:220 (2%)	view
6	x 2	prompts/stanford prompts/stanford	2_generate_course_outlines.ipynb 2_generate_course_outlines.ipynb	27:32 (2%) 127:132 (2%)	view
6	x 2	prompts/stanford prompts/stanford	2_generate_course_outlines.ipynb 2_generate_course_outlines.ipynb	102:107 (2%) 252:257 (2%)	view
6	x 2	prompts/stanford prompts/stanford	2_generate_course_outlines.ipynb 2_generate_course_outlines.ipynb	71:76 (2%) 102:107 (2%)	view
6	x 2	prompts/stanford prompts/stanford	2_generate_course_outlines.ipynb 2_generate_course_outlines.ipynb	102:107 (2%) 127:132 (2%)	view
6	x 2	prompts/stanford prompts/stanford	2_generate_course_outlines.ipynb 2_generate_course_outlines.ipynb	27:32 (2%) 252:257 (2%)	view
6	x 2	prompts/stanford prompts/stanford	2_generate_course_outlines.ipynb 2_generate_course_outlines.ipynb	90:95 (2%) 102:107 (2%)	view
6	x 2	prompts/stanford prompts/stanford	2_generate_course_outlines.ipynb 2_generate_course_outlines.ipynb	215:220 (2%) 252:257 (2%)	view
6	x 2	prompts/stories prompts/stories	build_openhermes_stories_prompts.py build_ultrachat_stories_prompts.py	47:55 (15%) 65:73 (13%)	view
6	x 2	prompts/stanford prompts/stanford	2_generate_course_outlines.ipynb 2_generate_course_outlines.ipynb	27:32 (2%) 90:95 (2%)	view