huggingface / evaluate Duplication

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

25% duplication:

8,236 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
2,119 duplicated lines

589 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Duplication Between Components (50+ lines)

Download: SVG DOT (open online Graphviz editor)

From Component --> To Component	Duplicated Lines	File Pairs	Details
measurements (17%) --> metrics (1%)	148	4 file pairs	details...

Open 2D force graph... Open 3D force graph...

Show more details on duplication between components...

Longest Duplicates

The list of 50 longest duplicates.

See data for all 589 duplicates...

Size	#	Folders	Files	Lines	Code
134	x 2	metrics/sari metrics/wiki_split	sari.py wiki_split.py	82:257 (81%) 109:284 (59%)	view
37	x 2	src/evaluate/evaluator src/evaluate/evaluator	text2text_generation.py text2text_generation.py	171:208 (24%) 230:267 (24%)	view
36	x 2	metrics/bleu metrics/google_bleu	tokenizer_13a.py tokenizer_13a.py	20:100 (100%) 20:100 (100%)	view
30	x 2	metrics/squad metrics/super_glue	compute_score.py record_evaluation.py	11:52 (46%) 15:56 (41%)	view
24	x 2	measurements/perplexity metrics/perplexity	perplexity.py perplexity.py	106:141 (28%) 105:140 (28%)	view
23	x 2	src/evaluate/evaluator src/evaluate/evaluator	audio_classification.py image_classification.py	92:117 (46%) 60:85 (46%)	view
21	x 2	measurements/perplexity metrics/perplexity	perplexity.py perplexity.py	143:168 (25%) 142:167 (25%)	view
21	x 2	src/evaluate/evaluator src/evaluate/evaluator	audio_classification.py image_classification.py	119:151 (42%) 87:119 (42%)	view
20	x 2	src/evaluate/evaluator src/evaluate/evaluator	text2text_generation.py text2text_generation.py	110:129 (13%) 230:249 (13%)	view
20	x 2	metrics/cuad metrics/squad	cuad.py squad.py	97:117 (40%) 91:111 (43%)	view
20	x 2	src/evaluate/evaluator src/evaluate/evaluator	automatic_speech_recognition.py text2text_generation.py	82:112 (42%) 128:149 (13%)	view
20	x 2	src/evaluate/evaluator src/evaluate/evaluator	text2text_generation.py text2text_generation.py	110:129 (13%) 171:190 (13%)	view
19	x 2	metrics/bleu metrics/wiki_split	bleu.py wiki_split.py	89:107 (40%) 325:343 (8%)	view
19	x 2	metrics/bleu metrics/meteor	bleu.py meteor.py	89:107 (40%) 93:111 (23%)	view
19	x 2	metrics/meteor metrics/wiki_split	meteor.py wiki_split.py	93:111 (23%) 325:343 (8%)	view
18	x 2	metrics/google_bleu metrics/wiki_split	google_bleu.py wiki_split.py	130:147 (42%) 326:343 (7%)	view
18	x 2	src/evaluate/evaluator src/evaluate/evaluator	base.py text_classification.py	246:268 (7%) 134:155 (24%)	view
18	x 2	metrics/bleu metrics/google_bleu	bleu.py google_bleu.py	90:107 (38%) 130:147 (42%)	view
18	x 2	src/evaluate/evaluator src/evaluate/evaluator	image_classification.py text_classification.py	68:85 (36%) 89:106 (24%)	view
18	x 2	metrics/google_bleu metrics/meteor	google_bleu.py meteor.py	130:147 (42%) 94:111 (22%)	view
18	x 2	src/evaluate/evaluator src/evaluate/evaluator	audio_classification.py text_classification.py	100:117 (36%) 89:106 (24%)	view
17	x 2	src/evaluate/evaluator src/evaluate/evaluator	base.py text_classification.py	218:234 (6%) 91:107 (22%)	view
17	x 2	src/evaluate/evaluator src/evaluate/evaluator	automatic_speech_recognition.py question_answering.py	63:79 (36%) 147:163 (14%)	view
17	x 2	measurements/perplexity metrics/perplexity	perplexity.py perplexity.py	171:193 (20%) 170:192 (20%)	view
16	x 2	metrics/accuracy metrics/matthews_correlation	accuracy.py matthews_correlation.py	82:97 (55%) 103:118 (38%)	view
16	x 2	metrics/confusion_matrix metrics/f1	confusion_matrix.py f1.py	64:79 (55%) 107:122 (57%)	view
16	x 2	metrics/accuracy metrics/precision	accuracy.py precision.py	82:97 (55%) 107:122 (37%)	view
16	x 2	metrics/precision metrics/recall	precision.py recall.py	107:122 (37%) 97:112 (37%)	view
16	x 2	metrics/f1 metrics/matthews_correlation	f1.py matthews_correlation.py	107:122 (57%) 103:118 (38%)	view
16	x 2	metrics/confusion_matrix metrics/precision	confusion_matrix.py precision.py	64:79 (55%) 107:122 (37%)	view
16	x 2	metrics/confusion_matrix metrics/matthews_correlation	confusion_matrix.py matthews_correlation.py	64:79 (55%) 103:118 (38%)	view
16	x 2	metrics/matthews_correlation metrics/recall	matthews_correlation.py recall.py	103:118 (38%) 97:112 (37%)	view
16	x 2	metrics/accuracy metrics/confusion_matrix	accuracy.py confusion_matrix.py	82:97 (55%) 64:79 (55%)	view
16	x 2	metrics/matthews_correlation metrics/precision	matthews_correlation.py precision.py	103:118 (38%) 107:122 (37%)	view
16	x 2	metrics/f1 metrics/precision	f1.py precision.py	107:122 (57%) 107:122 (37%)	view
16	x 2	src/evaluate/evaluator src/evaluate/evaluator	base.py image_classification.py	218:233 (6%) 70:85 (32%)	view
16	x 2	metrics/confusion_matrix metrics/recall	confusion_matrix.py recall.py	64:79 (55%) 97:112 (37%)	view
16	x 2	metrics/f1 metrics/recall	f1.py recall.py	107:122 (57%) 97:112 (37%)	view
16	x 2	metrics/accuracy metrics/recall	accuracy.py recall.py	82:97 (55%) 97:112 (37%)	view
16	x 2	src/evaluate/evaluator src/evaluate/evaluator	audio_classification.py base.py	102:117 (32%) 218:233 (6%)	view
16	x 2	metrics/accuracy metrics/f1	accuracy.py f1.py	82:97 (55%) 107:122 (57%)	view
15	x 2	metrics/bertscore metrics/google_bleu	bertscore.py google_bleu.py	107:121 (12%) 133:147 (35%)	view
15	x 2	metrics/google_bleu metrics/sacrebleu	google_bleu.py sacrebleu.py	133:147 (35%) 117:131 (20%)	view
15	x 2	metrics/bertscore metrics/ter	bertscore.py ter.py	107:121 (12%) 165:179 (25%)	view
15	x 2	metrics/bleu metrics/chrf	bleu.py chrf.py	93:107 (31%) 138:152 (24%)	view
15	x 2	metrics/bertscore metrics/sacrebleu	bertscore.py sacrebleu.py	107:121 (12%) 117:131 (20%)	view
15	x 2	metrics/bertscore metrics/meteor	bertscore.py meteor.py	107:121 (12%) 97:111 (18%)	view
15	x 2	src/evaluate/evaluator src/evaluate/evaluator	automatic_speech_recognition.py text2text_generation.py	65:79 (31%) 111:125 (10%)	view
15	x 2	metrics/bertscore metrics/chrf	bertscore.py chrf.py	107:121 (12%) 138:152 (24%)	view
15	x 2	src/evaluate/evaluator src/evaluate/evaluator	automatic_speech_recognition.py text2text_generation.py	65:79 (31%) 172:186 (10%)	view

Duplicated Units

The list of top 4 duplicated units.

See data for all 4 unit duplicates...

Size	#	Folders	Files	Lines	Code
60	x 2	metrics/wiki_split metrics/sari	wiki_split.py sari.py	0:0 0:0	view
13	x 2	measurements/text_duplicates measurements/word_length	text_duplicates.py word_length.py	0:0 0:0	view
9	x 4	metrics/mase metrics/mape metrics/mae metrics/mse	mase.py mape.py mae.py mse.py	0:0 0:0 0:0 0:0	view
14	x 2	metrics/bleu metrics/google_bleu	tokenizer_13a.py tokenizer_13a.py	0:0 0:0	view