huggingface / ratchet Duplication

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

18% duplication:

15,591 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
2,858 duplicated lines

331 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Longest Duplicates

The list of 50 longest duplicates.

See data for all 331 duplicates...

Size	#	Folders	Files	Lines	Code
35	x 2	crates/ratchet-models/src/phi2 crates/ratchet-models/src/phi3	attn.rs attn.rs	15:55 (28%) 13:53 (27%)	view
34	x 2	crates/ratchet-core/src/ops/matmul crates/ratchet-core/src/ops/matmul	gemm.rs workgroup_gemv.rs	133:170 (6%) 69:106 (11%)	view
30	x 2	crates/ratchet-models/src/phi3 crates/ratchet-models/src/phi3	generate.rs generate.rs	11:43 (29%) 67:99 (29%)	view
26	x 2	crates/ratchet-hub/src crates/ratchet-hub/src	lib.rs lib.rs	141:173 (11%) 217:248 (11%)	view
25	x 2	crates/ratchet-core/src/ops/norm crates/ratchet-core/src/ops	mod.rs unary.rs	339:369 (6%) 324:354 (7%)	view
25	x 2	crates/ratchet-models/src/phi2 crates/ratchet-models/src/phi3	model.rs model.rs	227:261 (9%) 269:303 (7%)	view
23	x 2	crates/ratchet-web/src crates/ratchet-web/src	model.rs model.rs	341:367 (7%) 380:406 (7%)	view
22	x 2	crates/ratchet-models/src/whisper crates/ratchet-models/src/whisper	decoder.rs decoder.rs	172:197 (7%) 227:252 (7%)	view
22	x 2	crates/ratchet-core/src/ops crates/ratchet-core/src/ops/norm	conv.rs mod.rs	34:57 (18%) 105:129 (5%)	view
21	x 2	crates/ratchet-core/src/ops crates/ratchet-core/src/ops	conv.rs softmax.rs	36:59 (17%) 51:74 (14%)	view
21	x 2	crates/ratchet-core/src/ops crates/ratchet-core/src/ops	binary.rs unary.rs	65:89 (8%) 85:109 (6%)	view
21	x 2	crates/ratchet-models/src/moondream crates/ratchet-models/src/phi3	text_model.rs attn.rs	33:58 (14%) 90:115 (16%)	view
20	x 2	crates/ratchet-models/src/phi2 crates/ratchet-models/src/phi3	model.rs model.rs	103:126 (7%) 132:155 (6%)	view
20	x 2	crates/ratchet-core/src/cpu crates/ratchet-core/src/cpu	norm.rs norm.rs	26:51 (11%) 150:175 (11%)	view
20	x 2	crates/ratchet-models/src/whisper crates/ratchet-models/src/whisper	model.rs model.rs	144:169 (8%) 184:209 (8%)	view
20	x 2	crates/ratchet-core/src/ops/norm crates/ratchet-core/src/ops	mod.rs softmax.rs	108:129 (5%) 51:72 (14%)	view
20	x 2	crates/ratchet-models/src/moondream crates/ratchet-models/src/phi3	text_model.rs attn.rs	78:101 (13%) 135:158 (15%)	view
19	x 2	crates/ratchet-models/src/phi2 crates/ratchet-models/src/phi3	model.rs model.rs	189:208 (6%) 227:246 (5%)	view
19	x 2	crates/ratchet-core/src/ops/matmul crates/ratchet-core/src/ops	workgroup_gemv.rs softmax.rs	217:238 (6%) 51:71 (13%)	view
19	x 2	crates/ratchet-core/src/ops crates/ratchet-core/src/ops/matmul	conv.rs workgroup_gemv.rs	36:56 (15%) 217:238 (6%)	view
19	x 2	crates/ratchet-core/src/ops/matmul crates/ratchet-core/src/ops/norm	workgroup_gemv.rs mod.rs	217:238 (6%) 108:128 (4%)	view
19	x 2	crates/ratchet-models/src/phi2 crates/ratchet-models/src/phi3	model.rs model.rs	163:185 (6%) 201:223 (5%)	view
19	x 2	crates/ratchet-core/src/ops/matmul crates/ratchet-core/src/ops/matmul	gemm.rs quantized.rs	89:111 (3%) 131:153 (9%)	view
18	x 2	crates/ratchet-models/src/whisper crates/ratchet-models/src/whisper	transcribe.rs transcribe.rs	35:57 (12%) 113:135 (12%)	view
18	x 2	crates/ratchet-models/src/moondream crates/ratchet-models/src/moondream	generate.rs generate.rs	46:68 (10%) 143:165 (10%)	view
17	x 2	crates/ratchet-models/src/phi3 crates/ratchet-models/src/phi3	model.rs model.rs	35:54 (5%) 71:90 (5%)	view
17	x 2	crates/ratchet-core/src/ops/matmul crates/ratchet-core/src/ops/matmul	gemm.rs quantized.rs	25:41 (3%) 30:46 (8%)	view
17	x 2	crates/ratchet-models/src/whisper crates/ratchet-models/src/whisper	transcribe.rs transcribe.rs	61:77 (12%) 143:159 (12%)	view
17	x 2	crates/ratchet-core/src/ops/matmul crates/ratchet-core/src/ops/matmul	subgroup_gemv.rs workgroup_gemv.rs	27:43 (7%) 41:57 (5%)	view
17	x 2	crates/ratchet-core/src/ops/matmul crates/ratchet-core/src/ops/matmul	quantized.rs workgroup_gemv.rs	30:46 (8%) 41:57 (5%)	view
17	x 2	crates/ratchet-core/src/ops/matmul crates/ratchet-core/src/ops/matmul	gemm.rs workgroup_gemv.rs	25:41 (3%) 41:57 (5%)	view
17	x 2	crates/ratchet-core/src/ops/matmul crates/ratchet-core/src/ops/matmul	gemm.rs subgroup_gemv.rs	25:41 (3%) 27:43 (7%)	view
17	x 2	crates/ratchet-core/src/ops/matmul crates/ratchet-core/src/ops/matmul	quantized.rs subgroup_gemv.rs	30:46 (8%) 27:43 (7%)	view
15	x 2	crates/ratchet-models/src/phi2 crates/ratchet-models/src/phi3	model.rs model.rs	43:57 (5%) 57:71 (4%)	view
15	x 2	crates/ratchet-core/src/ops crates/ratchet-core/src/ops	binary.rs cast.rs	72:86 (6%) 57:71 (7%)	view
15	x 2	crates/ratchet-core/src/ops crates/ratchet-core/src/ops	cast.rs unary.rs	57:71 (7%) 92:106 (4%)	view
15	x 2	crates/ratchet-hub/src crates/ratchet-web/src	lib.rs model.rs	259:274 (6%) 298:314 (4%)	view
15	x 2	crates/ratchet-models/src/phi3 crates/ratchet-models/src/phi3	generate.rs generate.rs	45:62 (14%) 101:118 (14%)	view
15	x 2	crates/ratchet-models/src/moondream crates/ratchet-models/src/phi3	text_model.rs attn.rs	61:75 (10%) 118:132 (11%)	view
15	x 2	crates/ratchet-models/src/whisper crates/ratchet-models/src/whisper	options.rs options.rs	208:222 (6%) 229:243 (6%)	view
14	x 2	crates/ratchet-core/src/ops/matmul crates/ratchet-core/src/ops/matmul	gemm.rs workgroup_gemv.rs	92:107 (2%) 217:232 (4%)	view
14	x 2	crates/ratchet-core/src/ops crates/ratchet-core/src/ops	cache.rs concat.rs	64:84 (15%) 62:78 (16%)	view
14	x 2	crates/ratchet-core/src/ops crates/ratchet-core/src/ops/matmul	conv.rs quantized.rs	36:51 (11%) 134:149 (6%)	view
14	x 2	crates/ratchet-core/src/ops/matmul crates/ratchet-core/src/ops/norm	gemm.rs mod.rs	92:107 (2%) 108:123 (3%)	view
14	x 2	crates/ratchet-loader/src/gguf crates/ratchet-loader/src/gguf	dtype.rs dtype.rs	58:74 (8%) 102:118 (8%)	view
14	x 2	crates/ratchet-core/src/ops/matmul crates/ratchet-core/src/ops	quantized.rs softmax.rs	134:149 (6%) 51:66 (9%)	view
14	x 2	crates/ratchet-core/src/ops crates/ratchet-core/src/ops	binary.rs cache.rs	67:82 (5%) 47:62 (15%)	view
14	x 2	crates/ratchet-core/src/ops crates/ratchet-core/src/ops/matmul	conv.rs gemm.rs	36:51 (11%) 92:107 (2%)	view
14	x 2	crates/ratchet-core/src/ops crates/ratchet-core/src/ops	cache.rs unary.rs	47:62 (15%) 87:102 (4%)	view
14	x 2	crates/ratchet-core/src/ops/matmul crates/ratchet-core/src/ops	gemm.rs softmax.rs	92:107 (2%) 51:66 (9%)	view

Duplicated Units

The list of top 4 duplicated units.

See data for all 4 unit duplicates...

Size	#	Folders	Files	Lines	Code
31	x 2	crates/ratchet-core/src/ops/norm crates/ratchet-core/src/ops	mod.rs unary.rs	339:371 324:356	view
28	x 2	crates/ratchet-core/src/ops/matmul crates/ratchet-core/src/ops/matmul	gemm.rs workgroup_gemv.rs	133:164 69:100	view
19	x 2	crates/ratchet-models/src/phi3 crates/ratchet-models/src/phi2	model.rs model.rs	134:156 105:127	view
8	x 2	crates/ratchet-core/src/ops crates/ratchet-core/src/ops	cast.rs binary.rs	174:183 223:233	view