huggingface / xet-core Duplication

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

9% duplication:

22,758 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
2,082 duplicated lines

249 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Longest Duplicates

The list of 50 longest duplicates.

See data for all 249 duplicates...

Size	#	Folders	Files	Lines	Code
27	x 2	merkledb/src merkledb/src	tests.rs tests.rs	400:433 (8%) 445:478 (8%)	view
25	x 2	cas_client/src cas_client/src	remote_client.rs remote_client.rs	1094:1125 (2%) 1166:1197 (2%)	view
25	x 2	cas_client/src cas_client/src	remote_client.rs remote_client.rs	1018:1049 (2%) 1094:1125 (2%)	view
25	x 2	cas_client/src cas_client/src	remote_client.rs remote_client.rs	1018:1049 (2%) 1166:1197 (2%)	view
24	x 2	cas_client/src cas_client/src	remote_client.rs remote_client.rs	1166:1194 (2%) 1265:1293 (2%)	view
24	x 2	hf_xet ROOT	pyproject.toml pyproject.toml	1:28 (92%) 1:28 (88%)	view
24	x 2	cas_client/src cas_client/src	remote_client.rs remote_client.rs	1018:1046 (2%) 1265:1293 (2%)	view
24	x 2	cas_client/src cas_client/src	remote_client.rs remote_client.rs	1094:1122 (2%) 1265:1293 (2%)	view
23	x 2	cas_client/src cas_client/src	remote_client.rs remote_client.rs	992:1015 (2%) 1140:1163 (2%)	view
21	x 2	mdb_shard/src mdb_shard/src	shard_format.rs shard_format.rs	1461:1484 (1%) 1513:1536 (1%)	view
19	x 2	chunk_cache/src chunk_cache/src	disk.rs disk.rs	1268:1292 (2%) 1302:1326 (2%)	view
19	x 2	cas_object/src cas_object/src	cas_object_format.rs cas_object_format.rs	1900:1925 (1%) 1930:1955 (1%)	view
17	x 2	cas_client/src cas_client/src	remote_client.rs remote_client.rs	999:1015 (1%) 1075:1091 (1%)	view
17	x 2	cas_client/src cas_client/src	remote_client.rs remote_client.rs	1075:1091 (1%) 1147:1163 (1%)	view
16	x 2	cas_object/src cas_object/src	cas_object_format.rs cas_object_format.rs	1869:1888 (1%) 1930:1949 (1%)	view
16	x 2	cas_object/src cas_object/src	cas_object_format.rs cas_object_format.rs	1869:1888 (1%) 1900:1919 (1%)	view
16	x 2	cas_object/src cas_object/src	cas_object_format.rs cas_object_format.rs	1989:2008 (1%) 2020:2039 (1%)	view
16	x 2	mdb_shard/src mdb_shard/src	shard_format.rs shard_format.rs	978:993 (1%) 1001:1016 (1%)	view
16	x 2	utils/src utils/src	singleflight.rs singleflight.rs	434:451 (3%) 465:482 (3%)	view
15	x 2	cas_object/src/byte_grouping cas_object/src/byte_grouping	bg4.rs bg4.rs	181:198 (6%) 237:254 (6%)	view
15	x 2	cas_object/src cas_object/src	cas_object_format.rs cas_object_format.rs	1960:1977 (1%) 2050:2067 (1%)	view
13	x 2	mdb_shard/src mdb_shard/src	shard_file_manager.rs shard_format.rs	656:678 (1%) 1562:1584 (1%)	view
13	x 2	cas_object/src cas_object/src	cas_object_format.rs cas_object_format.rs	1877:1895 (<1%) 1997:2015 (<1%)	view
13	x 2	cas_object/src cas_object/src	cas_object_format.rs cas_object_format.rs	1783:1796 (<1%) 2050:2064 (<1%)	view
13	x 2	cas_object/src cas_object/src	cas_object_format.rs cas_object_format.rs	1783:1796 (<1%) 1960:1974 (<1%)	view
13	x 2	merkledb/benches merkledb/benches	rolling_hash_benchmark.rs rolling_hash_benchmark.rs	79:95 (6%) 149:165 (6%)	view
13	x 2	hf_xet/src hf_xet/src	lib.rs lib.rs	51:65 (7%) 82:96 (7%)	view
13	x 2	merkledb/src merkledb/src	chunk_iterator.rs chunk_iterator.rs	39:60 (7%) 148:169 (7%)	view
12	x 2	mdb_shard/src mdb_shard/src	shard_format.rs shard_format.rs	1361:1372 (1%) 1378:1389 (1%)	view
12	x 2	cas_client/src cas_client/src	remote_client.rs remote_client.rs	167:180 (1%) 241:254 (1%)	view
12	x 2	cas_object/src cas_object/src	cas_object_format.rs cas_object_format.rs	1908:1925 (<1%) 2028:2045 (<1%)	view
12	x 2	chunk_cache/src chunk_cache/src	disk.rs disk.rs	851:864 (1%) 883:897 (1%)	view
12	x 2	cas_object/src cas_object/src/cas_chunk_format	cas_chunk_format.rs deserialize_async.rs	198:213 (5%) 61:76 (9%)	view
12	x 2	cas_object/src cas_object/src	cas_object_format.rs cas_object_format.rs	1938:1955 (<1%) 2028:2045 (<1%)	view
12	x 2	file_utils/src file_utils/src	safe_file_creator.rs safe_file_creator.rs	241:258 (5%) 305:322 (5%)	view
11	x 2	merkledb/src merkledb/src	chunk_iterator.rs chunk_iterator.rs	63:77 (6%) 172:186 (6%)	view
11	x 2	progress_tracking/src progress_tracking/src	upload_tracking.rs upload_tracking.rs	186:200 (1%) 414:425 (1%)	view
10	x 2	cas_client/src cas_client/src	retry_utils.rs retry_utils.rs	120:131 (6%) 224:235 (6%)	view
10	x 2	cas_object/src/byte_grouping cas_object/src/byte_grouping	bg4.rs bg4.rs	163:174 (4%) 202:213 (4%)	view
10	x 2	cas_object/src cas_object/src	cas_object_format.rs cas_object_format.rs	1850:1861 (<1%) 1869:1880 (<1%)	view
10	x 2	cas_object/src cas_object/src	cas_object_format.rs cas_object_format.rs	1850:1861 (<1%) 1900:1911 (<1%)	view
10	x 2	cas_object/src cas_object/src	cas_object_format.rs cas_object_format.rs	1850:1861 (<1%) 1930:1941 (<1%)	view
10	x 2	data/src data/src	data_client.rs data_client.rs	308:322 (3%) 343:357 (3%)	view
10	x 2	merkledb/src merkledb/src	async_chunk_iterator.rs async_chunk_iterator.rs	148:158 (2%) 366:376 (2%)	view
10	x 2	cas_client/src cas_client/src	http_client.rs http_client.rs	443:453 (2%) 470:480 (2%)	view
10	x 2	utils/src utils/src	constant_declarations.rs constant_declarations.rs	45:57 (9%) 80:92 (9%)	view
10	x 2	cas_object/src cas_object/src	cas_object_format.rs cas_object_format.rs	1540:1550 (<1%) 1593:1604 (<1%)	view
10	x 2	cas_object/src cas_object/src	cas_object_format.rs cas_object_format.rs	2078:2089 (<1%) 2108:2119 (<1%)	view
9	x 2	hf_xet/src hf_xet/src	lib.rs lib.rs	39:48 (5%) 70:79 (5%)	view
9	x 2	mdb_shard/src mdb_shard/src	file_structs.rs file_structs.rs	280:295 (2%) 324:339 (2%)	view