apache / arrow-go - Duplication

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

25% duplication:

157,590 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
39,962 duplicated lines

7,921 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Duplication Between Components (50+ lines)

Download: SVG DOT (open online Graphviz editor)

From Component --> To Component	Duplicated Lines	File Pairs	Details
internal (<1%) --> parquet (<1%)	95	6 file pairs	details...

Open 2D force graph... Open 3D force graph...

Show more details on duplication between components...

Longest Duplicates

The list of 50 longest duplicates.

See data for all 7,921 duplicates...

Size	#	Folders	Files	Lines	Code
169	x 2	arrow/math/_lib arrow/math/_lib	s int64_avx2.s uint64_avx2.s	8:176 (94%) 8:176 (94%)	view
157	x 2	arrow/math arrow/math	s int64_avx2_amd64.s uint64_avx2_amd64.s	6:173 (98%) 6:173 (98%)	view
96	x 2	arrow/math/_lib arrow/math/_lib	s int64_sse4.s uint64_sse4.s	8:103 (90%) 8:103 (90%)	view
92	x 2	parquet/internal/utils/_lib parquet/internal/utils/_lib	s unpack_bool_avx2.s unpack_bool_sse4.s	8:99 (88%) 8:99 (88%)	view
90	x 2	parquet/metadata/_lib parquet/metadata/_lib	s bloom_filter_block_avx2_amd64.s bloom_filter_block_sse4_amd64.s	87:176 (30%) 87:176 (27%)	view
88	x 2	arrow/compute/internal/kernels/_lib arrow/compute/internal/kernels/_lib	s cast_numeric_neon.s cast_numeric_neon.s	4993:5080 (1%) 5841:5928 (1%)	view
88	x 2	arrow/compute/internal/kernels arrow/compute/internal/kernels	s cast_numeric_neon_arm64.s cast_numeric_neon_arm64.s	5004:5091 (1%) 5852:5939 (1%)	view
84	x 2	arrow/math arrow/math	s int64_sse4_amd64.s uint64_sse4_amd64.s	6:100 (96%) 6:100 (96%)	view
80	x 2	arrow/compute/internal/kernels/_lib arrow/compute/internal/kernels/_lib	s constant_factor_avx2_amd64.s constant_factor_sse4_amd64.s	753:832 (9%) 570:649 (12%)	view
80	x 2	arrow/compute/internal/kernels/_lib arrow/compute/internal/kernels/_lib	s constant_factor_avx2_amd64.s constant_factor_sse4_amd64.s	133:212 (9%) 126:205 (12%)	view
80	x 2	arrow/compute/internal/kernels/_lib arrow/compute/internal/kernels/_lib	s constant_factor_avx2_amd64.s constant_factor_sse4_amd64.s	303:382 (9%) 274:353 (12%)	view
80	x 2	arrow/compute/internal/kernels/_lib arrow/compute/internal/kernels/_lib	s constant_factor_avx2_amd64.s constant_factor_sse4_amd64.s	457:536 (9%) 422:501 (12%)	view
79	x 2	parquet/internal/utils parquet/internal/utils	s unpack_bool_avx2_amd64.s unpack_bool_sse4_amd64.s	6:88 (96%) 6:88 (96%)	view
73	x 2	parquet/metadata/_lib parquet/metadata/_lib	s bloom_filter_block_avx2_amd64.s bloom_filter_block_sse4_amd64.s	8:80 (24%) 8:80 (22%)	view
66	x 2	internal/utils internal/utils	transpose_ints_arm64.go transpose_ints_noasm.go	24:96 (100%) 24:96 (100%)	view
66	x 2	internal/utils internal/utils	transpose_ints_arm64.go transpose_ints_s390x.go	24:96 (100%) 24:96 (100%)	view
66	x 2	internal/utils internal/utils	transpose_ints_ppc64le.go transpose_ints_s390x.go	24:96 (100%) 24:96 (100%)	view
66	x 2	internal/utils internal/utils	transpose_ints_noasm.go transpose_ints_ppc64le.go	24:96 (100%) 24:96 (100%)	view
66	x 2	internal/utils internal/utils	transpose_ints_noasm.go transpose_ints_s390x.go	24:96 (100%) 24:96 (100%)	view
66	x 2	internal/utils internal/utils	transpose_ints_arm64.go transpose_ints_ppc64le.go	24:96 (100%) 24:96 (100%)	view
64	x 2	internal/utils internal/utils	transpose_ints_amd64.go transpose_ints_arm64.go	252:322 (24%) 25:95 (96%)	view
64	x 2	internal/utils internal/utils	transpose_ints_amd64.go transpose_ints_s390x.go	252:322 (24%) 25:95 (96%)	view
64	x 2	internal/utils internal/utils	transpose_ints_amd64.go transpose_ints_ppc64le.go	252:322 (24%) 25:95 (96%)	view
64	x 2	internal/utils internal/utils	transpose_ints_amd64.go transpose_ints_noasm.go	252:322 (24%) 25:95 (96%)	view
63	x 2	arrow/compute/internal/kernels arrow/compute/internal/kernels	s constant_factor_avx2_amd64.s constant_factor_sse4_amd64.s	708:781 (9%) 524:597 (12%)	view
63	x 2	arrow/compute/internal/kernels arrow/compute/internal/kernels	s constant_factor_avx2_amd64.s constant_factor_sse4_amd64.s	423:496 (9%) 390:463 (12%)	view
63	x 2	arrow/compute/internal/kernels arrow/compute/internal/kernels	s constant_factor_avx2_amd64.s constant_factor_sse4_amd64.s	284:357 (9%) 256:329 (12%)	view
63	x 2	arrow/compute/internal/kernels arrow/compute/internal/kernels	s constant_factor_avx2_amd64.s constant_factor_sse4_amd64.s	129:202 (9%) 122:195 (12%)	view
60	x 2	parquet/metadata parquet/metadata	s bloom_filter_block_avx2_amd64.s bloom_filter_block_sse4_amd64.s	13:74 (43%) 14:75 (37%)	view
59	x 2	arrow/compute/internal/kernels arrow/compute/internal/kernels	s cast_numeric_neon_arm64.s cast_numeric_neon_arm64.s	4776:4834 (1%) 5624:5682 (1%)	view
59	x 2	arrow/compute/internal/kernels/_lib arrow/compute/internal/kernels/_lib	s cast_numeric_neon.s cast_numeric_neon.s	4765:4823 (<1%) 5613:5671 (<1%)	view
54	x 2	arrow/math arrow/math	s int64_neon_arm64.s uint64_neon_arm64.s	10:65 (90%) 10:65 (90%)	view
51	x 2	arrow/compute/internal/kernels/_lib arrow/compute/internal/kernels/_lib	s cast_numeric_neon.s cast_numeric_neon.s	3246:3296 (<1%) 3318:3368 (<1%)	view
51	x 2	arrow/compute/internal/kernels arrow/compute/internal/kernels	s cast_numeric_neon_arm64.s cast_numeric_neon_arm64.s	3255:3305 (<1%) 3327:3377 (<1%)	view
49	x 2	arrow/compute/internal/kernels/_lib arrow/compute/internal/kernels/_lib	s constant_factor_avx2_amd64.s constant_factor_sse4_amd64.s	17:65 (5%) 17:65 (7%)	view
48	x 2	arrow/math/_lib arrow/math/_lib	s int64_neon.s uint64_neon.s	7:54 (81%) 7:54 (81%)	view
46	x 2	internal/utils/_lib internal/utils/_lib	s transpose_ints_avx2_amd64.s transpose_ints_sse4_amd64.s	528:573 (1%) 528:573 (1%)	view
46	x 2	internal/utils/_lib internal/utils/_lib	s transpose_ints_avx2_amd64.s transpose_ints_sse4_amd64.s	2972:3017 (1%) 2972:3017 (1%)	view
46	x 2	internal/utils/_lib internal/utils/_lib	s transpose_ints_avx2_amd64.s transpose_ints_sse4_amd64.s	1724:1769 (1%) 1724:1769 (1%)	view
46	x 2	internal/utils/_lib internal/utils/_lib	s transpose_ints_avx2_amd64.s transpose_ints_sse4_amd64.s	1360:1405 (1%) 1360:1405 (1%)	view
46	x 2	internal/utils/_lib internal/utils/_lib	s transpose_ints_avx2_amd64.s transpose_ints_sse4_amd64.s	2920:2965 (1%) 2920:2965 (1%)	view
46	x 2	internal/utils/_lib internal/utils/_lib	s transpose_ints_avx2_amd64.s transpose_ints_sse4_amd64.s	216:261 (1%) 216:261 (1%)	view
46	x 2	internal/utils/_lib internal/utils/_lib	s transpose_ints_avx2_amd64.s transpose_ints_sse4_amd64.s	1048:1093 (1%) 1048:1093 (1%)	view
46	x 2	internal/utils/_lib internal/utils/_lib	s transpose_ints_avx2_amd64.s transpose_ints_sse4_amd64.s	2244:2289 (1%) 2244:2289 (1%)	view
46	x 2	internal/utils/_lib internal/utils/_lib	s transpose_ints_avx2_amd64.s transpose_ints_sse4_amd64.s	112:157 (1%) 112:157 (1%)	view
46	x 2	internal/utils/_lib internal/utils/_lib	s transpose_ints_avx2_amd64.s transpose_ints_sse4_amd64.s	1568:1613 (1%) 1568:1613 (1%)	view
46	x 2	internal/utils/_lib internal/utils/_lib	s transpose_ints_avx2_amd64.s transpose_ints_sse4_amd64.s	2816:2861 (1%) 2816:2861 (1%)	view
46	x 2	internal/utils/_lib internal/utils/_lib	s transpose_ints_avx2_amd64.s transpose_ints_sse4_amd64.s	2452:2497 (1%) 2452:2497 (1%)	view
46	x 2	internal/utils/_lib internal/utils/_lib	s transpose_ints_avx2_amd64.s transpose_ints_sse4_amd64.s	944:989 (1%) 944:989 (1%)	view
46	x 2	internal/utils/_lib internal/utils/_lib	s transpose_ints_avx2_amd64.s transpose_ints_sse4_amd64.s	892:937 (1%) 892:937 (1%)	view

Duplicated Units

The list of top 43 duplicated units.

See data for all 43 unit duplicates...

Size	#	Folders	Files	Lines	Code
34	x 2	parquet/internal/utils parquet/internal/utils	typed_rle_dict.gen.go rle.go	132:170 399:437	view
21	x 2	arrow/array arrow/array	fixed_size_list.go list.go	337:362 592:617	view
16	x 2	arrow/array arrow/array	fixed_size_list.go list.go	87:104 132:149	view
16	x 2	parquet/internal/encoding parquet/internal/encoding	fixed_len_byte_array_encoder.go byte_array_encoder.go	57:75 59:77	view
15	x 3	arrow/array arrow/array arrow/array	numeric_generic.go timestamp.go interval.go	48:64 61:77 68:84	view
15	x 2	arrow/array arrow/array	boolean.go decimal.go	67:83 63:79	view
15	x 2	arrow/array arrow/array	fixedsize_binary.go string.go	63:79 105:121	view
13	x 2	arrow/ipc/cmd/arrow-cat arrow/ipc/cmd/arrow-ls	main.go main.go	70:86 68:84	view
13	x 3	arrow/array arrow/array arrow/array	struct.go list.go union.go	217:232 115:130 328:343	view
12	x 5	arrow/array arrow/array arrow/array arrow/array arrow/array	float16_builder.go timestamp.go booleanbuilder.go numericbuilder.gen.go interval.go	50:64 145:159 50:64 53:67 152:166	view
11	x 2	arrow/array arrow/array	fixed_size_list.go fixedsize_binarybuilder.go	205:218 54:67	view
10	x 2	arrow/internal/flatbuf arrow/internal/flatbuf	DictionaryEncoding.go SparseMatrixIndexCSX.go	68:79 61:72	view
10	x 2	arrow/internal/flatbuf arrow/internal/flatbuf	SparseMatrixIndexCSX.go SparseTensorIndexCOO.go	98:109 123:134	view
10	x 2	arrow/internal/flatbuf arrow/internal/flatbuf	SparseMatrixIndexCSX.go Tensor.go	158:169 123:134	view
10	x 3	arrow/internal/flatbuf arrow/internal/flatbuf arrow/internal/flatbuf	SparseTensorIndexCOO.go SparseTensorIndexCSF.go SparseMatrixIndexCSR.go	78:89 77:88 47:58	view
10	x 2	arrow/internal/flatbuf arrow/internal/flatbuf	SparseTensorIndexCSF.go SparseMatrixIndexCSR.go	169:180 119:130	view
10	x 2	arrow/array arrow/array	fixed_size_list.go list.go	126:138 101:113	view
10	x 2	arrow/array arrow/array	binary.go string.go	434:445 421:432	view
9	x 3	arrow/internal/flatbuf arrow/internal/flatbuf arrow/internal/flatbuf	Schema.go SparseTensor.go Tensor.go	85:95 73:83 71:81	view
9	x 2	arrow/internal/flatbuf arrow/internal/flatbuf	Footer.go Message.go	112:122 90:100	view
9	x 16	arrow/array arrow/array arrow/array arrow/array arrow/array arrow/array arrow/array arrow/array arrow/array arrow/array ...	float16_builder.go null.go struct.go timestamp.go fixed_size_list.go list.go binarybuilder.go numericbuilder.gen.go interval.go dictionary.go ...	252:264 202:214 519:531 366:378 373:385 628:640 360:372 263:275 331:343 668:680 ...	view
9	x 4	arrow/array arrow/array arrow/array arrow/array	timestamp.go boolean.go interval.go string.go	115:125 112:122 123:133 172:182	view
9	x 3	arrow/array arrow/array arrow/array	boolean.go fixedsize_binary.go string.go	100:110 97:107 160:170	view
9	x 2	arrow/array arrow/array	fixedsize_binary.go binary.go	109:119 172:182	view
9	x 2	parquet/internal/encoding parquet/internal/encoding	byte_stream_split.go fixed_len_byte_array_decoder.go	385:396 153:164	view
8	x 4	arrow/scalar arrow/scalar arrow/scalar arrow/scalar	scalar.go numeric.gen.go temporal.go nested.go	164:173 49:58 60:69 203:212	view
8	x 2	arrow/internal/flatbuf arrow/internal/flatbuf	RecordBatch.go SparseTensorIndexCSF.go	65:74 135:144	view
8	x 2	arrow/array arrow/array	numeric_generic.go decimal.go	66:75 81:90	view
8	x 2	arrow/array arrow/array	struct.go interval.go	128:138 359:368	view
8	x 3	arrow/array arrow/array arrow/array	struct.go fixed_size_list.go list.go	381:391 281:291 508:518	view
8	x 2	arrow/array arrow/array	binary.go string.go	342:353 345:356	view
8	x 3	internal/utils internal/utils internal/utils	min_max_s390x.go min_max_ppc64le.go min_max_noasm.go	21:30 21:30 22:31	view
7	x 2	arrow/ipc/cmd/arrow-cat arrow/ipc/cmd/arrow-ls	main.go main.go	113:121 110:118	view
7	x 3	arrow/decimal128 arrow/decimal256 arrow/float16	decimal128.go decimal256.go float16.go	396:404 418:426 121:129	view
7	x 3	arrow/decimal128 arrow/decimal256 arrow/float16	decimal128.go decimal256.go float16.go	407:415 429:437 132:140	view
7	x 3	arrow/decimal128 arrow/decimal256 arrow/float16	decimal128.go decimal256.go float16.go	422:430 444:452 147:155	view
7	x 2	parquet/internal/encoding parquet/internal/encoding	delta_length_byte_array.go delta_byte_array.go	59:67 112:120	view
6	x 2	arrow/csv arrow/avro	reader.go reader.go	933:941 328:336	view
6	x 2	arrow/internal/flatbuf arrow/internal/flatbuf	SparseTensor.go Tensor.go	60:67 59:66	view
6	x 9	arrow/array arrow/array arrow/array arrow/array arrow/array arrow/array arrow/array arrow/array arrow/array	float16_builder.go struct.go timestamp.go fixed_size_list.go builder.go list.go numericbuilder.gen.go interval.go decimal.go	99:106 350:357 201:208 260:267 248:255 480:487 101:108 200:207 244:251	view
6	x 16	arrow/array arrow/array arrow/array arrow/array arrow/array arrow/array arrow/array arrow/array arrow/array arrow/array ...	float16_builder.go null.go struct.go timestamp.go fixed_size_list.go list.go binarybuilder.go union.go booleanbuilder.go numericbuilder.gen.go ...	240:247 193:200 510:517 357:364 364:371 619:626 351:358 1012:1019 234:241 254:261 ...	view
6	x 2	arrow/array arrow/array	fixed_size_list.go union.go	328:335 1021:1028	view
6	x 2	parquet/internal/encoding parquet/internal/encoding	memo_table.go memo_table_types.gen.go	342:349 70:77	view