apache / datafusion-comet

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

12% duplication:

49,725 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
6,433 duplicated lines

1,042 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Duplication Between Components (50+ lines)

Download: SVG DOT (open online Graphviz editor)

From Component --> To Component	Duplicated Lines	File Pairs	Details
native (<1%) --> ROOT (294%)	112	6 file pairs	details...

Open 2D force graph... Open 3D force graph...

Show more details on duplication between components...

Longest Duplicates

The list of 50 longest duplicates.

See data for all 1,042 duplicates...

Size	#	Folders	Files	Lines	Code
89	x 2	native/core/src/parquet/util native/core/src/parquet/util	bit_packing.rs bit_packing.rs	2363:2471 (2%) 2379:2487 (2%)	view
85	x 2	native/core/src/parquet/util native/core/src/parquet/util	bit_packing.rs bit_packing.rs	2921:3025 (2%) 2957:3061 (2%)	view
76	x 2	native/core/src/parquet/util native/core/src/parquet/util	bit_packing.rs bit_packing.rs	2363:2455 (2%) 2395:2487 (2%)	view
73	x 2	native/core/src/parquet/util native/core/src/parquet/util	bit_packing.rs bit_packing.rs	1849:1935 (2%) 1879:1965 (2%)	view
63	x 2	native/core/src/parquet/util native/core/src/parquet/util	bit_packing.rs bit_packing.rs	2363:2439 (2%) 2411:2487 (2%)	view
61	x 2	native/core/src/parquet/util native/core/src/parquet/util	bit_packing.rs bit_packing.rs	611:677 (2%) 621:687 (2%)	view
61	x 2	native/core/src/parquet/util native/core/src/parquet/util	bit_packing.rs bit_packing.rs	977:1045 (2%) 1001:1069 (2%)	view
59	x 2	native/core/src/parquet/util native/core/src/parquet/util	bit_packing.rs bit_packing.rs	3219:3291 (1%) 3295:3367 (1%)	view
56	x 2	native/core/src/parquet/util native/core/src/parquet/util	bit_packing.rs bit_packing.rs	2921:2989 (1%) 2993:3061 (1%)	view
55	x 2	native/core/src/parquet/util native/core/src/parquet/util	bit_packing.rs bit_packing.rs	2633:2699 (1%) 2703:2769 (1%)	view
52	x 2	native/core/src/parquet/util native/core/src/parquet/util	bit_packing.rs bit_packing.rs	611:667 (1%) 631:687 (1%)	view
51	x 2	native/core/src/parquet/util native/core/src/parquet/util	bit_packing.rs bit_packing.rs	2099:2159 (1%) 2163:2223 (1%)	view
50	x 2	native/core/src/parquet/util native/core/src/parquet/util	bit_packing.rs bit_packing.rs	2363:2423 (1%) 2427:2487 (1%)	view
49	x 2	native/core/src/parquet/util native/core/src/parquet/util	bit_packing.rs bit_packing.rs	289:339 (1%) 307:357 (1%)	view
48	x 2	native/core/src/parquet/util native/core/src/parquet/util	bit_packing.rs bit_packing.rs	1849:1905 (1%) 1909:1965 (1%)	view
47	x 2	native/core/src/parquet/util native/core/src/parquet/util	bit_packing.rs bit_packing.rs	1609:1663 (1%) 1667:1721 (1%)	view
43	x 2	native/core/src/parquet/util native/core/src/parquet/util	bit_packing.rs bit_packing.rs	611:657 (1%) 641:687 (1%)	view
43	x 2	native/core/src/parquet/util native/core/src/parquet/util	bit_packing.rs bit_packing.rs	1179:1227 (1%) 1231:1279 (1%)	view
40	x 2	native/core/src/parquet/util native/core/src/parquet/util	bit_packing.rs bit_packing.rs	977:1021 (1%) 1025:1069 (1%)	view
39	x 2	native/core/src/parquet/util native/core/src/parquet/util	bit_packing.rs bit_packing.rs	785:827 (1%) 831:873 (1%)	view
37	x 2	native/core/src/parquet/util native/core/src/parquet/util	bit_packing.rs bit_packing.rs	2363:2407 (1%) 2443:2487 (1%)	view
35	x 2	native/core/src/parquet/util native/core/src/parquet/util	bit_packing.rs bit_packing.rs	443:479 (1%) 483:519 (1%)	view
34	x 2	native/core/src/parquet/util native/core/src/parquet/util	bit_packing.rs bit_packing.rs	611:647 (1%) 651:687 (1%)	view
32	x 2	native/core/src/parquet/util native/core/src/parquet/util	bit_packing.rs bit_packing.rs	289:321 (1%) 325:357 (1%)	view
31	x 2	native/core/src/parquet/util native/core/src/parquet/util	bit_packing.rs bit_packing.rs	146:176 (1%) 179:209 (1%)	view
27	x 2	native/core/src/parquet/util native/core/src/parquet/util	bit_packing.rs bit_packing.rs	2921:2953 (<1%) 3029:3061 (<1%)	view
25	x 2	native/core/src/parquet/util native/core/src/parquet/util	bit_packing.rs bit_packing.rs	611:637 (<1%) 661:687 (<1%)	view
24	x 2	native/spark-expr/src/agg_funcs native/spark-expr/src/agg_funcs	correlation.rs covariance.rs	82:107 (14%) 91:116 (10%)	view
24	x 2	native/core/src/parquet/util native/core/src/parquet/util	bit_packing.rs bit_packing.rs	2363:2391 (<1%) 2459:2487 (<1%)	view
23	x 2	native/spark-expr/src/datetime_funcs native/spark-expr/src/datetime_funcs	hour.rs minute.rs	70:102 (30%) 70:102 (30%)	view
23	x 2	native/spark-expr/src/datetime_funcs native/spark-expr/src/datetime_funcs	minute.rs second.rs	70:102 (30%) 70:102 (30%)	view
23	x 2	native/core/src/parquet/util native/core/src/parquet/util	bit_packing.rs bit_packing.rs	1849:1875 (<1%) 1939:1965 (<1%)	view
23	x 2	native/spark-expr/src/datetime_funcs native/spark-expr/src/datetime_funcs	hour.rs second.rs	70:102 (30%) 70:102 (30%)	view
20	x 2	native/spark-expr/src/agg_funcs native/spark-expr/src/agg_funcs	stddev.rs variance.rs	100:127 (18%) 90:117 (12%)	view
20	x 2	native/core/src/parquet/read native/core/src/parquet/read	column.rs column.rs	167:187 (2%) 243:263 (2%)	view
20	x 2	native/core/src/parquet/read native/core/src/parquet/read	column.rs column.rs	134:154 (2%) 167:187 (2%)	view
20	x 2	native/core/src/parquet/read native/core/src/parquet/read	column.rs column.rs	134:154 (2%) 243:263 (2%)	view
19	x 2	native/core/src/parquet/util native/core/src/parquet/util	bit_packing.rs bit_packing.rs	977:997 (<1%) 1049:1069 (<1%)	view
19	x 2	native/spark-expr/src/agg_funcs native/spark-expr/src/agg_funcs	covariance.rs covariance.rs	173:196 (8%) 214:236 (8%)	view
18	x 2	native/spark-expr/src/kernels native/spark-expr/src/kernels	temporal.rs temporal.rs	901:921 (2%) 1072:1092 (2%)	view
18	x 2	common/src/main/java/org/apache/comet/parquet common/src/main/java/org/apache/comet/parquet	BatchReader.java NativeBatchReader.java	380:410 (5%) 410:440 (5%)	view
17	x 2	native/proto/src/proto native/proto/src/proto	proto expr.proto operator.proto	1:20 (4%) 1:20 (8%)	view
17	x 2	native/proto/src/proto native/proto/src/proto	proto expr.proto partitioning.proto	1:20 (4%) 1:20 (51%)	view
17	x 2	native/proto/src/proto native/proto/src/proto	proto metric.proto operator.proto	1:20 (73%) 1:20 (8%)	view
17	x 2	native/core native/proto	Cargo.toml Cargo.toml	1:18 (15%) 1:18 (56%)	view
17	x 2	native/proto/src/proto native/proto/src/proto	proto operator.proto partitioning.proto	1:20 (8%) 1:20 (51%)	view
17	x 2	native/proto native/spark-expr	Cargo.toml Cargo.toml	1:18 (56%) 1:18 (26%)	view
17	x 2	native/proto/src/proto native/proto/src/proto	proto metric.proto partitioning.proto	1:20 (73%) 1:20 (51%)	view
17	x 2	native/proto/src/proto native/proto/src/proto	proto expr.proto metric.proto	1:20 (4%) 1:20 (73%)	view
17	x 2	native/core native/spark-expr	Cargo.toml Cargo.toml	1:18 (15%) 1:18 (26%)	view

Duplicated Units

The list of top 8 duplicated units.

See data for all 8 unit duplicates...

Size	#	Folders	Files	Lines	Code
17	x 2	spark/src/main/scala/org/apache/spark/sql/comet spark/src/main/scala/org/apache/spark/sql/comet	CometScanExec.scala CometBatchScanExec.scala	250:271 50:71	view
16	x 2	native/spark-expr/src/agg_funcs native/spark-expr/benches	sum_decimal.rs aggregate.rs	534:557 164:187	view
13	x 2	native/spark-expr/src/agg_funcs native/spark-expr/src/agg_funcs	variance.rs stddev.rs	95:109 105:119	view
7	x 2	common/src/main/scala/or...l/comet/execution/arrow spark/src/main/scala/org...comet/execution/shuffle	ArrowReaderIterator.scala NativeBatchDecoderIterator.scala	60:70 87:97	view
7	x 2	common/src/main/java/org/apache/comet/parquet common/src/main/java/org/apache/comet/parquet	BatchReader.java NativeBatchReader.java	186:197 183:194	view
7	x 2	native/spark-expr/src/array_funcs native/spark-expr/src/array_funcs	list_extract.rs get_array_struct_fields.rs	79:87 57:65	view
6	x 2	common/src/main/java/org/apache/arrow/c common/src/main/java/org/apache/arrow/c	ArrowImporter.java AbstractCometSchemaImporter.java	42:49 45:52	view
6	x 3	native/spark-expr/src/datetime_funcs native/spark-expr/src/datetime_funcs native/spark-expr/src/datetime_funcs	minute.rs second.rs hour.rs	78:85 78:85 78:85	view