apache / arrow-datafusionDuplication

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

20% duplication:

144,516 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
28,915 duplicated lines

8,471 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Duplication Between Components (50+ lines)

Download: SVG DOT (open online Graphviz editor)

From Component --> To Component	Duplicated Lines	File Pairs	Details
datafusion-examples (250%) --> datafusion (<1%)	281	11 file pairs	details...
datafusion-cli (9%) --> datafusion (<1%)	256	14 file pairs	details...
ROOT (136%) --> datafusion (<1%)	208	22 file pairs	details...
datafusion (<1%) --> test-utils (32%)	204	11 file pairs	details...
benchmarks (7%) --> datafusion (<1%)	204	11 file pairs	details...

Open 3D force graph...

Show more details on duplication between components...

Longest Duplicates

The list of 50 longest duplicates.

See data for all 8,471 duplicates...

Size	#	Folders	Files	Lines	Code
66	x 2	datafusion/core/src/physical_optimizer datafusion/core/src/physical_optimizer	dist_enforcement.rs dist_enforcement.rs	1692:1765 (3%) 1813:1886 (3%)	view
47	x 2	datafusion/optimizer/src datafusion/optimizer/src	eliminate_cross_join.rs eliminate_cross_join.rs	593:642 (5%) 820:871 (5%)	view
46	x 2	datafusion/physical-expr/src/expressions datafusion/physical-expr/src/expressions	binary.rs binary.rs	2216:2265 (1%) 2434:2483 (1%)	view
46	x 2	datafusion/physical-expr/src/expressions datafusion/physical-expr/src/expressions	binary.rs binary.rs	1580:1629 (1%) 1796:1845 (1%)	view
46	x 2	datafusion/physical-expr/src/expressions datafusion/physical-expr/src/expressions	binary.rs binary.rs	1580:1629 (1%) 2434:2483 (1%)	view
46	x 2	datafusion/physical-expr/src/expressions datafusion/physical-expr/src/expressions	binary.rs binary.rs	1580:1629 (1%) 2216:2265 (1%)	view
46	x 2	datafusion/physical-expr/src/expressions datafusion/physical-expr/src/expressions	binary.rs binary.rs	1796:1845 (1%) 2434:2483 (1%)	view
46	x 2	datafusion/physical-expr/src/expressions datafusion/physical-expr/src/expressions	binary.rs binary.rs	1796:1845 (1%) 2216:2265 (1%)	view
45	x 2	datafusion/optimizer/src datafusion/optimizer/src	eliminate_cross_join.rs eliminate_cross_join.rs	583:631 (4%) 660:710 (4%)	view
41	x 2	datafusion/physical-expr/src/aggregate datafusion/physical-expr/src/aggregate	array_agg.rs array_agg_distinct.rs	203:246 (22%) 239:284 (20%)	view
40	x 2	datafusion/physical-expr/src/aggregate datafusion/physical-expr/src/aggregate	bit_and_or_xor.rs sum_distinct.rs	696:751 (6%) 124:179 (20%)	view
40	x 2	datafusion/physical-expr/src/aggregate datafusion/physical-expr/src/aggregate	build_in.rs build_in.rs	941:985 (3%) 988:1032 (3%)	view
39	x 2	datafusion/proto/src/physical_plan datafusion/proto/src/physical_plan	mod.rs mod.rs	625:664 (2%) 672:711 (2%)	view
39	x 2	datafusion/core/src/physical_optimizer datafusion/core/src/physical_optimizer	join_selection.rs join_selection.rs	1304:1342 (3%) 1464:1506 (3%)	view
39	x 2	datafusion/core/src datafusion/core/src	physical_planner.rs physical_planner.rs	1652:1690 (1%) 1704:1743 (1%)	view
37	x 2	datafusion/common/src datafusion/physical-expr/src/aggregate	scalar.rs array_agg.rs	5372:5410 (<1%) 203:241 (19%)	view
37	x 2	datafusion/common/src datafusion/physical-expr/src/aggregate	scalar.rs array_agg_distinct.rs	5372:5410 (<1%) 239:279 (18%)	view
37	x 2	datafusion/core/src/physical_optimizer datafusion/core/src/physical_optimizer	join_selection.rs join_selection.rs	1347:1386 (2%) 1511:1550 (2%)	view
37	x 2	datafusion/optimizer/src datafusion/optimizer/src	eliminate_cross_join.rs eliminate_cross_join.rs	670:710 (4%) 820:860 (4%)	view
36	x 2	datafusion/physical-expr...gate/groups_accumulator datafusion/physical-expr...gate/groups_accumulator	accumulate.rs accumulate.rs	664:713 (7%) 780:830 (7%)	view
35	x 2	datafusion/optimizer/src datafusion/optimizer/src	eliminate_cross_join.rs eliminate_cross_join.rs	915:949 (3%) 988:1022 (3%)	view
33	x 2	datafusion/optimizer/src datafusion/optimizer/src	eliminate_cross_join.rs eliminate_cross_join.rs	583:618 (3%) 735:771 (3%)	view
33	x 2	datafusion/optimizer/src datafusion/optimizer/src	eliminate_cross_join.rs eliminate_cross_join.rs	660:696 (3%) 735:771 (3%)	view
32	x 2	datafusion/core/src/physical_optimizer datafusion/core/src/physical_optimizer	aggregate_statistics.rs aggregate_statistics.rs	542:579 (7%) 589:626 (7%)	view
32	x 2	datafusion/physical-expr/src/aggregate datafusion/physical-expr/src/aggregate	build_in.rs build_in.rs	941:972 (2%) 1035:1066 (2%)	view
32	x 2	datafusion/physical-expr/src/aggregate datafusion/physical-expr/src/aggregate	build_in.rs build_in.rs	988:1019 (2%) 1035:1066 (2%)	view
31	x 2	datafusion/physical-expr/src/aggregate datafusion/physical-expr/src/aggregate	build_in.rs build_in.rs	865:900 (2%) 903:938 (2%)	view
31	x 2	datafusion/physical-expr/src/aggregate datafusion/physical-expr/src/aggregate	build_in.rs build_in.rs	789:824 (2%) 827:862 (2%)	view
28	x 2	datafusion/core/src/physical_plan/joins datafusion/core/src/physical_plan/joins	symmetric_hash_join.rs symmetric_hash_join.rs	1853:1883 (2%) 1906:1936 (2%)	view
28	x 2	datafusion/physical-expr/src/expressions datafusion/physical-expr/src/expressions	binary.rs binary.rs	1580:1608 (<1%) 2004:2032 (<1%)	view
28	x 2	datafusion/physical-expr/src/expressions datafusion/physical-expr/src/expressions	binary.rs binary.rs	1796:1824 (<1%) 2004:2032 (<1%)	view
28	x 2	datafusion/physical-expr/src/expressions datafusion/physical-expr/src/expressions	binary.rs binary.rs	2004:2032 (<1%) 2216:2244 (<1%)	view
28	x 2	datafusion/physical-expr/src/expressions datafusion/physical-expr/src/expressions	binary.rs binary.rs	2004:2032 (<1%) 2434:2462 (<1%)	view
27	x 2	datafusion/core/src/physical_plan datafusion/core/src/physical_plan	filter.rs filter.rs	811:838 (3%) 867:894 (3%)	view
27	x 2	datafusion/physical-expr/src/aggregate datafusion/physical-expr/src/aggregate	covariance.rs covariance.rs	87:117 (4%) 166:196 (4%)	view
27	x 2	datafusion/core/benches datafusion/core/benches	aggregate_query_sql.rs window_query_sql.rs	18:50 (20%) 18:50 (14%)	view
26	x 2	datafusion/core/src/physical_plan/joins datafusion/core/src/physical_plan/joins	symmetric_hash_join.rs symmetric_hash_join.rs	1673:1700 (1%) 1784:1811 (1%)	view
26	x 2	datafusion/physical-expr/src/aggregate datafusion/physical-expr/src/aggregate	stddev.rs variance.rs	434:462 (8%) 506:534 (7%)	view
26	x 2	datafusion/physical-expr/src/aggregate datafusion/physical-expr/src/aggregate	correlation.rs variance.rs	492:520 (6%) 506:534 (7%)	view
26	x 2	datafusion/physical-expr/src/aggregate datafusion/physical-expr/src/aggregate	correlation.rs covariance.rs	492:520 (6%) 741:769 (4%)	view
26	x 2	datafusion/physical-expr/src/aggregate datafusion/physical-expr/src/aggregate	covariance.rs stddev.rs	741:769 (4%) 434:462 (8%)	view
26	x 2	datafusion/physical-expr/src/aggregate datafusion/physical-expr/src/aggregate	min_max.rs min_max.rs	167:197 (2%) 804:833 (2%)	view
26	x 2	datafusion/expr/src/type_coercion datafusion/expr/src/type_coercion	binary.rs binary.rs	1164:1193 (2%) 1171:1200 (2%)	view
26	x 2	datafusion/core/src/physical_plan/joins datafusion/core/src/physical_plan/joins	symmetric_hash_join.rs symmetric_hash_join.rs	1645:1670 (1%) 1736:1761 (1%)	view
26	x 2	datafusion/physical-expr/src/aggregate datafusion/physical-expr/src/aggregate	covariance.rs variance.rs	741:769 (4%) 506:534 (7%)	view
26	x 2	datafusion/physical-expr/src/aggregate datafusion/physical-expr/src/aggregate	correlation.rs stddev.rs	492:520 (6%) 434:462 (8%)	view
26	x 2	datafusion/physical-expr/src/window datafusion/physical-expr/src/window	aggregate.rs sliding_aggregate.rs	124:158 (23%) 108:142 (24%)	view
25	x 2	datafusion/optimizer/src datafusion/optimizer/src	eliminate_cross_join.rs eliminate_cross_join.rs	745:771 (2%) 820:846 (2%)	view
25	x 2	datafusion/physical-expr/src datafusion/physical-expr/src	array_expressions.rs array_expressions.rs	605:635 (<1%) 695:725 (<1%)	view
25	x 2	datafusion/physical-expr/src/expressions datafusion/physical-expr/src/expressions	binary.rs binary.rs	2177:2206 (<1%) 2395:2424 (<1%)	view

Duplicated Units

The list of top 36 duplicated units.

See data for all 36 unit duplicates...

Size	#	Folders	Files	Lines	Code
32	x 2	datafusion/physical-expr/src/aggregate datafusion/physical-expr/src/aggregate	correlation.rs regr.rs	77:110 125:158	view
24	x 4	datafusion/physical-expr/src/aggregate datafusion/physical-expr/src/aggregate datafusion/physical-expr/src/aggregate datafusion/physical-expr/src/aggregate	covariance.rs variance.rs stddev.rs correlation.rs	744:770 509:535 437:463 495:521	view
19	x 3	datafusion/core/src/datasource/physical_plan datafusion/core/src/datasource/physical_plan datafusion/core/src/execution	parquet.rs csv.rs context.rs	1901:1927 991:1017 2721:2747	view
19	x 2	datafusion/core/src/physical_plan/joins datafusion/core/src/physical_plan/joins	symmetric_hash_join.rs hash_join.rs	379:399 232:252	view
19	x 2	datafusion/physical-expr/src/aggregate datafusion/physical-expr/src/aggregate	bit_and_or_xor.rs sum_distinct.rs	727:749 155:177	view
18	x 2	datafusion/physical-expr/src/window datafusion/physical-expr/src/window	aggregate.rs sliding_aggregate.rs	136:155 120:139	view
16	x 2	datafusion/core/src/physical_plan/windows datafusion/core/src/physical_plan/windows	bounded_window_agg_exec.rs window_agg_exec.rs	335:353 259:277	view
13	x 2	datafusion/physical-expr/src/aggregate datafusion/physical-expr/src/aggregate	variance.rs stddev.rs	86:100 81:95	view
12	x 3	datafusion/core/src/physical_optimizer datafusion/core/src/physical_optimizer datafusion/core/src/physical_optimizer	sort_enforcement.rs dist_enforcement.rs sort_pushdown.rs	197:211 921:935 84:98	view
12	x 2	datafusion/physical-expr/src/expressions datafusion/physical-expr/src/expressions/binary	binary.rs kernels_arrow.rs	3238:3251 277:291	view
11	x 2	datafusion/core/src/datasource/file_format datafusion/core/src/datasource/file_format	json.rs csv.rs	247:259 468:480	view
11	x 2	datafusion/core/src/physical_optimizer datafusion/physical-expr/src	replace_with_order_preserving_variant... utils.rs	121:133 543:556	view
11	x 2	datafusion/physical-expr/src/aggregate datafusion/physical-expr/src/aggregate	bit_and_or_xor.rs sum_distinct.rs	696:710 124:138	view
11	x 2	datafusion/physical-expr/src/aggregate datafusion/physical-expr/src/aggregate	bit_and_or_xor.rs sum_distinct.rs	712:725 140:153	view
9	x 7	datafusion/core/src/physical_plan/aggregates datafusion/core/src/physical_plan datafusion/core/src/physical_plan datafusion/core/src/physical_plan datafusion/core/src/physical_plan datafusion/core/src/physical_plan/joins datafusion/core/src/physical_plan	mod.rs values.rs coalesce_partitions.rs explain.rs union.rs cross_join.rs unnest.rs	1640:1650 98:108 63:73 75:85 149:159 144:154 69:79	view
9	x 3	datafusion/core/src/physical_plan/joins datafusion/core/src/physical_plan/joins datafusion/core/src/physical_plan/joins	symmetric_hash_join.rs sort_merge_join.rs hash_join.rs	453:463 272:282 378:388	view
9	x 5	datafusion/physical-expr/src/aggregate datafusion/physical-expr/src/aggregate datafusion/physical-expr/src/aggregate datafusion/physical-expr/src/aggregate datafusion/physical-expr/src/aggregate	bool_and_or.rs grouping.rs bit_and_or_xor.rs sum.rs min_max.rs	158:168 97:107 234:244 194:204 263:273	view
9	x 2	datafusion/physical-expr/src/aggregate datafusion/physical-expr/src/aggregate	array_agg_ordered.rs first_last.rs	130:140 144:154	view
8	x 2	datafusion/core/benches datafusion/core/benches	window_query_sql.rs aggregate_query_sql.rs	38:47 38:47	view
8	x 2	datafusion/core/src/datasource/physical_plan datafusion/core/src/datasource/physical_plan	parquet.rs csv.rs	1890:1899 1205:1214	view
8	x 3	datafusion/core/src/physical_plan/aggregates datafusion/core/src/physical_plan/windows datafusion/core/src/physical_optimizer	mod.rs mod.rs replace_with_order_preserving_variant...	1277:1286 398:407 818:827	view
8	x 3	datafusion/physical-expr/src/aggregate datafusion/physical-expr/src/aggregate datafusion/physical-expr/src/aggregate	array_agg_distinct.rs array_agg.rs approx_distinct.rs	100:109 91:100 119:128	view
8	x 3	datafusion/physical-expr/src/window datafusion/physical-expr/src/window datafusion/physical-expr/src/window	row_number.rs ntile.rs rank.rs	68:78 69:79 114:124	view
7	x 2	datafusion/core/src/datasource datafusion/expr/src	provider.rs table_source.rs	91:99 92:100	view
7	x 2	datafusion/core/src/physical_plan/aggregates datafusion/core/src/physical_plan	mod.rs projection.rs	885:893 237:245	view
7	x 2	datafusion/core/src/physical_plan/aggregates datafusion/core/src/physical_plan/windows	mod.rs mod.rs	1260:1269 382:390	view
7	x 2	datafusion/core/src/physical_plan/joins datafusion/core/src/physical_plan/joins	symmetric_hash_join.rs sort_merge_join.rs	438:446 254:262	view
7	x 2	datafusion/physical-expr...gate/groups_accumulator datafusion/physical-expr...gate/groups_accumulator	bool_op.rs prim_op.rs	125:134 127:136	view
7	x 2	datafusion/proto/src/physical_plan datafusion/proto/src/logical_plan	mod.rs mod.rs	79:87 202:210	view
7	x 2	datafusion/proto/src/physical_plan datafusion/proto/src/logical_plan	mod.rs mod.rs	1362:1370 157:165	view
6	x 3	datafusion/core/src/datasource/physical_plan datafusion/core/src/datasource/physical_plan datafusion/core/src/datasource/physical_plan	arrow_file.rs avro.rs json.rs	72:79 69:76 89:96	view
6	x 2	datafusion/core/src/physical_plan/aggregates datafusion/core/src/physical_plan	mod.rs memory.rs	1673:1680 119:126	view
6	x 2	datafusion/core/src/physical_plan/joins datafusion/core/src/physical_plan/joins	sort_merge_join.rs hash_join.rs	372:382 506:516	view
6	x 2	datafusion/core/src/execution datafusion/execution/src	context.rs task.rs	1989:1997 147:155	view
6	x 2	datafusion/core/src/execution datafusion/execution/src	context.rs task.rs	1999:2007 157:165	view
6	x 2	datafusion/physical-expr/src/aggregate datafusion/physical-expr/src/aggregate	covariance.rs correlation.rs	122:129 122:129	view