spotify / tfx - Duplication

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

9% duplication:

39,356 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
3,712 duplicated lines

650 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Duplication Between Components (50+ lines)

Download: SVG DOT (open online Graphviz editor)

From Component --> To Component	Duplicated Lines	File Pairs	Details
package_build (63%) --> ROOT (19%)	91	2 file pairs	details...

Open 2D force graph... Open 3D force graph...

Show more details on duplication between components...

Longest Duplicates

The list of 50 longest duplicates.

See data for all 650 duplicates...

Size	#	Folders	Files	Lines	Code
63	x 2	tfx/orchestration/launcher tfx/orchestration/portable	kubernetes_component_launcher.py kubernetes_executor_operator.py	88:161 (48%) 87:160 (48%)	view
34	x 2	tfx/experimental/templates/taxi tfx/experimental/templates/taxi	data_validation.ipynb model_analysis.ipynb	92:125 (27%) 72:105 (32%)	view
31	x 2	package_build/tfx ROOT	pyproject.toml pyproject.toml	7:37 (86%) 7:37 (65%)	view
29	x 2	tfx/orchestration/portab...esolution/mlmd_resolver tfx/orchestration/portab...esolution/mlmd_resolver	metadata_resolver.py metadata_resolver.py	167:231 (6%) 462:526 (6%)	view
29	x 2	tfx/tools/cli/commands tfx/tools/cli/commands	pipeline.py pipeline.py	92:120 (10%) 170:198 (10%)	view
29	x 2	tfx/tools/cli/commands tfx/tools/cli/commands	run.py run.py	41:69 (11%) 210:238 (11%)	view
29	x 2	package_build/ml-pipelines-sdk ROOT	pyproject.toml pyproject.toml	9:37 (80%) 9:37 (61%)	view
29	x 2	package_build/ml-pipelines-sdk package_build/tfx	pyproject.toml pyproject.toml	9:37 (80%) 9:37 (80%)	view
28	x 2	tfx/orchestration/portab...esolution/mlmd_resolver tfx/orchestration/portab...esolution/mlmd_resolver	metadata_resolver.py metadata_resolver.py	301:332 (6%) 590:621 (6%)	view
26	x 2	tfx/tools/cli/commands tfx/tools/cli/commands	run.py run.py	32:57 (10%) 156:181 (10%)	view
24	x 2	tfx/experimental/templates/penguin tfx/experimental/templates/taxi	local_runner.py local_runner.py	28:80 (100%) 28:80 (100%)	view
23	x 2	tfx/tools/cli/handler tfx/tools/cli/handler	airflow_handler.py beam_handler.py	54:88 (22%) 43:77 (34%)	view
23	x 2	tfx/tools/cli/commands tfx/tools/cli/commands	run.py run.py	204:226 (9%) 259:281 (9%)	view
18	x 2	tfx/orchestration/kubeflow/v2 tfx/orchestration/kubeflow/v2	step_builder.py step_builder.py	509:529 (3%) 576:594 (3%)	view
18	x 2	tfx/dsl/components/common tfx/dsl/components/common	importer.py resolver.py	217:235 (9%) 135:153 (12%)	view
18	x 2	tfx/proto/orchestration tfx/proto/orchestration	proto pipeline.proto placeholder.proto	1:20 (2%) 1:20 (6%)	view
17	x 2	tfx/tools/cli/commands tfx/tools/cli/commands	run.py run.py	41:57 (6%) 265:281 (6%)	view
17	x 2	tfx/tools/cli/commands tfx/tools/cli/commands	pipeline.py pipeline.py	98:114 (6%) 228:244 (6%)	view
17	x 2	tfx/tools/cli/commands tfx/tools/cli/commands	run.py run.py	41:57 (6%) 126:142 (6%)	view
17	x 2	tfx/tools/cli/commands tfx/tools/cli/commands	run.py run.py	126:142 (6%) 265:281 (6%)	view
17	x 2	tfx/tools/cli/commands tfx/tools/cli/commands	pipeline.py pipeline.py	98:114 (6%) 261:277 (6%)	view
17	x 2	tfx/tools/cli/commands tfx/tools/cli/commands	pipeline.py pipeline.py	176:192 (6%) 228:244 (6%)	view
17	x 2	tfx/tools/cli/commands tfx/tools/cli/commands	pipeline.py pipeline.py	176:192 (6%) 261:277 (6%)	view
17	x 2	tfx/tools/cli/commands tfx/tools/cli/commands	run.py run.py	126:142 (6%) 165:181 (6%)	view
17	x 2	tfx/tools/cli/commands tfx/tools/cli/commands	run.py run.py	126:142 (6%) 210:226 (6%)	view
17	x 2	tfx/types tfx/types	standard_component_specs.py standard_component_specs.py	232:248 (3%) 255:271 (3%)	view
17	x 2	tfx/tools/cli/commands tfx/tools/cli/commands	pipeline.py pipeline.py	228:244 (6%) 261:277 (6%)	view
17	x 2	tfx/tools/cli/commands tfx/tools/cli/commands	run.py run.py	165:181 (6%) 265:281 (6%)	view
17	x 2	tfx/experimental/templates/taxi tfx/experimental/templates/taxi	data_validation.ipynb model_analysis.ipynb	17:33 (13%) 17:33 (16%)	view
17	x 2	tfx/tools/cli/commands tfx/tools/cli/commands	run.py run.py	165:181 (6%) 210:226 (6%)	view
16	x 2	tfx/orchestration/portab...esolution/mlmd_resolver tfx/orchestration/portab...esolution/mlmd_resolver	metadata_resolver.py metadata_resolver.py	435:451 (3%) 727:743 (3%)	view
16	x 2	tfx/proto/orchestration tfx/proto/orchestration	proto driver_output.proto execution_invocation.proto	1:19 (59%) 1:19 (28%)	view
16	x 2	tfx/proto/orchestration tfx/proto/orchestration	proto execution_result.proto placeholder.proto	1:19 (33%) 1:18 (5%)	view
16	x 2	tfx/orchestration/portable tfx/orchestration/portable	launcher.py launcher.py	279:294 (3%) 353:368 (3%)	view
16	x 2	tfx/tools/cli/commands tfx/tools/cli/commands	pipeline.py pipeline.py	60:75 (5%) 290:305 (5%)	view
16	x 2	tfx/proto/orchestration tfx/proto/orchestration	proto execution_result.proto pipeline.proto	1:19 (33%) 1:18 (2%)	view
16	x 2	tfx/orchestration/launcher tfx/orchestration/portable	kubernetes_component_launcher.py kubernetes_executor_operator.py	195:210 (12%) 191:206 (12%)	view
15	x 2	tfx/proto/orchestration tfx/proto/orchestration	proto driver_output.proto placeholder.proto	1:17 (55%) 1:16 (5%)	view
15	x 2	tfx/proto/orchestration tfx/proto/orchestration	proto driver_output.proto pipeline.proto	1:17 (55%) 1:16 (1%)	view
15	x 2	tfx/proto/orchestration tfx/proto/orchestration	proto execution_invocation.proto execution_result.proto	1:17 (26%) 1:17 (31%)	view
15	x 2	tfx/proto/orchestration tfx/proto/orchestration	proto driver_output.proto execution_hook.proto	2:19 (55%) 2:19 (45%)	view
15	x 2	tfx/proto/orchestration tfx/proto/orchestration	proto driver_output.proto execution_watcher.proto	2:19 (55%) 2:19 (53%)	view
15	x 2	tfx/proto/orchestration tfx/proto/orchestration	proto execution_hook.proto execution_watcher.proto	2:19 (45%) 2:19 (53%)	view
15	x 2	tfx/proto/orchestration tfx/proto/orchestration	proto execution_invocation.proto placeholder.proto	1:17 (26%) 1:16 (5%)	view
15	x 2	tfx/proto/orchestration tfx/proto/orchestration	proto execution_watcher.proto run_state.proto	1:17 (53%) 1:17 (30%)	view
15	x 2	tfx/proto/orchestration tfx/proto/orchestration	proto garbage_collection_policy.proto placeholder.proto	1:16 (21%) 1:16 (5%)	view
15	x 2	tfx/proto/orchestration tfx/proto/orchestration	proto execution_watcher.proto metadata.proto	1:17 (53%) 1:16 (28%)	view
15	x 2	tfx/proto/orchestration tfx/proto/orchestration	proto execution_invocation.proto garbage_collection_policy.proto	1:17 (26%) 1:16 (21%)	view
15	x 2	tfx/proto/orchestration tfx/proto/orchestration	proto execution_result.proto garbage_collection_policy.proto	1:17 (31%) 1:16 (21%)	view
15	x 2	tfx/proto/orchestration tfx/proto/orchestration	proto driver_output.proto garbage_collection_policy.proto	1:17 (55%) 1:16 (21%)	view