GoogleCloudPlatform / ai-on-gke

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

61% duplication:

50,915 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
31,226 duplicated lines

12,697 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Duplication Between Components (50+ lines)

Download: SVG DOT (open online Graphviz editor)

From Component --> To Component	Duplicated Lines	File Pairs	Details
applications (99%) --> ray-on-gke (51%)	2168	57 file pairs	details...
benchmarks (3%) --> tutorials-and-examples (8%)	1173	131 file pairs	details...
benchmarks (3%) --> modules (9%)	1112	358 file pairs	details...
benchmarks (3%) --> slurm-on-gke (103%)	1101	294 file pairs	details...
best-practices (3%) --> tutorials-and-examples (7%)	1049	54 file pairs	details...
infrastructure (19%) --> ray-on-gke (8%)	364	51 file pairs	details...
applications (16%) --> infrastructure (19%)	364	51 file pairs	details...
modules (2%) --> tutorials-and-examples (2%)	348	28 file pairs	details...
modules (2%) --> slurm-on-gke (27%)	289	57 file pairs	details...
infrastructure (13%) --> modules (1%)	241	12 file pairs	details...
slurm-on-gke (22%) --> tutorials-and-examples (1%)	235	21 file pairs	details...
applications (6%) --> modules (1%)	140	9 file pairs	details...
modules (1%) --> ray-on-gke (3%)	140	9 file pairs	details...
benchmarks (<1%) --> tpu-provisioner (2%)	82	5 file pairs	details...
charts (2%) --> scripts (25%)	60	4 file pairs	details...

Open 2D force graph... Open 3D force graph...

Show more details on duplication between components...

Longest Duplicates

The list of 50 longest duplicates.

See data for all 12,697 duplicates...

Size	#	Folders	Files	Lines	Code
252	x 2	benchmarks/benchmark/too...-load-benchmark/results benchmarks/benchmark/too...-load-benchmark/results	case_20.yaml case_4.yaml	2:253 (99%) 2:253 (99%)	view
252	x 2	benchmarks/benchmark/too...-load-benchmark/results benchmarks/benchmark/too...-load-benchmark/results	case_18.yaml case_2.yaml	2:253 (99%) 2:253 (99%)	view
252	x 2	applications/ray ray-on-gke/templates	metadata.display.yaml metadata.display.yaml	1:253 (100%) 1:253 (100%)	view
252	x 2	benchmarks/benchmark/too...-load-benchmark/results benchmarks/benchmark/too...-load-benchmark/results	case_0.yaml case_16.yaml	2:253 (99%) 2:253 (99%)	view
252	x 2	benchmarks/benchmark/too...-load-benchmark/results benchmarks/benchmark/too...-load-benchmark/results	case_21.yaml case_5.yaml	2:253 (99%) 2:253 (99%)	view
252	x 2	benchmarks/benchmark/too...-load-benchmark/results benchmarks/benchmark/too...-load-benchmark/results	case_19.yaml case_3.yaml	2:253 (99%) 2:253 (99%)	view
252	x 2	benchmarks/benchmark/too...-load-benchmark/results benchmarks/benchmark/too...-load-benchmark/results	case_22.yaml case_6.yaml	2:253 (99%) 2:253 (99%)	view
252	x 2	benchmarks/benchmark/too...-load-benchmark/results benchmarks/benchmark/too...-load-benchmark/results	case_1.yaml case_17.yaml	2:253 (99%) 2:253 (99%)	view
252	x 2	benchmarks/benchmark/too...-load-benchmark/results benchmarks/benchmark/too...-load-benchmark/results	case_23.yaml case_7.yaml	2:253 (99%) 2:253 (99%)	view
243	x 2	applications/ray ray-on-gke/templates	variables.tf variables.tf	15:304 (100%) 15:304 (100%)	view
177	x 2	applications/ray ray-on-gke/templates	main.tf main.tf	19:219 (100%) 19:219 (100%)	view
161	x 2	benchmarks/benchmark/too...-load-benchmark/results benchmarks/benchmark/too...-load-benchmark/results	case_10.yaml case_2.yaml	93:253 (63%) 93:253 (63%)	view
161	x 2	benchmarks/benchmark/too...-load-benchmark/results benchmarks/benchmark/too...-load-benchmark/results	case_17.yaml case_9.yaml	93:253 (63%) 93:253 (63%)	view
161	x 2	benchmarks/benchmark/too...-load-benchmark/results benchmarks/benchmark/too...-load-benchmark/results	case_15.yaml case_23.yaml	93:253 (63%) 93:253 (63%)	view
161	x 2	benchmarks/benchmark/too...-load-benchmark/results benchmarks/benchmark/too...-load-benchmark/results	case_0.yaml case_8.yaml	93:253 (63%) 93:253 (63%)	view
161	x 2	benchmarks/benchmark/too...-load-benchmark/results benchmarks/benchmark/too...-load-benchmark/results	case_13.yaml case_5.yaml	93:253 (63%) 93:253 (63%)	view
161	x 2	benchmarks/benchmark/too...-load-benchmark/results benchmarks/benchmark/too...-load-benchmark/results	case_14.yaml case_22.yaml	93:253 (63%) 93:253 (63%)	view
161	x 2	benchmarks/benchmark/too...-load-benchmark/results benchmarks/benchmark/too...-load-benchmark/results	case_16.yaml case_8.yaml	93:253 (63%) 93:253 (63%)	view
161	x 2	benchmarks/benchmark/too...-load-benchmark/results benchmarks/benchmark/too...-load-benchmark/results	case_11.yaml case_19.yaml	93:253 (63%) 93:253 (63%)	view
161	x 2	benchmarks/benchmark/too...-load-benchmark/results benchmarks/benchmark/too...-load-benchmark/results	case_12.yaml case_4.yaml	93:253 (63%) 93:253 (63%)	view
161	x 2	benchmarks/benchmark/too...-load-benchmark/results benchmarks/benchmark/too...-load-benchmark/results	case_13.yaml case_21.yaml	93:253 (63%) 93:253 (63%)	view
161	x 2	benchmarks/benchmark/too...-load-benchmark/results benchmarks/benchmark/too...-load-benchmark/results	case_10.yaml case_18.yaml	93:253 (63%) 93:253 (63%)	view
161	x 2	benchmarks/benchmark/too...-load-benchmark/results benchmarks/benchmark/too...-load-benchmark/results	case_15.yaml case_7.yaml	93:253 (63%) 93:253 (63%)	view
161	x 2	benchmarks/benchmark/too...-load-benchmark/results benchmarks/benchmark/too...-load-benchmark/results	case_12.yaml case_20.yaml	93:253 (63%) 93:253 (63%)	view
161	x 2	benchmarks/benchmark/too...-load-benchmark/results benchmarks/benchmark/too...-load-benchmark/results	case_11.yaml case_3.yaml	93:253 (63%) 93:253 (63%)	view
161	x 2	benchmarks/benchmark/too...-load-benchmark/results benchmarks/benchmark/too...-load-benchmark/results	case_1.yaml case_9.yaml	93:253 (63%) 93:253 (63%)	view
161	x 2	benchmarks/benchmark/too...-load-benchmark/results benchmarks/benchmark/too...-load-benchmark/results	case_14.yaml case_6.yaml	93:253 (63%) 93:253 (63%)	view
115	x 2	best-practices/gke-batch...iority/workloads/team-a best-practices/gke-batch...iority/workloads/team-a	team-a-high-priority-job-0.yaml team-a-high-priority-job-2.yaml	29:167 (59%) 29:167 (59%)	view
115	x 2	best-practices/gke-batch...cement/workloads/team-a best-practices/gke-batch...cement/workloads/team-a	team-a-compact-job-0.yaml team-a-compact-job-3.yaml	29:167 (56%) 29:167 (56%)	view
115	x 2	best-practices/gke-batch...cement/workloads/team-c best-practices/gke-batch...cement/workloads/team-c	team-c-compact-job-0.yaml team-c-compact-job-3.yaml	29:167 (56%) 29:167 (56%)	view
115	x 2	best-practices/gke-batch...cement/workloads/team-a best-practices/gke-batch...iority/workloads/team-a	team-a-compact-job-3.yaml team-a-high-priority-job-1.yaml	29:167 (56%) 29:167 (59%)	view
115	x 2	best-practices/gke-batch...iority/workloads/team-c best-practices/gke-batch...iority/workloads/team-c	team-c-high-priority-job-3.yaml team-c-low-priority-job-3.yaml	29:167 (59%) 29:167 (60%)	view
115	x 2	best-practices/gke-batch...iority/workloads/team-d best-practices/gke-batch...iority/workloads/team-d	team-d-high-priority-job-1.yaml team-d-high-priority-job-3.yaml	29:167 (59%) 29:167 (59%)	view
115	x 2	best-practices/gke-batch...cement/workloads/team-a best-practices/gke-batch...iority/workloads/team-a	team-a-compact-job-1.yaml team-a-high-priority-job-3.yaml	29:167 (56%) 29:167 (59%)	view
115	x 2	best-practices/gke-batch...iority/workloads/team-b best-practices/gke-batch...iority/workloads/team-b	team-b-high-priority-job-3.yaml team-b-low-priority-job-0.yaml	29:167 (59%) 29:167 (60%)	view
115	x 2	best-practices/gke-batch...cement/workloads/team-b best-practices/gke-batch...iority/workloads/team-b	team-b-compact-job-0.yaml team-b-high-priority-job-1.yaml	29:167 (56%) 29:167 (59%)	view
115	x 2	best-practices/gke-batch...iority/workloads/team-b best-practices/gke-batch...iority/workloads/team-b	team-b-high-priority-job-2.yaml team-b-high-priority-job-3.yaml	29:167 (59%) 29:167 (59%)	view
115	x 2	best-practices/gke-batch...cement/workloads/team-d best-practices/gke-batch...iority/workloads/team-d	team-d-compact-job-1.yaml team-d-low-priority-job-2.yaml	29:167 (56%) 29:167 (60%)	view
115	x 2	best-practices/gke-batch...cement/workloads/team-d best-practices/gke-batch...cement/workloads/team-d	team-d-compact-job-0.yaml team-d-compact-job-2.yaml	29:167 (56%) 29:167 (56%)	view
115	x 2	best-practices/gke-batch...cement/workloads/team-a best-practices/gke-batch...iority/workloads/team-a	team-a-compact-job-1.yaml team-a-low-priority-job-2.yaml	29:167 (56%) 29:167 (60%)	view
115	x 2	best-practices/gke-batch...cement/workloads/team-c best-practices/gke-batch...iority/workloads/team-c	team-c-compact-job-1.yaml team-c-low-priority-job-2.yaml	29:167 (56%) 29:167 (60%)	view
115	x 2	best-practices/gke-batch...cement/workloads/team-a best-practices/gke-batch...iority/workloads/team-a	team-a-compact-job-2.yaml team-a-low-priority-job-3.yaml	29:167 (56%) 29:167 (60%)	view
115	x 2	best-practices/gke-batch...cement/workloads/team-d best-practices/gke-batch...iority/workloads/team-d	team-d-compact-job-1.yaml team-d-high-priority-job-1.yaml	29:167 (56%) 29:167 (59%)	view
115	x 2	best-practices/gke-batch...iority/workloads/team-d best-practices/gke-batch...iority/workloads/team-d	team-d-high-priority-job-1.yaml team-d-low-priority-job-2.yaml	29:167 (59%) 29:167 (60%)	view
115	x 2	best-practices/gke-batch...iority/workloads/team-a best-practices/gke-batch...iority/workloads/team-a	team-a-low-priority-job-0.yaml team-a-low-priority-job-2.yaml	29:167 (60%) 29:167 (60%)	view
115	x 2	best-practices/gke-batch...cement/workloads/team-d best-practices/gke-batch...iority/workloads/team-d	team-d-compact-job-2.yaml team-d-high-priority-job-2.yaml	29:167 (56%) 29:167 (59%)	view
115	x 2	best-practices/gke-batch...cement/workloads/team-b best-practices/gke-batch...cement/workloads/team-b	team-b-compact-job-0.yaml team-b-compact-job-2.yaml	29:167 (56%) 29:167 (56%)	view
115	x 2	best-practices/gke-batch...iority/workloads/team-b best-practices/gke-batch...iority/workloads/team-b	team-b-high-priority-job-3.yaml team-b-low-priority-job-2.yaml	29:167 (59%) 29:167 (60%)	view
115	x 2	best-practices/gke-batch...iority/workloads/team-c best-practices/gke-batch...iority/workloads/team-c	team-c-high-priority-job-2.yaml team-c-low-priority-job-2.yaml	29:167 (59%) 29:167 (60%)	view
115	x 2	best-practices/gke-batch...cement/workloads/team-d best-practices/gke-batch...iority/workloads/team-d	team-d-compact-job-1.yaml team-d-high-priority-job-3.yaml	29:167 (56%) 29:167 (59%)	view

Duplicated Units

The list of top 2 duplicated units.

See data for all 2 unit duplicates...

Size	#	Folders	Files	Lines	Code
60	x 2	tutorials-and-examples/w...chestration/indexed-job best-practices/gke-batch-refarch/jobset	mnist.py mnist.py	0:0 0:0	view
13	x 2	benchmarks/benchmark/tools/model-load-benchmark benchmarks/benchmark/too...l-load-benchmark/runner	main.go runner.go	40:54 20:34	view