GoogleCloudPlatform / ai-infra-cluster-provisioning

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

76% duplication:

9,561 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
7,333 duplicated lines

3,640 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Duplication Between Components (50+ lines)

Download: SVG DOT (open online Graphviz editor)

From Component --> To Component	Duplicated Lines	File Pairs	Details
a3-mega (104%) --> a3 (73%)	5291	1050 file pairs	details...
a2 (109%) --> a3 (36%)	2624	560 file pairs	details...
a2 (102%) --> a3-mega (48%)	2452	480 file pairs	details...

Open 2D force graph... Open 3D force graph...

Show more details on duplication between components...

Longest Duplicates

The list of 50 longest duplicates.

See data for all 3,640 duplicates...

Size	#	Folders	Files	Lines	Code
256	x 2	a3-mega/terraform/modules/cluster/mig-cos a3/terraform/modules/cluster/mig-cos	variables.tf variables.tf	1:325 (100%) 1:325 (100%)	view
230	x 2	a3-mega/terraform/modules/cluster/gke a3/terraform/modules/cluster/gke	main.tf main.tf	1:311 (97%) 1:311 (97%)	view
219	x 2	a3-mega/terraform/modules/common/instance_template a3/terraform/modules/common/instance_template	variables.tf variables.tf	1:275 (100%) 1:275 (100%)	view
217	x 2	a3-mega/terraform/modules/cluster/mig a3/terraform/modules/cluster/mig	variables.tf variables.tf	29:312 (88%) 29:312 (88%)	view
201	x 2	a3-mega/terraform/module...uster/mig-cos/cloudinit a3/terraform/modules/cluster/mig-cos/cloudinit	main.tf main.tf	1:209 (100%) 1:209 (100%)	view
149	x 2	sample_workloads/lit-gpt-demo/utilities sample_workloads/megatron-gke/docker	monitor_collectives.py monitor_collectives.py	79:396 (52%) 54:486 (57%)	view
125	x 2	a3-mega/terraform/modules/common/instance_template a3/terraform/modules/common/instance_template	main.tf main.tf	1:144 (100%) 1:144 (100%)	view
123	x 2	a3-mega/terraform/modules/cluster/mig a3/terraform/modules/cluster/mig	main.tf main.tf	1:138 (100%) 1:138 (100%)	view
117	x 2	a3-mega/terraform/modules/common/network a3/terraform/modules/common/network	main.tf main.tf	1:138 (77%) 1:138 (77%)	view
107	x 2	a2/terraform/modules/common/instance_template a3-mega/terraform/modules/common/instance_template	variables.tf variables.tf	1:131 (52%) 1:131 (48%)	view
107	x 2	a2/terraform/modules/common/instance_template a3/terraform/modules/common/instance_template	variables.tf variables.tf	1:131 (52%) 1:131 (48%)	view
90	x 2	a3-mega/terraform/modules/cluster/mig-cos a3/terraform/modules/cluster/mig-cos	main.tf main.tf	1:101 (100%) 1:101 (100%)	view
86	x 2	a3-mega/terraform/module...uster/mig-cos/cloudinit a3/terraform/modules/cluster/mig-cos/cloudinit	variables.tf variables.tf	1:97 (100%) 1:97 (100%)	view
81	x 2	a2/terraform/modules/common/dashboard a3/terraform/modules/common/dashboard	main.tf main.tf	1:91 (100%) 1:91 (100%)	view
81	x 2	a3-mega/terraform/modules/common/dashboard a3/terraform/modules/common/dashboard	main.tf main.tf	1:91 (100%) 1:91 (100%)	view
81	x 2	a2/terraform/modules/common/instance_template a3-mega/terraform/modules/common/instance_template	variables.tf variables.tf	131:235 (39%) 141:245 (36%)	view
81	x 2	a2/terraform/modules/common/instance_template a3/terraform/modules/common/instance_template	variables.tf variables.tf	131:235 (39%) 141:245 (36%)	view
81	x 2	a2/terraform/modules/common/dashboard a3-mega/terraform/modules/common/dashboard	main.tf main.tf	1:91 (100%) 1:91 (100%)	view
75	x 2	a3-mega/terraform/module...uster/gke/kubectl-apply a3/terraform/modules/cluster/gke/kubectl-apply	aiinfra-ksa.tf aiinfra-ksa.tf	1:92 (100%) 1:92 (100%)	view
68	x 2	a2/terraform/modules/common/instance_template a3-mega/terraform/modules/common/instance_template	main.tf main.tf	1:79 (57%) 1:79 (54%)	view
68	x 2	a2/terraform/modules/common/instance_template a3/terraform/modules/common/instance_template	main.tf main.tf	1:79 (57%) 1:79 (54%)	view
64	x 2	a3-mega/terraform/module...uster/gke/kubectl-apply a3/terraform/modules/cluster/gke/kubectl-apply	variables.tf variables.tf	1:73 (100%) 1:73 (100%)	view
52	x 2	a2/terraform/modules/cluster/mig a3/terraform/modules/cluster/mig	variables.tf variables.tf	242:307 (20%) 236:301 (21%)	view
52	x 2	a2/terraform/modules/cluster/mig a3-mega/terraform/modules/cluster/mig	variables.tf variables.tf	242:307 (20%) 236:301 (21%)	view
52	x 2	a3-mega/terraform/modules/cluster/gke a3/terraform/modules/cluster/gke	variables.tf variables.tf	1:61 (55%) 1:61 (55%)	view
51	x 2	a2/terraform/modules/cluster/mig a3/terraform/modules/cluster/mig	main.tf main.tf	1:55 (37%) 1:55 (41%)	view
51	x 2	a2/terraform/modules/cluster/mig a3-mega/terraform/modules/cluster/mig	main.tf main.tf	1:55 (37%) 1:55 (41%)	view
50	x 2	a3-mega/terraform/modules/common/network a3/terraform/modules/common/network	outputs.tf outputs.tf	1:55 (100%) 1:55 (100%)	view
48	x 2	sample_workloads/lit-gpt-demo/utilities sample_workloads/megatron-gke/docker	monitor_collectives.py monitor_collectives.py	411:516 (16%) 504:641 (18%)	view
46	x 2	a3-mega/terraform/modules/common/resource_policy a3/terraform/modules/common/resource_policy	variables.tf variables.tf	1:54 (100%) 1:54 (100%)	view
45	x 2	a3-mega/terraform/modules/common/resource_policy a3/terraform/modules/common/resource_policy	main.tf main.tf	1:48 (100%) 1:48 (100%)	view
44	x 2	a3-mega/terraform/modules/cluster/mig-cos a3/terraform/modules/cluster/mig	variables.tf variables.tf	1:48 (17%) 1:48 (18%)	view
44	x 2	a3/terraform/modules/cluster/mig-cos a3/terraform/modules/cluster/mig	variables.tf variables.tf	1:48 (17%) 1:48 (18%)	view
41	x 2	a3/terraform/modules/cluster/mig-cos a3/terraform/modules/cluster/mig	variables.tf variables.tf	257:309 (16%) 226:278 (16%)	view
41	x 2	a3-mega/terraform/modules/cluster/mig-cos a3/terraform/modules/cluster/mig	variables.tf variables.tf	257:309 (16%) 226:278 (16%)	view
41	x 2	a3-mega/terraform/modules/cluster/mig a3/terraform/modules/cluster/mig-cos	variables.tf variables.tf	226:278 (16%) 257:309 (16%)	view
41	x 2	a3-mega/terraform/modules/cluster/mig-cos a3-mega/terraform/modules/cluster/mig	variables.tf variables.tf	257:309 (16%) 226:278 (16%)	view
40	x 2	a3-mega/terraform/modules/common/network a3/terraform/modules/common/network	variables.tf variables.tf	1:46 (100%) 1:46 (100%)	view
38	x 2	a3-mega/terraform/module.../instance_group_manager a3/terraform/modules/common/instance_group_manager	main.tf main.tf	1:42 (100%) 1:42 (100%)	view
35	x 2	a2/terraform/modules/common/dashboard a3-mega/terraform/modules/common/dashboard	variables.tf variables.tf	1:40 (100%) 1:40 (100%)	view
35	x 2	a3-mega/terraform/modules/cluster/mig-cos a3-mega/terraform/modules/cluster/mig	variables.tf variables.tf	104:155 (13%) 60:111 (14%)	view
35	x 2	a2/terraform/modules/cluster/mig a3-mega/terraform/modules/cluster/mig	main.tf main.tf	82:120 (25%) 82:120 (28%)	view
35	x 2	a2/terraform/modules/cluster/mig a3/terraform/modules/cluster/mig	main.tf main.tf	82:120 (25%) 82:120 (28%)	view
35	x 2	a3-mega/terraform/modules/cluster/mig-cos a3/terraform/modules/cluster/mig	variables.tf variables.tf	104:155 (13%) 60:111 (14%)	view
35	x 2	a3/terraform/modules/cluster/mig-cos a3/terraform/modules/cluster/mig	variables.tf variables.tf	104:155 (13%) 60:111 (14%)	view
35	x 2	a2/terraform/modules/common/dashboard a3/terraform/modules/common/dashboard	variables.tf variables.tf	1:40 (100%) 1:40 (100%)	view
35	x 2	a3-mega/terraform/modules/common/dashboard a3/terraform/modules/common/dashboard	variables.tf variables.tf	1:40 (100%) 1:40 (100%)	view
35	x 2	a3-mega/terraform/modules/cluster/mig a3/terraform/modules/cluster/mig-cos	variables.tf variables.tf	60:111 (14%) 104:155 (13%)	view
33	x 2	a2/terraform/modules/cluster/mig a3/terraform/modules/cluster/mig-cos	variables.tf variables.tf	242:284 (13%) 267:309 (12%)	view
33	x 2	a3-mega/terraform/modules/cluster/gke a3/terraform/modules/cluster/gke	variables.tf variables.tf	71:108 (35%) 71:108 (35%)	view

Duplicated Units

The list of top 1 duplicated units.

See data for all 1 unit duplicate

Size	#	Folders	Files	Lines	Code
27	x 2	sample_workloads/lit-gpt-demo/utilities sample_workloads/megatron-gke/docker	monitor_collectives.py monitor_collectives.py	0:0 0:0	view