GoogleCloudPlatform / data-analytics-golden-demo

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

49% duplication:

67,197 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
33,322 duplicated lines

82,851 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Duplication Between Components (50+ lines)

Download: SVG DOT (open online Graphviz editor)

From Component --> To Component	Duplicated Lines	File Pairs	Details
colab-enterprise (11%) --> data-analytics-demos (16%)	5958	44 file pairs	details...
data-analytics-demos (3%) --> terraform-modules (10%)	1246	110 file pairs	details...
cloud-composer (7%) --> deprecated-code (30%)	854	391 file pairs	details...
data-analytics-demos (1%) --> terraform (34%)	567	2 file pairs	details...
ROOT (54%) --> data-analytics-demos (1%)	467	7 file pairs	details...
deprecated-code (10%) --> terraform-modules (2%)	306	45 file pairs	details...
dataproc (13%) --> sql-scripts (2%)	276	2 file pairs	details...
colab-enterprise (<1%) --> dataplex (21%)	270	4 file pairs	details...
data-analytics-demos (<1%) --> deprecated-code (8%)	235	40 file pairs	details...
ROOT (26%) --> colab-enterprise (<1%)	224	5 file pairs	details...

Open 2D force graph... Open 3D force graph...

Show more details on duplication between components...

Longest Duplicates

The list of 50 longest duplicates.

See data for all 82,851 duplicates...

Size	#	Folders	Files	Lines	Code
900	x 2	cloud-composer/data/terraform/dataplex-with-hms cloud-composer/data/terraform/dataplex	terraform.tf terraform.tf	129:1186 (92%) 142:1199 (90%)	view
251	x 2	data-analytics-demos/big...rnance/colab-enterprise data-analytics-demos/big...rnance/colab-enterprise	02-Data-Profile.ipynb 03-Data-Insights.ipynb	96:346 (32%) 91:341 (33%)	view
162	x 2	cloud-composer/dags cloud-composer/dags	sample-dataplex-deploy.py sample-dataplex-with-hms-deploy.py	41:250 (97%) 41:250 (97%)	view
139	x 2	sql-scripts/rideshare_llm_curated sql-scripts/rideshare_llm_enriched	sp_demo_duet_ai.sql sp_step_01_quantitative_analysis.sql	104:262 (88%) 108:266 (69%)	view
137	x 2	cloud-composer/data/terraform/dataplex-with-hms cloud-composer/data/terraform/dataplex	variables.tf variables.tf	25:181 (100%) 25:181 (100%)	view
116	x 2	dataproc sql-scripts/rideshare_lakehouse_enriched	rideshare_iceberg_serverless.py sp_iceberg_spark_transformation.py	97:249 (72%) 103:255 (78%)	view
113	x 2	data-analytics-demos/big...dules/apis-batch-enable terraform-modules/apis-batch-enable	tf-batch-enable-apis.tf tf-batch-enable-apis.tf	26:206 (40%) 26:206 (50%)	view
109	x 2	data-analytics-demos/big...rnance/colab-enterprise data-analytics-demos/big...rnance/colab-enterprise	04-Data-Governance.ipynb 06-Data-Discovery-Scan.ipynb	121:229 (3%) 93:201 (14%)	view
103	x 2	cloud-composer/dags cloud-composer/dags	sample-dataplex-deploy.py sample-terraform-bucket-demo-deploy.py	112:250 (62%) 87:225 (73%)	view
103	x 2	cloud-composer/dags cloud-composer/dags	sample-dataplex-with-hms-deploy.py sample-terraform-bucket-demo-deploy.py	112:250 (62%) 87:225 (73%)	view
94	x 2	data-analytics-demos/big...rnance/colab-enterprise data-analytics-demos/big...rnance/colab-enterprise	03-Data-Insights.ipynb 05-Data-Quality.ipynb	91:184 (12%) 92:185 (6%)	view
94	x 2	data-analytics-demos/big...rnance/colab-enterprise data-analytics-demos/big...rnance/colab-enterprise	02-Data-Profile.ipynb 05-Data-Quality.ipynb	96:189 (12%) 92:185 (6%)	view
88	x 2	data-analytics-demos/big...rnance/colab-enterprise data-analytics-demos/big...rnance/colab-enterprise	03-Data-Insights.ipynb 04-Data-Governance.ipynb	202:289 (11%) 232:319 (2%)	view
88	x 2	data-analytics-demos/big...rnance/colab-enterprise data-analytics-demos/big...rnance/colab-enterprise	02-Data-Profile.ipynb 04-Data-Governance.ipynb	207:294 (11%) 232:319 (2%)	view
82	x 2	colab-enterprise data-analytics-demos/big...rnance/colab-enterprise	BigQuery-Analytics-with-GenAI.ipynb 07-Sensitive-Data-Protection-Scan.ipynb	86:167 (4%) 89:170 (10%)	view
81	x 2	data-analytics-demos/big...rnance/colab-enterprise data-analytics-demos/big...rnance/colab-enterprise	04-Data-Governance.ipynb 06-Data-Discovery-Scan.ipynb	242:322 (2%) 218:298 (10%)	view
78	x 2	data-analytics-demos/big...rnance/colab-enterprise data-analytics-demos/big...rnance/colab-enterprise	03-Data-Insights.ipynb 06-Data-Discovery-Scan.ipynb	212:289 (10%) 218:295 (10%)	view
78	x 2	data-analytics-demos/big...rnance/colab-enterprise data-analytics-demos/big...rnance/colab-enterprise	02-Data-Profile.ipynb 06-Data-Discovery-Scan.ipynb	217:294 (10%) 218:295 (10%)	view
77	x 2	data-analytics-demos/big...rnance/colab-enterprise data-analytics-demos/big...rnance/colab-enterprise	03-Data-Insights.ipynb 05-Data-Quality.ipynb	265:341 (10%) 276:352 (5%)	view
77	x 2	data-analytics-demos/big...rnance/colab-enterprise data-analytics-demos/big...rnance/colab-enterprise	02-Data-Profile.ipynb 05-Data-Quality.ipynb	270:346 (10%) 276:352 (5%)	view
66	x 2	colab-enterprise colab-enterprise	GenAI-Generate-Table-and-Column-Descr... Veo-2-text-to-video.ipynb	312:377 (9%) 376:441 (8%)	view
61	x 2	data-analytics-demos/big...ta-governance/terraform terraform	tf-main.tf tf-main.tf	50:158 (24%) 50:147 (11%)	view
58	x 2	data-analytics-demos/big...rnance/colab-enterprise data-analytics-demos/big...rnance/colab-enterprise	04-Data-Governance.ipynb 04-Data-Governance.ipynb	2452:2509 (1%) 3036:3093 (1%)	view
57	x 2	data-analytics-demos/big...rnance/colab-enterprise data-analytics-demos/big...rnance/colab-enterprise	04-Data-Governance.ipynb 04-Data-Governance.ipynb	2357:2413 (1%) 2941:2997 (1%)	view
56	x 2	colab-enterprise data-analytics-demos/big...rnance/colab-enterprise	BigQuery-Analytics-with-GenAI.ipynb 04-Data-Governance.ipynb	230:285 (3%) 267:322 (1%)	view
56	x 2	colab-enterprise data-analytics-demos/big...rnance/colab-enterprise	BigQuery-Analytics-with-GenAI.ipynb 06-Data-Discovery-Scan.ipynb	230:285 (3%) 243:298 (7%)	view
56	x 2	cloud-composer/dags cloud-composer/dags	sample-dataplex-run-data-quality.py sample-rideshare-run-data-quality.py	143:213 (20%) 142:212 (20%)	view
56	x 2	data-analytics-demos/big...rnance/colab-enterprise data-analytics-demos/big...rnance/colab-enterprise	Template.ipynb Template.ipynb	357:412 (7%) 465:520 (7%)	view
54	x 2	data-analytics-demos/big...rnance/colab-enterprise data-analytics-demos/big...rnance/colab-enterprise	02-Data-Profile.ipynb 06-Data-Discovery-Scan.ipynb	96:149 (7%) 93:146 (6%)	view
54	x 2	data-analytics-demos/big...rnance/colab-enterprise data-analytics-demos/big...rnance/colab-enterprise	03-Data-Insights.ipynb 06-Data-Discovery-Scan.ipynb	147:200 (7%) 147:200 (6%)	view
54	x 2	data-analytics-demos/big...rnance/colab-enterprise data-analytics-demos/big...rnance/colab-enterprise	03-Data-Insights.ipynb 04-Data-Governance.ipynb	91:144 (7%) 121:174 (1%)	view
54	x 2	data-analytics-demos/big...rnance/colab-enterprise data-analytics-demos/big...rnance/colab-enterprise	02-Data-Profile.ipynb 04-Data-Governance.ipynb	152:205 (7%) 175:228 (1%)	view
54	x 2	data-analytics-demos/big...rnance/colab-enterprise data-analytics-demos/big...rnance/colab-enterprise	03-Data-Insights.ipynb 06-Data-Discovery-Scan.ipynb	91:144 (7%) 93:146 (6%)	view
54	x 2	data-analytics-demos/big...rnance/colab-enterprise data-analytics-demos/big...rnance/colab-enterprise	03-Data-Insights.ipynb 04-Data-Governance.ipynb	147:200 (7%) 175:228 (1%)	view
54	x 2	data-analytics-demos/big...rnance/colab-enterprise data-analytics-demos/big...rnance/colab-enterprise	02-Data-Profile.ipynb 06-Data-Discovery-Scan.ipynb	152:205 (7%) 147:200 (6%)	view
54	x 2	data-analytics-demos/big...rnance/colab-enterprise data-analytics-demos/big...rnance/colab-enterprise	04-Data-Governance.ipynb 05-Data-Quality.ipynb	121:174 (1%) 92:145 (3%)	view
54	x 2	data-analytics-demos/big...rnance/colab-enterprise data-analytics-demos/big...rnance/colab-enterprise	05-Data-Quality.ipynb 06-Data-Discovery-Scan.ipynb	92:145 (3%) 93:146 (6%)	view
54	x 2	data-analytics-demos/big...rnance/colab-enterprise data-analytics-demos/big...rnance/colab-enterprise	02-Data-Profile.ipynb 04-Data-Governance.ipynb	96:149 (7%) 121:174 (1%)	view
53	x 2	colab-enterprise data-analytics-demos/big...rnance/colab-enterprise	BigQuery-Analytics-with-GenAI.ipynb 03-Data-Insights.ipynb	230:282 (2%) 237:289 (7%)	view
53	x 2	colab-enterprise data-analytics-demos/big...rnance/colab-enterprise	BigQuery-Analytics-with-GenAI.ipynb 02-Data-Profile.ipynb	230:282 (2%) 242:294 (6%)	view
53	x 2	colab-enterprise colab-enterprise	rideshare_llm_step_02_driver_themes.i... rideshare_llm_step_03_customer_themes...	181:233 (18%) 180:232 (18%)	view
52	x 2	cloud-composer/dags cloud-composer/dags	sample-dataplex-run-data-quality.py sample-rideshare-run-data-quality.py	76:140 (18%) 75:139 (18%)	view
51	x 2	colab-enterprise colab-enterprise	rideshare_llm_ai_lakehouse_embeddings... rideshare_llm_ai_lakehouse_embeddings...	7660:7710 (<1%) 7894:7944 (<1%)	view
51	x 2	colab-enterprise colab-enterprise	rideshare_llm_ai_lakehouse_embeddings... rideshare_llm_ai_lakehouse_embeddings...	7660:7710 (<1%) 7826:7876 (<1%)	view
51	x 2	colab-enterprise colab-enterprise	rideshare_llm_ai_lakehouse_embeddings... rideshare_llm_ai_lakehouse_embeddings...	3045:3095 (<1%) 4157:4207 (<1%)	view
51	x 2	colab-enterprise colab-enterprise	rideshare_llm_ai_lakehouse_embeddings... rideshare_llm_ai_lakehouse_embeddings...	7002:7052 (<1%) 7105:7155 (<1%)	view
51	x 2	colab-enterprise colab-enterprise	rideshare_llm_ai_lakehouse_embeddings... rideshare_llm_ai_lakehouse_embeddings...	7660:7710 (<1%) 7712:7762 (<1%)	view
51	x 2	colab-enterprise colab-enterprise	rideshare_llm_ai_lakehouse_embeddings... rideshare_llm_ai_lakehouse_embeddings...	7002:7052 (<1%) 7200:7250 (<1%)	view
51	x 2	colab-enterprise colab-enterprise	rideshare_llm_ai_lakehouse_embeddings... rideshare_llm_ai_lakehouse_embeddings...	3045:3095 (<1%) 4039:4089 (<1%)	view
51	x 2	colab-enterprise colab-enterprise	rideshare_llm_ai_lakehouse_embeddings... rideshare_llm_ai_lakehouse_embeddings...	7002:7052 (<1%) 7274:7324 (<1%)	view

Duplicated Units

The list of top 2 duplicated units.

See data for all 2 unit duplicates...

Size	#	Folders	Files	Lines	Code
25	x 3	cloud-composer/dags cloud-composer/dags cloud-composer/dags	sample-dataplex-with-hms-deploy.py sample-terraform-bucket-demo-deploy.py sample-dataplex-deploy.py	0:0 0:0 0:0	view
22	x 2	deprecated-code/dags cloud-composer/dags	sample-bigquery-export-spanner-import.py sample-bigquery-start-spanner.py	0:0 0:0	view