GoogleCloudPlatform / data-migration-tool

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

40% duplication:

10,048 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
4,048 duplicated lines

945 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Longest Duplicates

The list of 50 longest duplicates.

See data for all 945 duplicates...

Size	#	Folders	Files	Lines	Code
138	x 2	src/datamigration/dags/redshift src/datamigration/dags/teradata	redshift_transfer_run_log_dag.py teradata_transfer_run_log_dag.py	306:488 (28%) 413:580 (23%)	view
74	x 2	src/datamigration/dags/redshift src/datamigration/dags/teradata	redshift_transfer_run_log_dag.py teradata_transfer_run_log_dag.py	498:578 (15%) 595:675 (12%)	view
73	x 2	src/datamigration/dags/redshift src/datamigration/dags/teradata	redshift_transfer_run_log_dag.py teradata_transfer_run_log_dag.py	185:280 (15%) 216:309 (12%)	view
59	x 2	src/translation/scripts/hive src/translation/scripts/hive	extract_hive_ddls.py extract_hive_ddls_manual.py	63:144 (35%) 66:147 (28%)	view
49	x 2	src/datamigration/dags/redshift src/datamigration/dags/teradata	redshift_data_load_dag.py teradata_data_load_dag.py	80:158 (20%) 104:182 (15%)	view
45	x 2	src/translation/dags src/translation/dvt	validation_dag.py main.py	58:120 (13%) 36:96 (19%)	view
43	x 2	terraform/datamigration/hive/gcs terraform/datamigration/teradata/gcs	main.tf main.tf	1:51 (100%) 1:51 (100%)	view
38	x 2	src/datamigration/dags/redshift src/datamigration/dags/teradata	redshift_transfer_run_log_dag.py teradata_transfer_run_log_dag.py	139:183 (7%) 170:214 (6%)	view
36	x 2	terraform/datamigration/hive/gcc terraform/datamigration/teradata/gcc	variables.tf variables.tf	1:42 (76%) 1:42 (76%)	view
29	x 2	src/datamigration/dags/redshift src/datamigration/dags/teradata	redshift_data_load_dag.py teradata_data_load_dag.py	283:315 (12%) 369:401 (9%)	view
28	x 2	terraform/datamigration/redshift/gcc terraform/datamigration/teradata/gcc	variables.tf variables.tf	9:41 (60%) 9:41 (59%)	view
28	x 2	src/datamigration/dags/redshift src/datamigration/dags/teradata	redshift_transfer_run_log_dag.py teradata_transfer_run_log_dag.py	54:86 (5%) 64:96 (4%)	view
28	x 2	terraform/datamigration/hive/gcc terraform/datamigration/redshift/gcc	variables.tf variables.tf	9:41 (59%) 9:41 (60%)	view
28	x 2	src/translation/dags src/translation/dags	validation_crun_dag.py validation_dag.py	148:179 (18%) 180:216 (8%)	view
28	x 2	terraform/datamigration/redshift/bq terraform/datamigration/teradata/bq	main.tf main.tf	1:33 (77%) 1:33 (77%)	view
28	x 2	terraform/datamigration/redshift/cloudrun terraform/datamigration/teradata/cloudrun	main.tf main.tf	31:61 (53%) 30:60 (54%)	view
27	x 2	terraform/datamigration/hive/gcs terraform/datamigration/teradata/gcs	variables.tf variables.tf	32:63 (49%) 26:57 (45%)	view
25	x 2	src/translation/scripts/hive src/translation/scripts/hive	extract_hive_ddls.py extract_hive_ddls_manual.py	157:181 (15%) 186:210 (12%)	view
23	x 2	terraform/datamigration/redshift/pubsub terraform/datamigration/teradata/pubsub	main.tf main.tf	28:55 (52%) 56:82 (34%)	view
23	x 2	terraform/datamigration/redshift/cloudrun terraform/datamigration/teradata/cloudrun	variables.tf variables.tf	8:30 (60%) 8:30 (60%)	view
22	x 2	terraform/datamigration/teradata/gcs terraform/translation/gcs	main.tf main.tf	1:24 (51%) 1:24 (51%)	view
22	x 2	terraform/datamigration/redshift/bq terraform/datamigration/teradata/bq	variables.tf variables.tf	1:26 (56%) 1:26 (56%)	view
22	x 2	src/datamigration/dags/datamigration_utils src/datamigration/dags/datamigration_utils	hive_bq_load_utils.py hive_dvt_check_utils.py	34:55 (6%) 17:38 (26%)	view
22	x 2	terraform/datamigration/hive/gcs terraform/translation/gcs	main.tf main.tf	1:24 (51%) 1:24 (51%)	view
22	x 2	terraform/datamigration/teradata/cloudrun terraform/translation/cloudrun	variables.tf variables.tf	1:22 (57%) 1:22 (25%)	view
21	x 2	src/translation/dags src/translation/dvt	validation_dag.py main.py	279:302 (6%) 211:234 (9%)	view
21	x 2	src/datamigration/dags/datamigration_utils src/datamigration/dags/datamigration_utils	hive_bq_load_utils.py hive_bq_load_utils_inc.py	176:201 (6%) 405:430 (3%)	view
20	x 2	terraform/datamigration/hive/gcs terraform/datamigration/teradata/gcs	variables.tf variables.tf	1:23 (36%) 1:23 (33%)	view
19	x 2	terraform/datamigration/hive/bq terraform/datamigration/hive/bq_inc	variables.tf variables.tf	1:22 (65%) 1:22 (48%)	view
19	x 2	terraform/datamigration/hive/bq terraform/datamigration/redshift/bq	variables.tf variables.tf	1:22 (65%) 1:22 (48%)	view
19	x 2	terraform/datamigration/hive/bq_inc terraform/datamigration/redshift/bq	variables.tf variables.tf	1:22 (48%) 1:22 (48%)	view
19	x 2	terraform/datamigration/hive/bq_inc terraform/datamigration/teradata/bq	variables.tf variables.tf	1:22 (48%) 1:22 (48%)	view
19	x 2	terraform/datamigration/hive/bq terraform/datamigration/teradata/bq	variables.tf variables.tf	1:22 (65%) 1:22 (48%)	view
18	x 2	terraform/datamigration/hive/gcc terraform/datamigration/teradata/gcc	main.tf main.tf	8:30 (40%) 8:30 (40%)	view
18	x 2	terraform/datamigration/hive/gcc terraform/datamigration/redshift/gcc	main.tf main.tf	8:30 (40%) 8:30 (40%)	view
18	x 2	terraform/datamigration/redshift/cloudrun terraform/datamigration/teradata/cloudrun	main.tf main.tf	1:22 (34%) 1:22 (35%)	view
18	x 2	terraform/datamigration/redshift/pubsub terraform/datamigration/teradata/pubsub	main.tf main.tf	1:22 (40%) 1:22 (26%)	view
18	x 2	src/datamigration/dags/datamigration_utils src/datamigration/dags/datamigration_utils	hive_bq_load_utils.py hive_bq_load_utils_inc.py	263:282 (5%) 473:492 (3%)	view
18	x 2	terraform/datamigration/redshift/gcc terraform/datamigration/teradata/gcc	main.tf main.tf	8:30 (40%) 8:30 (40%)	view
17	x 2	src/datamigration/dags/datamigration_utils src/datamigration/dags/datamigration_utils	hive_bq_load_utils_inc.py hive_bq_load_utils_inc.py	326:342 (2%) 368:384 (2%)	view
17	x 2	src/translation/dags src/translation/dvt	validation_dag.py main.py	130:146 (5%) 130:146 (7%)	view
17	x 2	src/datamigration/dags/redshift src/datamigration/dags/teradata	redshift_transfer_run_log_dag.py teradata_transfer_run_log_dag.py	117:137 (3%) 148:168 (2%)	view
16	x 2	src/translation/dags src/translation/dvt	validation_dag.py main.py	223:241 (4%) 153:171 (6%)	view
16	x 2	terraform/datamigration/hive/gcs terraform/datamigration/redshift/gcs	variables.tf variables.tf	45:63 (29%) 31:49 (38%)	view
16	x 2	src/translation/dags src/translation/dags	controller_dag.py schema_dag.py	34:52 (6%) 54:72 (8%)	view
16	x 2	terraform/datamigration/redshift/gcs terraform/datamigration/teradata/gcs	variables.tf variables.tf	31:49 (38%) 39:57 (27%)	view
15	x 2	terraform/datamigration/redshift/cloudrun terraform/translation/cloudrun	variables.tf variables.tf	8:22 (39%) 8:22 (17%)	view
15	x 2	src/translation/dags src/translation/dags	controller_dag.py dml_validation_dag.py	35:52 (6%) 48:64 (5%)	view
15	x 2	terraform/datamigration/redshift/gcc terraform/translation/gcc	variables.tf variables.tf	19:35 (32%) 28:42 (5%)	view
15	x 2	terraform/datamigration/redshift/gcc terraform/datamigration/teradata/gcc	main.tf main.tf	44:60 (34%) 44:60 (34%)	view