GoogleCloudPlatform / dataproc-templates

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

36% duplication:

20,117 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
7,261 duplicated lines

2,514 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Longest Duplicates

The list of 50 longest duplicates.

See data for all 2,514 duplicates...

Size	#	Folders	Files	Lines	Code
62	x 2	notebooks/oracle2bq notebooks/oracle2spanner	OracleToBigQuery_notebook.ipynb OracleToSpanner_notebook.ipynb	695:756 (7%) 713:774 (6%)	view
56	x 2	notebooks/mysql2spanner notebooks/oracle2bq	MySqlToSpanner_notebook.ipynb OracleToBigQuery_notebook.ipynb	708:763 (6%) 701:756 (6%)	view
56	x 2	notebooks/mysql2spanner notebooks/oracle2spanner	MySqlToSpanner_notebook.ipynb OracleToSpanner_notebook.ipynb	708:763 (6%) 719:774 (6%)	view
44	x 2	notebooks/mssql2bq notebooks/postgresql2bq	mssql-to-bigquery-notebook.ipynb postgresql-to-bigquery-notebook.ipynb	643:686 (4%) 835:878 (3%)	view
44	x 2	notebooks/oracle2bq notebooks/oracle2postgres	OracleToBigQuery_notebook.ipynb OracleToPostgres_notebook.ipynb	180:223 (4%) 194:237 (4%)	view
36	x 2	notebooks/hive2bq notebooks/postgresql2bq	HiveToBigquery_notebook.ipynb postgresql-to-bigquery-notebook.ipynb	897:932 (3%) 1104:1139 (3%)	view
35	x 2	notebooks/oracle2bq notebooks/oracle2postgres	OracleToBigQuery_notebook.ipynb OracleToPostgres_notebook.ipynb	121:155 (3%) 135:169 (3%)	view
35	x 2	notebooks/util/jdbc/engines notebooks/util/jdbc/engines	mysql_input_manager.py oracle_input_manager.py	69:107 (25%) 76:116 (22%)	view
35	x 2	notebooks/hive2bq notebooks/mssql2bq	HiveToBigquery_notebook.ipynb mssql-to-bigquery-notebook.ipynb	898:932 (3%) 877:911 (3%)	view
35	x 2	notebooks/mssql2bq notebooks/postgresql2bq	mssql-to-bigquery-notebook.ipynb postgresql-to-bigquery-notebook.ipynb	877:911 (3%) 1105:1139 (3%)	view
32	x 2	notebooks/mysql2spanner notebooks/oracle2spanner	MySqlToSpanner_notebook.ipynb OracleToSpanner_notebook.ipynb	170:201 (3%) 182:213 (3%)	view
31	x 2	notebooks/mysql2spanner notebooks/oracle2bq	MySqlToSpanner_notebook.ipynb OracleToBigQuery_notebook.ipynb	166:196 (3%) 171:201 (3%)	view
31	x 2	notebooks/mssql2bq notebooks/mysql2spanner	mssql-to-bigquery-notebook.ipynb MySqlToSpanner_notebook.ipynb	874:904 (3%) 866:896 (3%)	view
30	x 2	java/src/main/java/com/g.../dataproc/templates/gcs java/src/main/java/com/g...dataproc/templates/jdbc	GCSToSpanner.java JDBCToSpanner.java	73:106 (50%) 76:108 (41%)	view
30	x 2	notebooks/mysql2spanner notebooks/oracle2bq	MySqlToSpanner_notebook.ipynb OracleToBigQuery_notebook.ipynb	1:30 (3%) 1:30 (3%)	view
28	x 2	notebooks/oracle2postgres notebooks/oracle2spanner	OracleToPostgres_notebook.ipynb OracleToSpanner_notebook.ipynb	475:502 (2%) 399:426 (3%)	view
28	x 2	notebooks/hive2bq notebooks/mysql2spanner	HiveToBigquery_notebook.ipynb MySqlToSpanner_notebook.ipynb	898:925 (3%) 869:896 (3%)	view
28	x 2	notebooks/mysql2spanner notebooks/postgresql2bq	MySqlToSpanner_notebook.ipynb postgresql-to-bigquery-notebook.ipynb	869:896 (3%) 1105:1132 (2%)	view
28	x 2	notebooks/mssql2bq notebooks/mssql2postgresql	mssql-to-bigquery-notebook.ipynb mssql-to-postgres-notebook.ipynb	720:747 (3%) 729:756 (3%)	view
28	x 2	notebooks/mssql2bq notebooks/postgresql2bq	mssql-to-bigquery-notebook.ipynb postgresql-to-bigquery-notebook.ipynb	721:748 (3%) 924:951 (2%)	view
27	x 2	notebooks/mysql2spanner notebooks/oracle2bq	MySqlToSpanner_notebook.ipynb OracleToBigQuery_notebook.ipynb	877:903 (2%) 855:881 (3%)	view
27	x 2	notebooks/generic_notebook notebooks/oracle2bq	vertex_pipeline_pyspark.ipynb OracleToBigQuery_notebook.ipynb	528:554 (4%) 855:881 (3%)	view
27	x 2	notebooks/generic_notebook notebooks/oracle2spanner	vertex_pipeline_pyspark.ipynb OracleToSpanner_notebook.ipynb	528:554 (4%) 868:894 (3%)	view
27	x 2	notebooks/oracle2postgres notebooks/oracle2spanner	OracleToPostgres_notebook.ipynb OracleToSpanner_notebook.ipynb	952:978 (2%) 868:894 (3%)	view
27	x 2	notebooks/mssql2postgresql notebooks/postgresql2bq	mssql-to-postgres-notebook.ipynb postgresql-to-bigquery-notebook.ipynb	730:756 (2%) 924:950 (2%)	view
27	x 2	notebooks/oracle2bq notebooks/oracle2spanner	OracleToBigQuery_notebook.ipynb OracleToSpanner_notebook.ipynb	855:881 (3%) 868:894 (3%)	view
27	x 2	notebooks/mysql2spanner notebooks/oracle2postgres	MySqlToSpanner_notebook.ipynb OracleToPostgres_notebook.ipynb	877:903 (2%) 952:978 (2%)	view
27	x 2	notebooks/oracle2bq notebooks/oracle2spanner	OracleToBigQuery_notebook.ipynb OracleToSpanner_notebook.ipynb	175:201 (3%) 182:208 (3%)	view
27	x 2	notebooks/mssql2postgresql notebooks/oracle2postgres	mssql-to-postgres-notebook.ipynb OracleToPostgres_notebook.ipynb	893:919 (2%) 952:978 (2%)	view
27	x 2	notebooks/mssql2postgresql notebooks/oracle2bq	mssql-to-postgres-notebook.ipynb OracleToBigQuery_notebook.ipynb	893:919 (2%) 855:881 (3%)	view
27	x 2	notebooks/mssql2postgresql notebooks/oracle2spanner	mssql-to-postgres-notebook.ipynb OracleToSpanner_notebook.ipynb	893:919 (2%) 868:894 (3%)	view
27	x 2	notebooks/generic_notebook notebooks/oracle2postgres	vertex_pipeline_pyspark.ipynb OracleToPostgres_notebook.ipynb	528:554 (4%) 952:978 (2%)	view
27	x 2	notebooks/mssql2postgresql notebooks/mysql2spanner	mssql-to-postgres-notebook.ipynb MySqlToSpanner_notebook.ipynb	893:919 (2%) 877:903 (2%)	view
27	x 2	notebooks/oracle2bq notebooks/oracle2postgres	OracleToBigQuery_notebook.ipynb OracleToPostgres_notebook.ipynb	855:881 (3%) 952:978 (2%)	view
27	x 2	notebooks/mysql2spanner notebooks/oracle2spanner	MySqlToSpanner_notebook.ipynb OracleToSpanner_notebook.ipynb	877:903 (2%) 868:894 (3%)	view
27	x 2	notebooks/generic_notebook notebooks/mssql2postgresql	vertex_pipeline_pyspark.ipynb mssql-to-postgres-notebook.ipynb	528:554 (4%) 893:919 (2%)	view
27	x 2	notebooks/generic_notebook notebooks/mysql2spanner	vertex_pipeline_pyspark.ipynb MySqlToSpanner_notebook.ipynb	528:554 (4%) 877:903 (2%)	view
26	x 2	notebooks/oracle2bq notebooks/oracle2postgres	OracleToBigQuery_notebook.ipynb OracleToPostgres_notebook.ipynb	301:326 (2%) 317:342 (2%)	view
25	x 2	notebooks/oracle2bq notebooks/oracle2spanner	OracleToBigQuery_notebook.ipynb OracleToSpanner_notebook.ipynb	1:25 (2%) 1:25 (2%)	view
25	x 2	notebooks/mysql2spanner notebooks/oracle2spanner	MySqlToSpanner_notebook.ipynb OracleToSpanner_notebook.ipynb	1:25 (2%) 1:25 (2%)	view
25	x 2	notebooks/oracle2bq notebooks/oracle2spanner	OracleToBigQuery_notebook.ipynb OracleToSpanner_notebook.ipynb	398:422 (2%) 399:423 (2%)	view
25	x 2	notebooks/oracle2bq notebooks/oracle2postgres	OracleToBigQuery_notebook.ipynb OracleToPostgres_notebook.ipynb	398:422 (2%) 475:499 (2%)	view
25	x 2	notebooks/oracle2bq notebooks/oracle2postgres	OracleToBigQuery_notebook.ipynb OracleToPostgres_notebook.ipynb	660:684 (2%) 762:786 (2%)	view
24	x 2	notebooks/oracle2bq notebooks/oracle2spanner	OracleToBigQuery_notebook.ipynb OracleToSpanner_notebook.ipynb	535:558 (2%) 558:581 (2%)	view
24	x 2	notebooks/oracle2bq notebooks/oracle2spanner	OracleToBigQuery_notebook.ipynb OracleToSpanner_notebook.ipynb	790:813 (2%) 808:831 (2%)	view
24	x 2	notebooks/generic_notebook notebooks/postgresql2bq	vertex_pipeline_pyspark.ipynb postgresql-to-bigquery-notebook.ipynb	7:30 (4%) 7:30 (2%)	view
24	x 2	notebooks/oracle2bq notebooks/oracle2postgres	OracleToBigQuery_notebook.ipynb OracleToPostgres_notebook.ipynb	790:813 (2%) 897:920 (2%)	view
24	x 2	notebooks/oracle2postgres notebooks/oracle2spanner	OracleToPostgres_notebook.ipynb OracleToSpanner_notebook.ipynb	897:920 (2%) 808:831 (2%)	view
24	x 2	notebooks/mysql2spanner notebooks/oracle2bq	MySqlToSpanner_notebook.ipynb OracleToBigQuery_notebook.ipynb	203:226 (2%) 207:230 (2%)	view
23	x 2	notebooks/mssql2bq notebooks/postgresql2bq	mssql-to-bigquery-notebook.ipynb postgresql-to-bigquery-notebook.ipynb	59:81 (2%) 71:93 (2%)	view

Duplicated Units

The list of top 4 duplicated units.

See data for all 4 unit duplicates...

Size	#	Folders	Files	Lines	Code
30	x 2	java/src/main/java/com/g...dataproc/templates/jdbc java/src/main/java/com/g.../dataproc/templates/gcs	JDBCToSpanner.java GCSToSpanner.java	76:109 73:107	view
16	x 2	java/src/main/java/com/g...dataproc/templates/jdbc java/src/main/java/com/g...dataproc/templates/jdbc	JDBCToGCSConfig.java JDBCToSpannerConfig.java	197:215 237:255	view
7	x 2	airflow/dags airflow/dags	submit_pyspark_dataproc_template.py submit_spark_dataproc_template.py	0:0 0:0	view
6	x 2	java/src/main/java/com/g...roc/templates/databases java/src/main/java/com/g...roc/templates/databases	CassandraToGCSConfig.java CassandraToBqConfig.java	74:81 70:77	view