GoogleCloudPlatform / ai-ml-recipes

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

42% duplication:

7,612 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
3,239 duplicated lines

6,196 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Duplication Between Components (50+ lines)

Download: SVG DOT (open online Graphviz editor)

From Component --> To Component	Duplicated Lines	File Pairs	Details
notebooks (2%) --> public_datasets (42%)	372	13 file pairs	details...

Open 2D force graph... Open 3D force graph...

Show more details on duplication between components...

Longest Duplicates

The list of 50 longest duplicates.

See data for all 6,196 duplicates...

Size	#	Folders	Files	Lines	Code
119	x 2	notebooks/generative_ai/content_generation notebooks/generative_ai/content_generation	description_from_video.ipynb product_attributes_from_image.ipynb	121:239 (17%) 119:237 (14%)	view
79	x 2	notebooks/classification..._support_vector_machine notebooks/classification..._support_vector_machine	predictive_maintenance.ipynb predictive_maintenance.ipynb	280:358 (14%) 436:514 (14%)	view
39	x 2	notebooks/classification..._support_vector_machine notebooks/classification...r_perceptron_classifier	predictive_maintenance.ipynb sms_spam_filtering.ipynb	48:86 (7%) 49:87 (10%)	view
37	x 2	notebooks/generative_ai/content_generation notebooks/generative_ai/content_generation	description_from_video.ipynb product_attributes_from_image.ipynb	1:37 (5%) 1:37 (4%)	view
35	x 2	notebooks/classification..._support_vector_machine notebooks/classification...r_perceptron_classifier	predictive_maintenance.ipynb sms_spam_filtering.ipynb	512:546 (6%) 355:389 (8%)	view
31	x 2	notebooks/generative_ai/content_generation notebooks/generative_ai/content_generation	description_from_video.ipynb product_attributes_from_image.ipynb	651:681 (4%) 775:805 (3%)	view
27	x 2	notebooks/classification..._support_vector_machine notebooks/classification...r_perceptron_classifier	predictive_maintenance.ipynb sms_spam_filtering.ipynb	481:507 (4%) 324:350 (6%)	view
27	x 2	notebooks/generative_ai/content_generation notebooks/generative_ai/content_generation	description_from_video.ipynb product_attributes_from_image.ipynb	341:367 (3%) 406:432 (3%)	view
27	x 2	notebooks/generative_ai/finetuning notebooks/quickstart/iceberg	translation_finetuning.ipynb iceberg_quickstart.ipynb	1:27 (3%) 1:27 (7%)	view
27	x 2	notebooks/generative_ai/content_generation notebooks/generative_ai/content_generation	description_from_video.ipynb product_attributes_from_image.ipynb	241:267 (3%) 239:265 (3%)	view
27	x 2	notebooks/classification..._support_vector_machine notebooks/classification...r_perceptron_classifier	predictive_maintenance.ipynb sms_spam_filtering.ipynb	325:351 (4%) 324:350 (6%)	view
25	x 2	notebooks/classification/logistic_regression notebooks/sampling/monte_carlo	wine_quality_classification_mlr.ipynb customer_price_index.ipynb	841:865 (2%) 621:645 (3%)	view
25	x 2	notebooks/generative_ai/content_generation notebooks/regression/decision_tree_regression	description_from_video.ipynb housing_prices_prediction.ipynb	6:30 (3%) 7:31 (4%)	view
25	x 2	notebooks/classification/logistic_regression notebooks/regression/decision_tree_regression	wine_quality_classification_mlr.ipynb housing_prices_prediction.ipynb	841:865 (2%) 599:623 (4%)	view
25	x 2	notebooks/regression/decision_tree_regression notebooks/regression/random_forest_regression	housing_prices_prediction.ipynb bike_trip_duration_prediction.ipynb	7:31 (4%) 7:31 (5%)	view
25	x 2	notebooks/generative_ai/content_generation notebooks/regression/decision_tree_regression	product_attributes_from_image.ipynb housing_prices_prediction.ipynb	6:30 (3%) 7:31 (4%)	view
25	x 2	notebooks/regression/random_forest_regression notebooks/sampling/monte_carlo	bike_trip_duration_prediction.ipynb customer_price_index.ipynb	476:500 (5%) 621:645 (3%)	view
25	x 2	notebooks/quickstart/dataproc_metastore notebooks/regression/decision_tree_regression	metastore_spark_quickstart.ipynb housing_prices_prediction.ipynb	7:31 (10%) 7:31 (4%)	view
25	x 2	notebooks/generative_ai/content_generation notebooks/quickstart/dataproc_metastore	description_from_video.ipynb metastore_spark_quickstart.ipynb	6:30 (3%) 7:31 (10%)	view
25	x 2	notebooks/generative_ai/content_generation notebooks/generative_ai/content_generation	description_from_video.ipynb product_attributes_from_image.ipynb	95:119 (3%) 91:115 (3%)	view
25	x 2	notebooks/generative_ai/content_generation notebooks/regression/random_forest_regression	product_attributes_from_image.ipynb bike_trip_duration_prediction.ipynb	6:30 (3%) 7:31 (5%)	view
25	x 2	notebooks/regression/decision_tree_regression notebooks/regression/random_forest_regression	housing_prices_prediction.ipynb bike_trip_duration_prediction.ipynb	599:623 (4%) 476:500 (5%)	view
25	x 2	notebooks/quickstart/dataproc_metastore notebooks/regression/random_forest_regression	metastore_spark_quickstart.ipynb bike_trip_duration_prediction.ipynb	7:31 (10%) 7:31 (5%)	view
25	x 2	notebooks/classification/logistic_regression notebooks/regression/random_forest_regression	wine_quality_classification_mlr.ipynb bike_trip_duration_prediction.ipynb	841:865 (2%) 476:500 (5%)	view
25	x 2	notebooks/regression/decision_tree_regression notebooks/sampling/monte_carlo	housing_prices_prediction.ipynb customer_price_index.ipynb	599:623 (4%) 621:645 (3%)	view
25	x 2	notebooks/generative_ai/content_generation notebooks/quickstart/dataproc_metastore	product_attributes_from_image.ipynb metastore_spark_quickstart.ipynb	6:30 (3%) 7:31 (10%)	view
25	x 2	notebooks/generative_ai/content_generation notebooks/regression/random_forest_regression	description_from_video.ipynb bike_trip_duration_prediction.ipynb	6:30 (3%) 7:31 (5%)	view
24	x 2	notebooks/generative_ai/content_generation notebooks/generative_ai/content_generation	description_from_video.ipynb product_attributes_from_image.ipynb	532:555 (3%) 531:554 (2%)	view
23	x 2	notebooks/classification..._support_vector_machine notebooks/classification/logistic_regression	predictive_maintenance.ipynb wine_quality_classification_mlr.ipynb	522:544 (4%) 841:863 (2%)	view
23	x 2	notebooks/classification..._support_vector_machine notebooks/sampling/monte_carlo	predictive_maintenance.ipynb customer_price_index.ipynb	522:544 (4%) 621:643 (3%)	view
23	x 2	notebooks/classification..._support_vector_machine notebooks/regression/decision_tree_regression	predictive_maintenance.ipynb housing_prices_prediction.ipynb	522:544 (4%) 599:621 (3%)	view
23	x 2	notebooks/classification...r_perceptron_classifier notebooks/regression/decision_tree_regression	sms_spam_filtering.ipynb housing_prices_prediction.ipynb	365:387 (5%) 599:621 (3%)	view
23	x 2	notebooks/classification...r_perceptron_classifier notebooks/sampling/monte_carlo	sms_spam_filtering.ipynb customer_price_index.ipynb	365:387 (5%) 621:643 (3%)	view
23	x 2	notebooks/classification..._support_vector_machine notebooks/regression/random_forest_regression	predictive_maintenance.ipynb bike_trip_duration_prediction.ipynb	522:544 (4%) 476:498 (4%)	view
23	x 2	notebooks/classification...r_perceptron_classifier notebooks/regression/random_forest_regression	sms_spam_filtering.ipynb bike_trip_duration_prediction.ipynb	365:387 (5%) 476:498 (4%)	view
23	x 2	notebooks/classification/logistic_regression notebooks/classification...r_perceptron_classifier	wine_quality_classification_mlr.ipynb sms_spam_filtering.ipynb	841:863 (2%) 365:387 (5%)	view
22	x 2	notebooks/generative_ai/content_generation notebooks/generative_ai/content_generation	description_from_video.ipynb product_attributes_from_image.ipynb	419:440 (3%) 533:554 (2%)	view
22	x 2	notebooks/generative_ai/content_generation notebooks/generative_ai/content_generation	description_from_video.ipynb description_from_video.ipynb	419:440 (3%) 534:555 (3%)	view
21	x 2	notebooks/quickstart/dataproc_metastore notebooks/sampling/monte_carlo	metastore_spark_quickstart.ipynb customer_price_index.ipynb	11:31 (8%) 14:34 (3%)	view
21	x 2	notebooks/generative_ai/content_generation notebooks/sampling/monte_carlo	description_from_video.ipynb customer_price_index.ipynb	10:30 (3%) 14:34 (3%)	view
21	x 2	notebooks/regression/random_forest_regression notebooks/sampling/monte_carlo	bike_trip_duration_prediction.ipynb customer_price_index.ipynb	11:31 (4%) 14:34 (3%)	view
21	x 2	notebooks/regression/decision_tree_regression notebooks/sampling/monte_carlo	housing_prices_prediction.ipynb customer_price_index.ipynb	11:31 (3%) 14:34 (3%)	view
21	x 2	notebooks/generative_ai/content_generation notebooks/sampling/monte_carlo	product_attributes_from_image.ipynb customer_price_index.ipynb	10:30 (2%) 14:34 (3%)	view
20	x 2	notebooks/generative_ai/sentiment_analysis notebooks/quickstart/delta_format	sentiment_analysis_movie_reviews.ipynb delta_quickstart.ipynb	7:26 (3%) 7:26 (10%)	view
19	x 2	notebooks/generative_ai/content_generation notebooks/generative_ai/sentiment_analysis	product_attributes_from_image.ipynb sentiment_analysis_movie_reviews.ipynb	12:30 (2%) 8:26 (3%)	view
19	x 2	notebooks/generative_ai/content_generation notebooks/quickstart/delta_format	product_attributes_from_image.ipynb delta_quickstart.ipynb	12:30 (2%) 8:26 (9%)	view
19	x 2	notebooks/generative_ai/content_generation notebooks/generative_ai/sentiment_analysis	description_from_video.ipynb sentiment_analysis_movie_reviews.ipynb	12:30 (2%) 8:26 (3%)	view
19	x 2	notebooks/generative_ai/finetuning notebooks/quickstart/iceberg	translation_finetuning.ipynb iceberg_quickstart.ipynb	716:734 (2%) 347:365 (5%)	view
19	x 2	notebooks/quickstart/delta_format notebooks/regression/decision_tree_regression	delta_quickstart.ipynb housing_prices_prediction.ipynb	8:26 (9%) 13:31 (3%)	view
19	x 2	notebooks/generative_ai/content_generation notebooks/quickstart/delta_format	description_from_video.ipynb delta_quickstart.ipynb	12:30 (2%) 8:26 (9%)	view