Places in code with 6 or more lines that are exactly the same.
Intro
- For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
- Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
- You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.
- To learn more about duplications and techniques for eliminating duplication, Sokrates recommends the following resources:
- Avoid Repetition, MartinFlower.com (IEEE Software article)
- Refactoring duplicated code, sourcemaking.com
- Beck Design Rules, MartinFowler.com
- DRY (Don't Repeat Yourself) Principle, Wikipedia
- The Cut-and-Paste Programming Software Development Anti-Pattern, sourcemaking.com
- Code Smells / Duplicated Code, Jeff Atwood, codinghorror.com:
- "Duplicated code is the bane of software development. Stamp out duplication whenever possible. You should always be on the lookout for more subtle cases of near-duplication, too. Don't Repeat Yourself!"
- Reducing Coupling, MartinFlower.com (IEEE Software article):
- "There are several ways to describe coupling, but it boils down to this: If changing one module in a program requires changing another module, then coupling exists. ... Duplication always implies coupling, because changing one piece of duplicate code implies changing the other."
Duplication Overall
- 53% duplication:
- 2,345 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
- 1,248 duplicated lines
- 88 duplicates
Duplication per Extension
Duplication per Component (primary)
Duplication Between Components (50+ lines)
Open 3D force graph...
Show more details on duplication between components...
From Component
--> To Component
Duplicated
Lines
File Pairs
Details
blog (89%)
--> pytorch_script_mode_local_training_and_serving (69%)
232
2 file pairs
details...
tensorflow_script_mode_debug_local_training (88%)
--> tensorflow_script_mode_local_training_and_serving (65%)
170
2 file pairs
details...
tensorflow_script_mode_california_housing_local_training_and_batch_transform (65%)
--> tensorflow_script_mode_california_housing_local_training_and_serving (68%)
142
2 file pairs
details...
tensorflow_bring_your_own_california_housing_local_training_and_batch_transform (48%)
--> tensorflow_script_mode_california_housing_local_training_and_batch_transform (46%)
100
1 file pair
details...
scikit_learn_bring_your_own_container_local_processing (85%)
--> scikit_learn_local_processing (84%)
98
2 file pairs
details...
tensorflow_bring_your_own_california_housing_local_training_and_serving (64%)
--> tensorflow_script_mode_california_housing_local_training_and_serving (38%)
80
1 file pair
details...
pytorch_script_mode_local_model_inference (60%)
--> pytorch_script_mode_local_training_and_serving (22%)
76
2 file pairs
details...
catboost_bring_your_own_container_local_training_and_serving (49%)
--> delta_lake_bring_your_own_container_local_training_and_serving (63%)
70
1 file pair
details...
catboost_bring_your_own_container_local_training_and_serving (46%)
--> scikit_learn_bring_your_own_container_and_own_model_local_serving (46%)
66
1 file pair
details...
delta_lake_bring_your_own_container_local_training_and_serving (60%)
--> scikit_learn_bring_your_own_container_and_own_model_local_serving (46%)
66
1 file pair
details...
catboost_bring_your_own_container_local_training_and_serving (42%)
--> lightgbm_bring_your_own_container_local_training_and_serving (40%)
60
2 file pairs
details...
tensorflow_bring_your_own_california_housing_local_training_and_batch_transform (28%)
--> tensorflow_bring_your_own_california_housing_local_training_and_serving (46%)
58
1 file pair
details...
scikit_learn_local_processing (46%)
--> scikit_learn_nltk_local_processing (42%)
54
2 file pairs
details...
prophet_bring_your_own_container_local_training_and_serving (45%)
--> scikit_learn_bring_your_own_container_and_own_model_local_serving (35%)
50
1 file pair
details...
delta_lake_bring_your_own_container_local_training_and_serving (45%)
--> prophet_bring_your_own_container_local_training_and_serving (45%)
50
1 file pair
details...
catboost_bring_your_own_container_local_training_and_serving (35%)
--> prophet_bring_your_own_container_local_training_and_serving (45%)
50
1 file pair
details...
| From Component --> To Component |
Duplicated Lines |
File Pairs | Details |
|---|---|---|---|
| blog (89%) --> pytorch_script_mode_local_training_and_serving (69%) |
232 | 2 file pairs | details... |
| tensorflow_script_mode_debug_local_training (88%) --> tensorflow_script_mode_local_training_and_serving (65%) |
170 | 2 file pairs | details... |
| tensorflow_script_mode_california_housing_local_training_and_batch_transform (65%) --> tensorflow_script_mode_california_housing_local_training_and_serving (68%) |
142 | 2 file pairs | details... |
| tensorflow_bring_your_own_california_housing_local_training_and_batch_transform (48%) --> tensorflow_script_mode_california_housing_local_training_and_batch_transform (46%) |
100 | 1 file pair | details... |
| scikit_learn_bring_your_own_container_local_processing (85%) --> scikit_learn_local_processing (84%) |
98 | 2 file pairs | details... |
| tensorflow_bring_your_own_california_housing_local_training_and_serving (64%) --> tensorflow_script_mode_california_housing_local_training_and_serving (38%) |
80 | 1 file pair | details... |
| pytorch_script_mode_local_model_inference (60%) --> pytorch_script_mode_local_training_and_serving (22%) |
76 | 2 file pairs | details... |
| catboost_bring_your_own_container_local_training_and_serving (49%) --> delta_lake_bring_your_own_container_local_training_and_serving (63%) |
70 | 1 file pair | details... |
| catboost_bring_your_own_container_local_training_and_serving (46%) --> scikit_learn_bring_your_own_container_and_own_model_local_serving (46%) |
66 | 1 file pair | details... |
| delta_lake_bring_your_own_container_local_training_and_serving (60%) --> scikit_learn_bring_your_own_container_and_own_model_local_serving (46%) |
66 | 1 file pair | details... |
| catboost_bring_your_own_container_local_training_and_serving (42%) --> lightgbm_bring_your_own_container_local_training_and_serving (40%) |
60 | 2 file pairs | details... |
| tensorflow_bring_your_own_california_housing_local_training_and_batch_transform (28%) --> tensorflow_bring_your_own_california_housing_local_training_and_serving (46%) |
58 | 1 file pair | details... |
| scikit_learn_local_processing (46%) --> scikit_learn_nltk_local_processing (42%) |
54 | 2 file pairs | details... |
| prophet_bring_your_own_container_local_training_and_serving (45%) --> scikit_learn_bring_your_own_container_and_own_model_local_serving (35%) |
50 | 1 file pair | details... |
| delta_lake_bring_your_own_container_local_training_and_serving (45%) --> prophet_bring_your_own_container_local_training_and_serving (45%) |
50 | 1 file pair | details... |
| catboost_bring_your_own_container_local_training_and_serving (35%) --> prophet_bring_your_own_container_local_training_and_serving (45%) |
50 | 1 file pair | details... |
Longest Duplicates
The list of 20 longest duplicates.
| Size | # | Folders | Files | Lines | Code |
|---|---|---|---|---|---|
| 99 | x 2 | blog/pytorch_cnn_cifar10/sourcepytorch_script_mode_loca...aining_and_serving/code |  cifar10.pycifar10_pytorch.py |
23:155 (91%)18:150 (94%) | view |
| 36 | x 2 | tensorflow_script_mode_d...cal_training/source_dirtensorflow_script_mode_l...aining_and_serving/code | mnist_tf2.pymnist_tf2.py |
25:81 (97%)21:77 (100%) | view |
| 35 | x 2 | catboost_bring_your_own_...iner/catboost_regressordelta_lake_bring_your_ow...iner/catboost_regressor | predictor.pypredictor.py |
18:83 (100%)18:83 (100%) | view |
| 35 | x 2 | tensorflow_bring_your_ow...ing_and_batch_transformtensorflow_script_mode_c...ing_and_batch_transform | tensorflow_bring_your_own_california_...tensorflow_script_mode_california_hou... |
21:70 (53%)23:72 (56%) | view |
| 33 | x 2 | scikit_learn_bring_your_...tainer_local_processingscikit_learn_local_processing | processing_script.pyprocessing_script.py |
5:50 (100%)5:50 (100%) | view |
| 32 | x 2 | tensorflow_bring_your_ow...al_training_and_servingtensorflow_script_mode_c...al_training_and_serving | tensorflow_bring_your_own_california_...tensorflow_script_mode_california_hou... |
24:69 (52%)25:70 (55%) | view |
| 29 | x 2 | tensorflow_script_mode_c...nd_batch_transform/codetensorflow_script_mode_c...aining_and_serving/code | california_housing_tf2.pycalifornia_housing_tf2.py |
40:83 (63%)38:81 (63%) | view |
| 26 | x 2 | catboost_bring_your_own_...iner/catboost_regressorscikit_learn_bring_your_...er/sklearn_rf_regressor | predictor.pypredictor.py |
33:83 (74%)32:82 (76%) | view |
| 26 | x 2 | delta_lake_bring_your_ow...iner/catboost_regressorscikit_learn_bring_your_...er/sklearn_rf_regressor | predictor.pypredictor.py |
33:83 (74%)32:82 (76%) | view |
| 21 | x 2 | pytorch_script_mode_local_model_inferencepytorch_script_mode_local_training_and_serving | utils_cifar.pyutils_cifar.py |
7:36 (100%)7:36 (100%) | view |
| 18 | x 2 | prophet_bring_your_own_c...rving/container/prophetscikit_learn_bring_your_...er/sklearn_rf_regressor | predictor.pypredictor.py |
54:89 (46%)45:78 (52%) | view |
| 18 | x 2 | delta_lake_bring_your_ow...iner/catboost_regressorprophet_bring_your_own_c...rving/container/prophet | predictor.pypredictor.py |
46:79 (51%)54:89 (46%) | view |
| 18 | x 2 | catboost_bring_your_own_...iner/catboost_regressorprophet_bring_your_own_c...rving/container/prophet | predictor.pypredictor.py |
46:79 (51%)54:89 (46%) | view |
| 17 | x 2 | pytorch_script_mode_local_model_inferencepytorch_script_mode_local_training_and_serving | pytorch_script_mode_local_model_infer...pytorch_script_mode_local_training_an... |
25:47 (40%)23:45 (41%) | view |
| 16 | x 2 | catboost_bring_your_own_...iner/catboost_regressorlightgbm_bring_your_own_...ner/lightgbm_regression | predictor.pypredictor.py |
33:67 (45%)33:70 (47%) | view |
| 16 | x 2 | scikit_learn_bring_your_...tainer_local_processingscikit_learn_local_processing | scikit_learn_bring_your_own_container...SKLearnProcessor_local_processing.py |
28:47 (66%)31:50 (64%) | view |
| 16 | x 2 | delta_lake_bring_your_ow...iner/catboost_regressorlightgbm_bring_your_own_...ner/lightgbm_regression | predictor.pypredictor.py |
33:67 (45%)33:70 (47%) | view |
| 16 | x 2 | lightgbm_bring_your_own_...ner/lightgbm_regressionscikit_learn_bring_your_...er/sklearn_rf_regressor | predictor.pypredictor.py |
33:70 (47%)32:66 (47%) | view |
| 15 | x 2 | tensorflow_script_mode_c...ing_and_batch_transformtensorflow_script_mode_c...al_training_and_serving | tensorflow_script_mode_california_hou...tensorflow_script_mode_california_hou... |
71:87 (24%)78:94 (25%) | view |
| 14 | x 2 | tensorflow_script_mode_debug_local_trainingtensorflow_script_mode_local_training_and_serving | tensorflow_script_mode_debug_local_tr...tensorflow_script_mode_local_training... |
22:37 (23%)26:41 (15%) | view |
Duplicated Units
The list of top 17 duplicated units.
| Size | # | Folders | Files | Lines | Code |
|---|---|---|---|---|---|
| 57 | x 2 | blog/pytorch_cnn_cifar10/sourcepytorch_script_mode_loca...aining_and_serving/code | cifar10.pycifar10_pytorch.py |
0:0 0:0 | view |
| 32 | x 2 | tensorflow_bring_your_ow...ing_and_batch_transformtensorflow_script_mode_c...ing_and_batch_transform | tensorflow_bring_your_own_california_...tensorflow_script_mode_california_hou... |
0:0 0:0 | view |
| 29 | x 2 | scikit_learn_local_processingscikit_learn_bring_your_...tainer_local_processing | processing_script.pyprocessing_script.py |
0:0 0:0 | view |
| 27 | x 2 | tensorflow_bring_your_ow...al_training_and_servingtensorflow_script_mode_c...al_training_and_serving | tensorflow_bring_your_own_california_...tensorflow_script_mode_california_hou... |
0:0 0:0 | view |
| 20 | x 3 | delta_lake_bring_your_ow...iner/catboost_regressorscikit_learn_bring_your_...er/sklearn_rf_regressorcatboost_bring_your_own_...iner/catboost_regressor | predictor.pypredictor.pypredictor.py |
0:0 0:0 0:0 | view |
| 13 | x 2 | tensorflow_script_mode_l...aining_and_serving/codetensorflow_script_mode_d...cal_training/source_dir | mnist_tf2.pymnist_tf2.py |
0:0 0:0 | view |
| 11 | x 2 | tensorflow_script_mode_local_training_and_servingtensorflow_script_mode_debug_local_training | tensorflow_script_mode_local_training...tensorflow_script_mode_debug_local_tr... |
0:0 0:0 | view |
| 10 | x 2 | tensorflow_script_mode_local_training_and_servingtensorflow_script_mode_debug_local_training | tensorflow_script_mode_local_training...tensorflow_script_mode_debug_local_tr... |
0:0 0:0 | view |
| 9 | x 2 | blog/pytorch_cnn_cifar10/sourcepytorch_script_mode_loca...aining_and_serving/code | cifar10.pycifar10_pytorch.py |
0:0 0:0 | view |
| 8 | x 2 | pytorch_script_mode_local_training_and_servingpytorch_script_mode_local_model_inference | pytorch_script_mode_local_training_an...pytorch_script_mode_local_model_infer... |
0:0 0:0 | view |
| 11 | x 2 | tensorflow_script_mode_c...nd_batch_transform/codetensorflow_script_mode_c...aining_and_serving/code | california_housing_tf2.pycalifornia_housing_tf2.py |
0:0 0:0 | view |
| 9 | x 2 | tensorflow_script_mode_l...aining_and_serving/codetensorflow_script_mode_d...cal_training/source_dir | mnist_tf2.pymnist_tf2.py |
0:0 0:0 | view |
| 7 | x 2 | blog/pytorch_cnn_cifar10/sourcepytorch_script_mode_loca...aining_and_serving/code | cifar10.pycifar10_pytorch.py |
0:0 0:0 | view |
| 7 | x 2 | blog/pytorch_cnn_cifar10/sourcepytorch_script_mode_loca...aining_and_serving/code | cifar10.pycifar10_pytorch.py |
0:0 0:0 | view |
| 7 | x 2 | pytorch_script_mode_local_training_and_servingpytorch_script_mode_local_model_inference | pytorch_script_mode_local_training_an...pytorch_script_mode_local_model_infer... |
0:0 0:0 | view |
| 7 | x 2 | huggingface_hebert_senti...ysis_local_serving/codepytorch_nlp_script_mode_local_model_inference/code | inference.pyinference.py |
0:0 0:0 | view |
| 6 | x 4 | delta_lake_bring_your_ow...iner/catboost_regressorlightgbm_bring_your_own_...ner/lightgbm_regressionscikit_learn_bring_your_...er/sklearn_rf_regressorcatboost_bring_your_own_...iner/catboost_regressor | predictor.pypredictor.pypredictor.pypredictor.py |
0:0 0:0 0:0 0:0 | view |