aws-samples / amazon-sagemaker-script-mode
Duplication

Places in code with 6 or more lines that are exactly the same.

Intro
  • For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
  • Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
  • You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.
Learn more...
Duplication Overall
  • 61% duplication:
    • 1,662 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
    • 1,015 duplicated lines
  • 53 duplicates
system61% (1,015 lines)
Duplication per Extension
py61% (1,015 lines)
Duplication per Component (primary)
tf-distribution-options92% (352 lines)
tf-batch-inference-script100% (294 lines)
tf-horovod-inference-pipeline55% (188 lines)
tf-2-workflow97% (46 lines)
tf-2-workflow-smpipelines95% (45 lines)
tf-eager-script-mode76% (38 lines)
tf-sentiment-script-mode29% (20 lines)
deploy-pretrained-model34% (14 lines)
tf-2-data-parallelism5% (11 lines)
tf-2-word-embeddings10% (7 lines)
hugging-face-lambda-step0% (0 lines)

Duplication Between Components (50+ lines)

G tf-batch-inference-script tf-batch-inference-script tf-distribution-options tf-distribution-options tf-batch-inference-script--tf-distribution-options 643 tf-horovod-inference-pipeline tf-horovod-inference-pipeline tf-batch-inference-script--tf-horovod-inference-pipeline 376 tf-distribution-options--tf-horovod-inference-pipeline 387 tf-2-workflow-smpipelines tf-2-workflow-smpipelines tf-2-workflow tf-2-workflow tf-2-workflow-smpipelines--tf-2-workflow 90 tf-eager-script-mode tf-eager-script-mode tf-2-workflow-smpipelines--tf-eager-script-mode 74 tf-2-workflow--tf-eager-script-mode 76

Download: SVG DOT (open online Graphviz editor)

Open 3D force graph...

Show more details on duplication between components...
Longest Duplicates
The list of 20 longest duplicates.
See data for all 53 duplicates...
Size#FoldersFilesLinesCode
98 x 2 tf-batch-inference-script/code
tf-distribution-options/code
train.py
train_hvd.py
 40:162 (88%)
40:166 (88%)
 view
69 x 2 tf-batch-inference-script
tf-distribution-options
generate_cifar10_tfrecords.py
generate_cifar10_tfrecords.py
 35:127 (100%)
35:127 (100%)
 view
69 x 2 tf-distribution-options
tf-horovod-inference-pipeline
generate_cifar10_tfrecords.py
generate_cifar10_tfrecords.py
 35:127 (100%)
35:127 (100%)
 view
69 x 2 tf-batch-inference-script
tf-horovod-inference-pipeline
generate_cifar10_tfrecords.py
generate_cifar10_tfrecords.py
 35:127 (100%)
35:127 (100%)
 view
50 x 2 tf-batch-inference-script/code
tf-distribution-options/code
model_def.py
model_def.py
 6:65 (100%)
6:65 (100%)
 view
45 x 2 tf-batch-inference-script/code
tf-distribution-options/code
utilities.py
utilities.py
 8:84 (100%)
8:84 (100%)
 view
31 x 2 tf-2-workflow-smpipelines/train_model
tf-2-workflow/train_model
train.py
train.py
 27:74 (75%)
27:74 (75%)
 view
29 x 2 tf-distribution-options/code
tf-horovod-inference-pipeline
model_def.py
train.py
 15:46 (58%)
43:74 (11%)
 view
29 x 2 tf-batch-inference-script/code
tf-horovod-inference-pipeline
model_def.py
train.py
 15:46 (58%)
43:74 (11%)
 view
23 x 2 tf-batch-inference-script/code
tf-horovod-inference-pipeline
train.py
train.py
 48:78 (20%)
192:224 (9%)
 view
23 x 2 tf-distribution-options/code
tf-horovod-inference-pipeline
train_hvd.py
train.py
 48:81 (20%)
192:224 (9%)
 view
23 x 2 tf-distribution-options/code
tf-distribution-options/code
train_hvd.py
train_ps.py
 17:56 (20%)
17:55 (26%)
 view
22 x 2 tf-2-workflow/train_model
tf-eager-script-mode/train_model
train.py
train.py
 8:50 (53%)
13:55 (50%)
 view
19 x 2 tf-batch-inference-script/code
tf-distribution-options/code
inference.py
inference.py
 20:65 (100%)
20:65 (100%)
 view
19 x 2 tf-distribution-options/code
tf-horovod-inference-pipeline
utilities.py
train.py
 19:50 (42%)
158:189 (7%)
 view
19 x 2 tf-batch-inference-script/code
tf-horovod-inference-pipeline
utilities.py
train.py
 19:50 (42%)
158:189 (7%)
 view
18 x 2 tf-batch-inference-script/code
tf-distribution-options/code
train.py
train_ps.py
 143:162 (16%)
122:141 (20%)
 view
18 x 2 tf-distribution-options/code
tf-distribution-options/code
train_hvd.py
train_ps.py
 147:166 (16%)
122:141 (20%)
 view
15 x 2 tf-batch-inference-script/code
tf-horovod-inference-pipeline
utilities.py
train.py
 58:82 (33%)
129:153 (5%)
 view
15 x 2 tf-distribution-options/code
tf-horovod-inference-pipeline
utilities.py
train.py
 58:82 (33%)
129:153 (5%)
 view
Duplicated Units
The list of top 15 duplicated units.
See data for all 15 unit duplicates...
Size#FoldersFilesLinesCode
45 x 2 tf-distribution-options/code
tf-batch-inference-script/code
model_def.py
model_def.py
 0:0 
0:0 
 view
25 x 2 tf-distribution-options/code
tf-batch-inference-script/code
utilities.py
utilities.py
 0:0 
0:0 
 view
19 x 3 tf-distribution-options
tf-batch-inference-script
tf-horovod-inference-pipeline
generate_cifar10_tfrecords.py
generate_cifar10_tfrecords.py
generate_cifar10_tfrecords.py
 0:0 
0:0 
0:0 
 view
15 x 3 tf-distribution-options
tf-batch-inference-script
tf-horovod-inference-pipeline
generate_cifar10_tfrecords.py
generate_cifar10_tfrecords.py
generate_cifar10_tfrecords.py
 0:0 
0:0 
0:0 
 view
14 x 2 tf-distribution-options/code
tf-batch-inference-script/code
utilities.py
utilities.py
 0:0 
0:0 
 view
10 x 2 tf-sentiment-script-mode
tf-batch-inference-script/code
sentiment.py
train.py
 0:0 
0:0 
 view
10 x 2 tf-distribution-options/code
tf-distribution-options/code
train_ps.py
train_hvd.py
 0:0 
0:0 
 view
11 x 2 tf-eager-script-mode/train_model
tf-2-workflow/train_model
train.py
train.py
 0:0 
0:0 
 view
8 x 4 tf-distribution-options/code
tf-distribution-options/code
tf-batch-inference-script/code
tf-horovod-inference-pipeline
train_ps.py
train_hvd.py
train.py
train.py
 0:0 
0:0 
0:0 
0:0 
 view
14 x 2 tf-distribution-options/code
tf-batch-inference-script/code
inference.py
inference.py
 0:0 
0:0 
 view
6 x 3 tf-distribution-options
tf-batch-inference-script
tf-horovod-inference-pipeline
generate_cifar10_tfrecords.py
generate_cifar10_tfrecords.py
generate_cifar10_tfrecords.py
 0:0 
0:0 
0:0 
 view
12 x 2 tf-distribution-options/code
tf-batch-inference-script/code
inference.py
inference.py
 0:0 
0:0 
 view
6 x 3 tf-distribution-options
tf-batch-inference-script
tf-horovod-inference-pipeline
generate_cifar10_tfrecords.py
generate_cifar10_tfrecords.py
generate_cifar10_tfrecords.py
 0:0 
0:0 
0:0 
 view
8 x 2 deploy-pretrained-model/GPT2/code
deploy-pretrained-model/BERT/code
inference_code.py
inference_code.py
 0:0 
0:0 
 view
7 x 2 tf-distribution-options/code
tf-batch-inference-script/code
utilities.py
utilities.py
 0:0 
0:0 
 view