aws-samples / amazon-sagemaker-examples-jp
Duplication

Places in code with 6 or more lines that are exactly the same.

Intro
  • For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
  • Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
  • You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.
Learn more...
Duplication Overall
  • 40% duplication:
    • 2,441 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
    • 984 duplicated lines
  • 24 duplicates
system40% (984 lines)
Duplication per Extension
py40% (828 lines)
yaml69% (156 lines)
Duplication per Component (primary)
distributed_training84% (504 lines)
sagemaker-experiments100% (162 lines)
hpo_pytorch_mnist72% (140 lines)
workshop67% (102 lines)
edge_inference15% (60 lines)
sagemaker_processing19% (16 lines)
nlp_amazon_review0% (0 lines)
amazon_forecast_tutorial0% (0 lines)
mlops0% (0 lines)
tensorflow2_training_and_serving0% (0 lines)
autogluon0% (0 lines)

Duplication Between Components (50+ lines)

G hpo_pytorch_mnist hpo_pytorch_mnist sagemaker-experiments sagemaker-experiments hpo_pytorch_mnist--sagemaker-experiments 302

Download: SVG DOT (open online Graphviz editor)

Open 3D force graph...

Show more details on duplication between components...
Longest Duplicates
The list of 20 longest duplicates.
See data for all 24 duplicates...
Size#FoldersFilesLinesCode
140 x 2 hpo_pytorch_mnist
sagemaker-experiments/pytorch_mnist/src
mnist.py
mnist_train.py
15:194 (100%)
15:193 (100%)
view
75 x 2 distributed_training
distributed_training
train_pytorch_single_maskrcnn.py
train_pytorch_smdataparallel_maskrcnn.py
127:211 (35%)
144:228 (36%)
view
44 x 2 distributed_training
distributed_training
train_pytorch_single_maskrcnn.py
train_pytorch_smdataparallel_maskrcnn.py
213:258 (20%)
230:275 (21%)
view
40 x 2 workshop/lab_bring-your-own-model/tensorflow
workshop/lab_bring-your-own-model/tensorflow
cnn_mnist_after.py
cnn_mnist_before.py
41:132 (47%)
26:118 (59%)
view
36 x 2 distributed_training
distributed_training
e2e_mask_rcnn_R_50_FPN_1x_64GPU_4bs.yaml
e2e_mask_rcnn_R_50_FPN_1x_8GPU_4bs.yaml
1:36 (80%)
1:36 (80%)
view
36 x 2 distributed_training
distributed_training
e2e_mask_rcnn_R_50_FPN_1x_32GPU_4bs.yaml
e2e_mask_rcnn_R_50_FPN_1x_64GPU_4bs.yaml
1:36 (80%)
1:36 (80%)
view
36 x 2 distributed_training
distributed_training
e2e_mask_rcnn_R_50_FPN_1x_32GPU_4bs.yaml
e2e_mask_rcnn_R_50_FPN_1x_8GPU_4bs.yaml
1:36 (80%)
1:36 (80%)
view
36 x 2 distributed_training
distributed_training
e2e_mask_rcnn_R_50_FPN_1x_16GPU_4bs.yaml
e2e_mask_rcnn_R_50_FPN_1x_32GPU_4bs.yaml
1:36 (80%)
1:36 (80%)
view
36 x 2 distributed_training
distributed_training
e2e_mask_rcnn_R_50_FPN_1x_16GPU_4bs.yaml
e2e_mask_rcnn_R_50_FPN_1x_64GPU_4bs.yaml
1:36 (80%)
1:36 (80%)
view
36 x 2 distributed_training
distributed_training
e2e_mask_rcnn_R_50_FPN_1x_16GPU_4bs.yaml
e2e_mask_rcnn_R_50_FPN_1x_8GPU_4bs.yaml
1:36 (80%)
1:36 (80%)
view
28 x 2 distributed_training
distributed_training
train_pytorch_single_maskrcnn.py
train_pytorch_smdataparallel_maskrcnn.py
75:117 (13%)
85:127 (13%)
view
16 x 2 sagemaker-experiments/pytorch_mnist/src
sagemaker-experiments/pytorch_mnist/src
mnist_deploy.py
mnist_train.py
11:27 (72%)
21:37 (11%)
view
16 x 2 hpo_pytorch_mnist
sagemaker-experiments/pytorch_mnist/src
mnist.py
mnist_deploy.py
21:37 (11%)
11:27 (72%)
view
14 x 2 distributed_training
distributed_training
train_pytorch_single_maskrcnn.py
train_pytorch_smdataparallel_maskrcnn.py
294:313 (6%)
312:331 (6%)
view
11 x 2 distributed_training
distributed_training
train_pytorch_single_maskrcnn.py
train_pytorch_smdataparallel_maskrcnn.py
279:291 (5%)
296:308 (5%)
view
11 x 2 workshop/lab_bring-your-own-model/tensorflow
workshop/lab_bring-your-own-model/tensorflow
cnn_mnist_after.py
cnn_mnist_before.py
163:175 (12%)
132:144 (16%)
view
10 x 2 edge_inference/greengras...mple.IoTPublisher/1.0.0
edge_inference/greengras...mple.IoTPublisher/1.0.1
IoTPublisher.py
IoTPublisher.py
27:41 (27%)
28:41 (25%)
view
8 x 2 edge_inference/greengras...mple.IoTPublisher/1.0.0
edge_inference/greengras...mple.IoTPublisher/1.0.1
IoTPublisher.py
IoTPublisher.py
53:62 (22%)
54:63 (20%)
view
8 x 2 sagemaker_processing/pro...ntainer/preprocess_code
sagemaker_processing/pro...inner/preprocess_script
preprocess.py
preprocess.py
8:15 (21%)
7:14 (17%)
view
8 x 2 distributed_training
distributed_training
train_pytorch_single_maskrcnn.py
train_pytorch_smdataparallel_maskrcnn.py
59:68 (3%)
57:71 (3%)
view
Duplicated Units
The list of top 10 duplicated units.
See data for all 10 unit duplicates...
Size#FoldersFilesLinesCode
56 x 2 hpo_pytorch_mnist
sagemaker-experiments/pytorch_mnist/src
mnist.py
mnist_train.py
0:0 
0:0 
view
77 x 2 workshop/lab_bring-your-own-model/tensorflow
workshop/lab_bring-your-own-model/tensorflow
cnn_mnist_after.py
cnn_mnist_before.py
0:0 
0:0 
view
27 x 2 distributed_training
distributed_training
train_pytorch_single_maskrcnn.py
train_pytorch_smdataparallel_maskrcnn.py
0:0 
0:0 
view
14 x 2 hpo_pytorch_mnist
sagemaker-experiments/pytorch_mnist/src
mnist.py
mnist_train.py
0:0 
0:0 
view
13 x 2 distributed_training
distributed_training
train_pytorch_single_maskrcnn.py
train_pytorch_smdataparallel_maskrcnn.py
0:0 
0:0 
view
7 x 3 hpo_pytorch_mnist
sagemaker-experiments/pytorch_mnist/src
sagemaker-experiments/pytorch_mnist/src
mnist.py
mnist_train.py
mnist_deploy.py
0:0 
0:0 
0:0 
view
7 x 2 hpo_pytorch_mnist
sagemaker-experiments/pytorch_mnist/src
mnist.py
mnist_train.py
0:0 
0:0 
view
7 x 2 sagemaker_processing/pro...ntainer/preprocess_code
sagemaker_processing/pro...inner/preprocess_script
preprocess.py
preprocess.py
0:0 
0:0 
view
6 x 3 hpo_pytorch_mnist
sagemaker-experiments/pytorch_mnist/src
sagemaker-experiments/pytorch_mnist/src
mnist.py
mnist_train.py
mnist_deploy.py
0:0 
0:0 
0:0 
view
6 x 2 hpo_pytorch_mnist
sagemaker-experiments/pytorch_mnist/src
mnist.py
mnist_train.py
0:0 
0:0 
view