microsoft / pai
Duplication

Places in code with 6 or more lines that are exactly the same.

Intro
  • For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
  • Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
  • You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.
Learn more...
Duplication Overall
  • 22% duplication:
    • 67,916 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
    • 14,989 duplicated lines
  • 3,668 duplicates
system22% (14,989 lines)
Duplication per Extension
jsx34% (8,259 lines)
js20% (3,457 lines)
py13% (1,216 lines)
yaml23% (788 lines)
yml25% (493 lines)
java5% (471 lines)
tsx14% (161 lines)
ejs12% (100 lines)
scss6% (28 lines)
lua9% (16 lines)
Duplication per Component (primary)
src/webportal32% (10,095 lines)
src/rest-server15% (1,331 lines)
subprojects/frameworklauncher5% (471 lines)
contrib/kubespray25% (448 lines)
src/database-controller19% (387 lines)
src/k8s-job-exit-spec29% (346 lines)
src/alert-manager17% (266 lines)
deployment/k8sPaiLibrary12% (199 lines)
contrib/api-stress-test20% (164 lines)
contrib/submit-job-v214% (161 lines)
src/grafana61% (141 lines)
deployment/paiLibrary18% (128 lines)
marketplace-v221% (126 lines)
deployment/utility70% (100 lines)
src/job-exporter4% (66 lines)
src/prometheus-pushgateway53% (50 lines)
deployment7% (44 lines)
src/log-manager19% (39 lines)
src/postgresql64% (31 lines)
src/prometheus47% (31 lines)
build/core14% (31 lines)
src/marketplace-db75% (30 lines)
deployment/confStorage6% (30 lines)
contrib/aks-engine7% (28 lines)
src/pylon31% (24 lines)
src/node-exporter68% (24 lines)
src/storage-manager57% (24 lines)
src/internal-storage38% (21 lines)
src/authentication19% (19 lines)
src/marketplace-restserver30% (13 lines)
src/marketplace-webportal36% (13 lines)
src/watchdog1% (13 lines)
src/docker-cache28% (13 lines)
src/frameworkcontroller38% (13 lines)
src/device-plugin11% (12 lines)
src/dshuttle-master66% (12 lines)
src/openpai-runtime18% (12 lines)
src/k8s-dashboard10% (9 lines)
src/fluentd2% (6 lines)
src/dshuttle9% (6 lines)
src/dshuttle-csi27% (6 lines)
src/cluster-configuration26% (6 lines)
build0% (0 lines)
build/model0% (0 lines)
contrib/autoscaler0% (0 lines)
contrib/debug-tools0% (0 lines)
contrib/hived-config-adapter0% (0 lines)
contrib/profiler0% (0 lines)
contrib/remote-dev-tool0% (0 lines)
ROOT0% (0 lines)
deployment/clusterObjectModel0% (0 lines)
deployment/quick-start0% (0 lines)
deployment/sysconf0% (0 lines)
deployment/tools0% (0 lines)
src/base-image0% (0 lines)
src/cluster0% (0 lines)
src/dev-box0% (0 lines)
src/dshuttle-worker0% (0 lines)
src/hivedscheduler0% (0 lines)
src/utilities0% (0 lines)

Duplication Between Components (50+ lines)

G src/alert-manager src/alert-manager src/database-controller src/database-controller src/alert-manager--src/database-controller 122 src/prometheus-pushgateway src/prometheus-pushgateway src/alert-manager--src/prometheus-pushgateway 88 src/rest-server src/rest-server src/alert-manager--src/rest-server 85 src/webportal src/webportal src/alert-manager--src/webportal 76 src/database-controller--src/rest-server 112 src/database-controller--src/webportal 72 src/rest-server--src/webportal 90 contrib/kubespray contrib/kubespray contrib/kubespray--src/rest-server 98 src/prometheus src/prometheus src/prometheus-pushgateway--src/prometheus 60 contrib/api-stress-test contrib/api-stress-test marketplace-v2 marketplace-v2 contrib/api-stress-test--marketplace-v2 50

Download: SVG DOT (open online Graphviz editor)

Open 3D force graph...

Show more details on duplication between components...
Longest Duplicates
The list of 20 longest duplicates.
See data for all 3,668 duplicates...
Size#FoldersFilesLinesCode
139 x 2 src/webportal/src/app/jo...c/job-detail/components
src/webportal/src/app/jo...iew/fabric/task-attempt
task-role-container-list.jsx
task-attempt-list.jsx
266:406 (15%)
252:392 (18%)
view
131 x 2 src/webportal/src/app/job-submission-demo/utils
src/webportal/src/app/job-submission/utils
constants.js
constants.js
3:173 (91%)
18:186 (100%)
view
95 x 2 src/webportal/src/app/jo...c/job-detail/components
src/webportal/src/app/jo...iew/fabric/task-attempt
task-role-container-list.jsx
task-attempt-list.jsx
845:944 (10%)
685:784 (12%)
view
83 x 2 src/webportal/src/app/jo...mo/components/task-role
src/webportal/src/app/jo...on/components/task-role
ports-list.jsx
ports-list.jsx
51:142 (43%)
72:162 (38%)
view
77 x 2 src/webportal/src/app/jo...n-demo/components/tools
src/webportal/src/app/jo...ission/components/tools
ssh-generator.jsx
ssh-generator.jsx
5:91 (51%)
20:106 (46%)
view
68 x 2 src/webportal/src/app/jo...n-demo/components/tools
src/webportal/src/app/jo...ission/components/tools
ssh-generator.jsx
ssh-generator.jsx
93:177 (45%)
108:192 (41%)
view
68 x 2 src/webportal/src/app/jo...c/job-detail/components
src/webportal/src/app/jo...iew/fabric/task-attempt
task-role-container-list.jsx
task-attempt-list.jsx
724:793 (7%)
546:615 (9%)
view
67 x 2 src/webportal/src/app/jo...c/job-detail/components
src/webportal/src/app/jo...iew/fabric/task-attempt
task-role-container-list.jsx
task-attempt-list.jsx
83:164 (7%)
54:135 (9%)
view
65 x 2 src/webportal/src/app/jo...c/job-detail/components
src/webportal/src/app/jo...iew/fabric/task-attempt
task-role-container-list.jsx
task-attempt-list.jsx
198:266 (7%)
164:232 (8%)
view
63 x 2 src/webportal/src/app/job-submission-demo/utils
src/webportal/src/app/job-submission/utils
conn.js
conn.js
44:126 (39%)
44:126 (50%)
view
61 x 2 src/webportal/src/app/job/job-view/fabric/JobList
src/webportal/src/app/user/fabric/userView
Paginator.jsx
Paginator.jsx
121:188 (35%)
78:145 (45%)
view
61 x 2 src/webportal/src/app/jo...mo/components/task-role
src/webportal/src/app/jo...on/components/task-role
ports-list.jsx
ports-list.jsx
143:215 (31%)
165:239 (28%)
view
50 x 2 deployment/utility
deployment/utility
sftp_copy.py
ssh.py
33:90 (84%)
30:88 (87%)
view
46 x 2 src/webportal/src/app/jo...emo/components/controls
src/webportal/src/app/job-submission/components
monaco-text-field.jsx
monaco-text-field.jsx
31:79 (37%)
54:102 (31%)
view
46 x 2 src/webportal/src/app/job-submission-demo/models
src/webportal/src/app/job-submission/models
protocol-schema.js
protocol-schema.js
74:120 (31%)
101:147 (30%)
view
41 x 2 src/webportal/src/app/jo...-view/fabric/job-detail
src/webportal/src/app/jo...iew/fabric/task-attempt
conn.js
conn.js
159:209 (26%)
55:105 (51%)
view
38 x 2 src/webportal/src/app/jo...ssion/components/topbar
src/webportal/src/app/jo...ponents/yamledit-topbar
import-config.jsx
yamledit-import-config.jsx
92:132 (29%)
32:72 (54%)
view
37 x 2 src/webportal/src/app/jo...emo/components/controls
src/webportal/src/app/jo...ion/components/controls
key-value-list.jsx
key-value-list.jsx
23:65 (23%)
46:88 (19%)
view
35 x 2 src/rest-server/deploy
src/rest-server/deploy
legacy_user_migrate.py
user_v2_migrate.py
84:119 (18%)
158:193 (13%)
view
34 x 2 src/webportal/src/app/jo...sion/components/sidebar
src/webportal/src/app/jo...sion/components/sidebar
parameters.jsx
secrets.jsx
1:35 (38%)
1:35 (37%)
view
Duplicated Units
The list of top 20 duplicated units.
See data for all 33 unit duplicates...
Size#FoldersFilesLinesCode
41 x 2 deployment/utility
deployment/utility
sftp_copy.py
ssh.py
0:0 
0:0 
view
38 x 2 src/webportal/src/app/job-submission-demo/utils
src/webportal/src/app/job-submission/utils
conn.js
conn.js
89:127 
89:127 
view
24 x 2 src/rest-server/deploy
src/rest-server/deploy
legacy_user_migrate.py
user_v2_migrate.py
0:0 
0:0 
view
24 x 2 src/webportal/src/app/jo...-view/fabric/job-detail
src/webportal/src/app/jo...iew/fabric/task-attempt
conn.js
conn.js
183:210 
79:106 
view
22 x 2 src/webportal/src/app/jo...-view/fabric/job-detail
src/webportal/src/app/jo...iew/fabric/task-attempt
conn.js
conn.js
159:181 
55:77 
view
20 x 2 src/job-exporter/src
src/job-exporter/src
docker_stats.py
nvidia.py
0:0 
0:0 
view
20 x 2 src/rest-server/deploy
src/rest-server/deploy
legacy_user_migrate.py
user_v2_migrate.py
0:0 
0:0 
view
20 x 2 src/webportal/src/app/job-submission-demo/utils
src/webportal/src/app/job-submission/utils
utils.js
utils.js
5:29 
26:50 
view
17 x 2 src/webportal/src/app/job-submission-demo/utils
src/webportal/src/app/job-submission/utils
conn.js
conn.js
58:75 
58:75 
view
17 x 2 src/webportal/src/app/job-submission-demo/utils
src/webportal/src/app/job-submission/utils
conn.js
conn.js
181:198 
155:172 
view
15 x 2 subprojects/frameworklau...ameworklauncher/service
subprojects/frameworklau...eworklauncher/webserver
Service.java
WebServer.java
104:123 
58:77 
view
14 x 2 deployment/paiLibrary/paiService
deployment/paiLibrary/paiService
service_management_delete.py
service_management_stop.py
0:0 
0:0 
view
13 x 2 src/rest-server/deploy
src/rest-server/deploy
legacy_user_migrate.py
user_v2_migrate.py
0:0 
0:0 
view
11 x 3 src/alert-manager/src/cert-expiration-checker
src/alert-manager/src/cluster-utilization
src/prometheus-pushgateway/src/metrics-cleaner
send_alert.py
send_alert.py
main.py
0:0 
0:0 
0:0 
view
11 x 2 src/webportal/src/app/job-submission-demo/utils
src/webportal/src/app/job-submission/utils
utils.js
utils.js
49:60 
222:233 
view
10 x 2 src/webportal/src/app/cluster-view/hardware
src/webportal/src/app/cluster-view/hardware
hardware.component.js
hardware.component.js
113:123 
214:224 
view
10 x 2 src/webportal/src/app/job-submission-demo/models
src/webportal/src/app/job-submission/models
container-size.js
container-size.js
9:19 
9:19 
view
11 x 2 build/core
contrib/aks-engine
build_utility.py
azure.py
0:0 
0:0 
view
11 x 4 contrib/kubespray
contrib/kubespray
src/rest-server/deploy
src/rest-server/deploy
namespace_secret_backup.py
namespace_secret_recover.py
legacy_user_migrate.py
user_v2_migrate.py
0:0 
0:0 
0:0 
0:0 
view
8 x 2 src/webportal/src/app/job-submission-demo/utils
src/webportal/src/app/job-submission/utils
ssh-keygen.js
ssh-keygen.js
6:15 
21:30 
view