anthropics / s5cmd Duplication

anthropics / s5cmd

Duplication

Places in code with 6 or more lines that are exactly the same.

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

To learn more about duplications and techniques for eliminating duplication, Sokrates recommends the following resources:

Avoid Repetition, MartinFlower.com (IEEE Software article)
Refactoring duplicated code, sourcemaking.com
Beck Design Rules, MartinFowler.com
DRY (Don't Repeat Yourself) Principle, Wikipedia
The Cut-and-Paste Programming Software Development Anti-Pattern, sourcemaking.com
Code Smells / Duplicated Code, Jeff Atwood, codinghorror.com:

"Duplicated code is the bane of software development. Stamp out duplication whenever possible. You should always be on the lookout for more subtle cases of near-duplication, too. Don't Repeat Yourself!"

Reducing Coupling, MartinFlower.com (IEEE Software article):

"There are several ways to describe coupling, but it boils down to this: If changing one module in a program requires changing another module, then coupling exists. ... Duplication always implies coupling, because changing one piece of duplicate code implies changing the other."

Duplication Overall

21% duplication:

6,807 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
1,441 duplicated lines

145 duplicates

dependency graphs: 2D graph | 2D graph (with duplicates)... | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Duplication Between Components (50+ lines)

flowchart TB
    n0["bench"]
    n1["benchmark"]
    n0 ---|"506"| n1
    classDef default fill:#00b2ee,stroke:#ffffff,color:#000000;
    classDef highlighted fill:#00b2ee,stroke:#ffffff,color:#000000;
    linkStyle 0 stroke:#DC143CFF,stroke-width:10px

Download: Mermaid (.mmd) (open online Mermaid editor)

2D force graph | 3D force graph

Show more details on duplication between components...

From Component --> To Component	Duplicated Lines	File Pairs	Details
bench (57%) --> benchmark (54%)	506	1 file pair	details...

Longest Duplicates

The list of 50 longest duplicates.

See data for all 145 duplicates...

Size	#	Folders	Files	Lines	Code
70	x 2	bench benchmark	ant_bench.py bench.py	198:275 (16%) 235:319 (15%)	view
47	x 2	bench benchmark	ant_bench.py bench.py	95:141 (10%) 84:130 (10%)	view
43	x 2	storage storage	s3.go s3.go	319:386 (4%) 411:478 (4%)	view
37	x 2	storage storage	s3.go s3.go	506:563 (4%) 822:879 (4%)	view
18	x 2	command command	cp.go pipe.go	124:141 (2%) 42:59 (7%)	view
17	x 2	command command	cp.go pipe.go	191:207 (2%) 85:101 (7%)	view
16	x 2	bench benchmark	ant_bench.py bench.py	432:447 (3%) 472:487 (3%)	view
16	x 2	bench benchmark	ant_bench.py bench.py	383:403 (3%) 415:435 (3%)	view
15	x 2	command command	cp.go pipe.go	143:157 (1%) 61:75 (6%)	view
14	x 2	bench benchmark	ant_bench.py bench.py	413:429 (3%) 445:461 (3%)	view
13	x 2	command command	cat.go cp.go	46:58 (12%) 127:139 (1%)	view
13	x 2	command command	cat.go pipe.go	46:58 (12%) 45:57 (5%)	view
12	x 2	bench benchmark	ant_bench.py bench.py	316:330 (2%) 351:365 (2%)	view
12	x 2	bench benchmark	ant_bench.py bench.py	161:174 (2%) 132:145 (2%)	view
12	x 2	storage storage	s3.go s3.go	189:207 (1%) 411:429 (1%)	view
12	x 2	command command	cp.go pipe.go	701:714 (1%) 232:245 (5%)	view
12	x 2	bench benchmark	ant_bench.py bench.py	473:485 (2%) 539:551 (2%)	view
12	x 2	command command	cat.go stat.go	134:155 (11%) 227:249 (7%)	view
12	x 2	storage storage	s3.go s3.go	189:207 (1%) 319:337 (1%)	view
11	x 2	command command	ls.go stat.go	69:79 (4%) 49:59 (6%)	view
11	x 2	command command	presign.go stat.go	113:133 (12%) 228:249 (6%)	view
11	x 2	command command	cat.go presign.go	135:155 (10%) 113:133 (12%)	view
10	x 2	command command	cp.go rm.go	183:192 (1%) 68:77 (5%)	view
10	x 2	storage storage	s3.go s3.go	301:316 (1%) 393:408 (1%)	view
10	x 2	storage storage	s3.go s3.go	280:296 (1%) 462:478 (1%)	view
10	x 2	storage storage	s3.go s3.go	280:296 (1%) 370:386 (1%)	view
10	x 2	command command	du.go rm.go	72:81 (5%) 76:85 (5%)	view
9	x 2	bench benchmark	ant_bench.py bench.py	53:61 (2%) 41:49 (1%)	view
9	x 2	command command	cp.go sync.go	434:444 (1%) 214:224 (2%)	view
9	x 2	command command	select.go select.go	152:160 (2%) 206:214 (2%)	view
9	x 2	command command	du.go ls.go	68:76 (5%) 84:92 (4%)	view
8	x 2	bench benchmark	ant_bench.py bench.py	103:110 (1%) 106:113 (1%)	view
8	x 2	command command	mb.go rb.go	38:46 (11%) 34:42 (14%)	view
8	x 2	command command	presign.go stat.go	51:60 (9%) 71:80 (5%)	view
8	x 2	bench benchmark	ant_bench.py bench.py	117:124 (1%) 92:99 (1%)	view
8	x 2	command command	mb.go rb.go	81:90 (11%) 69:78 (14%)	view
8	x 2	benchmark benchmark	bench.py bench.py	92:99 (1%) 106:113 (1%)	view
8	x 2	bench benchmark	ant_bench.py bench.py	43:50 (1%) 35:42 (1%)	view
8	x 2	command command	cp.go cp.go	540:547 (1%) 588:595 (1%)	view
8	x 2	command command	cat.go stat.go	63:72 (7%) 71:80 (5%)	view
8	x 2	command command	cp.go sync.go	260:268 (1%) 100:109 (2%)	view
8	x 2	command command	cp.go cp.go	549:558 (1%) 574:583 (1%)	view
8	x 2	command command	cp.go cp.go	549:558 (1%) 597:606 (1%)	view
8	x 2	command command	cat.go presign.go	63:72 (7%) 51:60 (9%)	view
8	x 2	command command	cp.go cp.go	574:583 (1%) 597:606 (1%)	view
8	x 2	bench bench	ant_bench.py ant_bench.py	103:110 (1%) 117:124 (1%)	view
8	x 2	command command	cp.go cp.go	695:704 (1%) 760:769 (1%)	view
7	x 2	command command	du.go select.go	72:78 (4%) 123:129 (2%)	view
7	x 2	bench benchmark	ant_bench.py bench.py	519:529 (1%) 569:580 (1%)	view
7	x 2	storage storage	s3.go s3.go	221:231 (<1%) 439:449 (<1%)	view

Most Frequent Duplicates

The list of 50 most frequently found duplicates.

Size	#	Folders	Files	Lines	Code
6	x 11	command command command command command command command command command command ...	presign.go du.go pipe.go stat.go cp.go mb.go ls.go rm.go rb.go sync.go ...	51:57 (7%) 84:90 (3%) 120:126 (2%) 71:77 (3%) 262:268 (<1%) 40:46 (8%) 100:106 (2%) 89:95 (3%) 36:42 (10%) 103:109 (1%) ...	view
6	x 4	bench benchmark benchmark bench	ant_bench.py bench.py bench.py ant_bench.py	105:110 (1%) 94:99 (1%) 108:113 (1%) 119:124 (1%)	view
6	x 4	storage storage storage storage	s3.go s3.go s3.go s3.go	222:231 (<1%) 260:269 (<1%) 348:357 (<1%) 440:449 (<1%)	view
6	x 4	bench benchmark benchmark bench	ant_bench.py bench.py bench.py ant_bench.py	104:109 (1%) 93:98 (1%) 107:112 (1%) 118:123 (1%)	view
6	x 4	bench benchmark benchmark bench	ant_bench.py bench.py bench.py ant_bench.py	103:108 (1%) 92:97 (1%) 106:111 (1%) 117:122 (1%)	view
6	x 4	storage storage storage storage	s3.go s3.go s3.go s3.go	221:228 (<1%) 259:266 (<1%) 347:354 (<1%) 439:446 (<1%)	view
6	x 3	command command command	cp.go cat.go pipe.go	127:132 (<1%) 46:51 (5%) 45:50 (2%)	view
6	x 3	command command command	stat.go cat.go presign.go	233:245 (3%) 140:151 (5%) 118:129 (7%)	view
6	x 3	storage storage storage	s3.go s3.go s3.go	197:207 (<1%) 327:337 (<1%) 419:429 (<1%)	view
6	x 3	storage storage storage	s3.go s3.go s3.go	280:288 (<1%) 370:378 (<1%) 462:470 (<1%)	view
6	x 3	command command command	presign.go du.go rm.go	42:47 (7%) 76:81 (3%) 80:85 (3%)	view
6	x 3	command command command	cp.go cp.go cp.go	549:554 (<1%) 574:579 (<1%) 597:602 (<1%)	view
6	x 3	command command command	cp.go cat.go pipe.go	129:134 (<1%) 48:53 (5%) 47:52 (2%)	view
6	x 3	command command command	cp.go cat.go pipe.go	130:135 (<1%) 49:54 (5%) 48:53 (2%)	view
6	x 3	command command command	stat.go du.go ls.go	54:59 (3%) 63:68 (3%) 74:79 (2%)	view
6	x 3	storage storage storage	s3.go s3.go s3.go	196:206 (<1%) 326:336 (<1%) 418:428 (<1%)	view
6	x 3	command command command	cp.go cat.go pipe.go	132:137 (<1%) 51:56 (5%) 50:55 (2%)	view
6	x 3	command command command	stat.go cat.go presign.go	74:80 (3%) 66:72 (5%) 54:60 (7%)	view
6	x 3	command command command	cp.go cp.go cp.go	619:626 (<1%) 681:687 (<1%) 778:784 (<1%)	view
6	x 3	command command command	select.go select.go select.go	185:191 (1%) 217:223 (1%) 232:238 (1%)	view
6	x 3	command command command	stat.go cat.go presign.go	229:240 (3%) 136:147 (5%) 114:125 (7%)	view
6	x 3	command command command	stat.go cat.go presign.go	232:241 (3%) 139:148 (5%) 117:126 (7%)	view
6	x 3	storage storage storage	s3.go s3.go s3.go	189:196 (<1%) 319:326 (<1%) 411:418 (<1%)	view
6	x 3	command command command	du.go select.go ls.go	21:28 (3%) 22:29 (1%) 19:26 (2%)	view
6	x 3	command command command	du.go select.go rm.go	73:78 (3%) 124:129 (1%) 77:82 (3%)	view
6	x 3	command command command	cp.go cat.go pipe.go	134:139 (<1%) 53:58 (5%) 52:57 (2%)	view
6	x 3	command command command	stat.go cat.go presign.go	236:246 (3%) 143:152 (5%) 121:130 (7%)	view
6	x 3	storage storage storage	s3.go s3.go s3.go	195:202 (<1%) 325:332 (<1%) 417:424 (<1%)	view
6	x 3	storage storage storage	s3.go s3.go s3.go	190:197 (<1%) 320:327 (<1%) 412:419 (<1%)	view
6	x 3	storage storage storage	s3.go s3.go s3.go	192:199 (<1%) 322:329 (<1%) 414:421 (<1%)	view
6	x 3	command command command	cp.go cp.go cp.go	551:558 (<1%) 576:583 (<1%) 599:606 (<1%)	view
6	x 3	storage storage storage	s3.go s3.go s3.go	176:185 (<1%) 307:316 (<1%) 399:408 (<1%)	view
6	x 3	command command command	cp.go cat.go pipe.go	133:138 (<1%) 52:57 (5%) 51:56 (2%)	view
6	x 3	storage storage storage	s3.go s3.go s3.go	286:294 (<1%) 376:384 (<1%) 468:476 (<1%)	view
6	x 3	storage storage storage	s3.go s3.go s3.go	283:291 (<1%) 373:381 (<1%) 465:473 (<1%)	view
6	x 3	storage storage storage	s3.go s3.go s3.go	284:292 (<1%) 374:382 (<1%) 466:474 (<1%)	view
6	x 3	storage storage storage	s3.go s3.go s3.go	191:198 (<1%) 321:328 (<1%) 413:420 (<1%)	view
6	x 3	command command command	cp.go cp.go cp.go	550:557 (<1%) 575:582 (<1%) 598:605 (<1%)	view
6	x 3	command command command	du.go select.go rm.go	72:77 (3%) 123:128 (1%) 76:81 (3%)	view
6	x 3	command command command	cp.go cat.go pipe.go	128:133 (<1%) 47:52 (5%) 46:51 (2%)	view
6	x 3	storage storage storage	s3.go s3.go s3.go	175:183 (<1%) 306:314 (<1%) 398:406 (<1%)	view
6	x 3	command command command	sync.go mv.go cp.go	31:38 (1%) 10:17 (14%) 36:43 (<1%)	view
6	x 3	command command command	stat.go cat.go presign.go	72:79 (3%) 64:71 (5%) 52:59 (7%)	view
6	x 3	command command command	du.go select.go ls.go	68:73 (3%) 107:112 (1%) 84:89 (2%)	view
6	x 3	command command command	stat.go cat.go presign.go	228:237 (3%) 135:144 (5%) 113:122 (7%)	view
6	x 3	command command command	cp.go cat.go pipe.go	131:136 (<1%) 50:55 (5%) 49:54 (2%)	view
6	x 3	storage storage storage	s3.go s3.go s3.go	287:296 (<1%) 377:386 (<1%) 469:478 (<1%)	view
6	x 3	command command command	sync.go mv.go cp.go	33:39 (1%) 12:18 (14%) 38:44 (<1%)	view
6	x 3	command command command	stat.go cat.go presign.go	237:249 (3%) 144:155 (5%) 122:133 (7%)	view
6	x 3	command command command	du.go select.go ls.go	23:29 (3%) 24:30 (1%) 21:27 (2%)	view