Duplication · guardian/multimedia-cloudworkflow-scripts

Duplication Overall

23% duplication:

2,060 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
487 duplicated lines

32 duplicates

dependency graphs: 2D graph | 2D graph (with duplicates)... | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Duplication Between Components (50+ lines)

flowchart TB
    n0["cloudtrail"]
    n1["fastlylogs"]
    n2["s3_elastic_index"]
    n0 ---|"172"| n1
    n1 ---|"110"| n2
    n0 ---|"84"| n2
    classDef default fill:#00b2ee,stroke:#ffffff,color:#000000;
    classDef highlighted fill:#00b2ee,stroke:#ffffff,color:#000000;
    linkStyle 0 stroke:#DC143CFF,stroke-width:10px
    linkStyle 1 stroke:#DC143CB7,stroke-width:6px
    linkStyle 2 stroke:#DC143C93,stroke-width:4px

Download: Mermaid (.mmd) (open online Mermaid editor)

2D force graph | 3D force graph

Show more details on duplication between components...

From Component --> To Component	Duplicated Lines	File Pairs	Details
cloudtrail (51%) --> fastlylogs (13%)	172	2 file pairs	details...
fastlylogs (8%) --> s3_elastic_index (21%)	110	2 file pairs	details...
cloudtrail (25%) --> s3_elastic_index (16%)	84	1 file pair	details...

Longest Duplicates

The list of 32 longest duplicates.

See data for all 32 duplicates...

Size	#	Folders	Files	Lines	Code
24	x 2	cloudtrail s3_elastic_index	read_cloudtrail.rb reindex_s3.rb	20:50 (14%) 24:54 (9%)	view
24	x 2	fastlylogs fastlylogs	fastly_log_chopper.rb parse_fastly_log.rb	114:137 (6%) 186:209 (13%)	view
23	x 2	fastlylogs s3_elastic_index	fastly_log_chopper.rb reindex_s3.rb	26:54 (5%) 24:52 (8%)	view
23	x 2	cloudtrail fastlylogs	read_cloudtrail.rb fastly_log_chopper.rb	20:48 (13%) 26:54 (5%)	view
20	x 2	cloudtrail fastlylogs	read_cloudtrail.rb fastly_log_chopper.rb	105:130 (11%) 341:366 (5%)	view
17	x 2	ROOT ROOT	asgroup_active_mode.rb asgroup_weekend_mode.rb	41:57 (36%) 36:52 (33%)	view
15	x 2	ROOT ROOT	asgroup_active_mode.rb asgroup_weekend_mode.rb	11:27 (31%) 13:29 (29%)	view
15	x 2	fastlylogs fastlylogs	fastly_log_chopper.rb parse_fastly_log.rb	301:317 (3%) 135:151 (8%)	view
14	x 2	cloudtrail fastlylogs	read_cloudtrail.rb parse_fastly_log.rb	30:44 (8%) 30:44 (8%)	view
14	x 2	cloudtrail fastlylogs	read_cloudtrail.rb parse_fastly_log.rb	47:63 (8%) 47:62 (8%)	view
14	x 2	fastlylogs s3_elastic_index	parse_fastly_log.rb reindex_s3.rb	30:44 (8%) 34:48 (5%)	view
14	x 2	fastlylogs fastlylogs	fastly_log_chopper.rb parse_fastly_log.rb	36:50 (3%) 30:44 (8%)	view
14	x 2	fastlylogs fastlylogs	fastly_log_chopper.rb parse_fastly_log.rb	239:253 (3%) 107:121 (8%)	view
12	x 2	cloudtrail fastlylogs	read_cloudtrail.rb parse_fastly_log.rb	14:27 (7%) 14:27 (6%)	view
12	x 2	fastlylogs fastlylogs	fastly_log_chopper.rb parse_fastly_log.rb	144:158 (3%) 73:88 (6%)	view
11	x 2	cloudtrail s3_elastic_index	read_cloudtrail.rb reindex_s3.rb	52:63 (6%) 56:67 (4%)	view
11	x 2	cloudtrail fastlylogs	read_cloudtrail.rb parse_fastly_log.rb	65:76 (6%) 64:75 (6%)	view
11	x 2	fastlylogs s3_elastic_index	parse_fastly_log.rb reindex_s3.rb	51:62 (6%) 56:67 (4%)	view
11	x 2	fastlylogs fastlylogs	fastly_log_chopper.rb parse_fastly_log.rb	319:332 (2%) 152:165 (6%)	view
10	x 2	ROOT ROOT	interactivepublisher.pl interactivepublisher.pl	81:92 (3%) 156:167 (3%)	view
7	x 2	cloudtrail s3_elastic_index	read_cloudtrail.rb reindex_s3.rb	69:76 (4%) 80:87 (2%)	view
7	x 2	fastlylogs s3_elastic_index	parse_fastly_log.rb reindex_s3.rb	20:27 (4%) 24:31 (2%)	view
7	x 2	fastlylogs s3_elastic_index	parse_fastly_log.rb reindex_s3.rb	68:75 (4%) 80:87 (2%)	view
7	x 2	fastlylogs s3_elastic_index	fastly_log_chopper.rb reindex_s3.rb	66:74 (1%) 60:67 (2%)	view
7	x 2	cloudtrail fastlylogs	read_cloudtrail.rb fastly_log_chopper.rb	56:63 (4%) 66:74 (1%)	view
7	x 2	fastlylogs fastlylogs	fastly_log_chopper.rb parse_fastly_log.rb	26:33 (1%) 20:27 (4%)	view
7	x 2	fastlylogs fastlylogs	fastly_log_chopper.rb parse_fastly_log.rb	66:74 (1%) 55:62 (4%)	view
7	x 2	fastlylogs fastlylogs	fastly_log_chopper.rb parse_fastly_log.rb	163:171 (1%) 88:96 (4%)	view
7	x 2	fastlylogs fastlylogs	fastly_log_chopper.rb parse_fastly_log.rb	291:298 (1%) 133:139 (4%)	view
6	x 2	ROOT ROOT	interactivepublisher.pl interactivepublisher.pl	86:92 (1%) 100:106 (1%)	view
6	x 2	ROOT ROOT	interactivepublisher.pl interactivepublisher.pl	100:106 (1%) 161:167 (1%)	view
6	x 2	fastlylogs fastlylogs	fastly_log_chopper.rb parse_fastly_log.rb	268:274 (1%) 128:133 (3%)	view

Most Frequent Duplicates

The list of 27 most frequently found duplicates.

Size	#	Folders	Files	Lines	Code
6	x 4	cloudtrail fastlylogs s3_elastic_index fastlylogs	read_cloudtrail.rb fastly_log_chopper.rb reindex_s3.rb parse_fastly_log.rb	31:36 (3%) 37:42 (1%) 35:40 (2%) 31:36 (3%)	view
6	x 4	cloudtrail fastlylogs s3_elastic_index fastlylogs	read_cloudtrail.rb fastly_log_chopper.rb reindex_s3.rb parse_fastly_log.rb	33:38 (3%) 39:44 (1%) 37:42 (2%) 33:38 (3%)	view
6	x 4	cloudtrail fastlylogs s3_elastic_index fastlylogs	read_cloudtrail.rb fastly_log_chopper.rb reindex_s3.rb parse_fastly_log.rb	34:39 (3%) 40:45 (1%) 38:43 (2%) 34:39 (3%)	view
6	x 4	cloudtrail fastlylogs s3_elastic_index fastlylogs	read_cloudtrail.rb fastly_log_chopper.rb reindex_s3.rb parse_fastly_log.rb	21:27 (3%) 27:33 (1%) 25:31 (2%) 21:27 (3%)	view
6	x 4	cloudtrail fastlylogs s3_elastic_index fastlylogs	read_cloudtrail.rb fastly_log_chopper.rb reindex_s3.rb parse_fastly_log.rb	57:63 (3%) 68:74 (1%) 61:67 (2%) 56:62 (3%)	view
6	x 4	cloudtrail fastlylogs s3_elastic_index fastlylogs	read_cloudtrail.rb fastly_log_chopper.rb reindex_s3.rb parse_fastly_log.rb	37:43 (3%) 43:49 (1%) 41:47 (2%) 37:43 (3%)	view
6	x 4	cloudtrail fastlylogs s3_elastic_index fastlylogs	read_cloudtrail.rb fastly_log_chopper.rb reindex_s3.rb parse_fastly_log.rb	38:44 (3%) 44:50 (1%) 42:48 (2%) 38:44 (3%)	view
6	x 4	cloudtrail fastlylogs s3_elastic_index fastlylogs	read_cloudtrail.rb fastly_log_chopper.rb reindex_s3.rb parse_fastly_log.rb	32:37 (3%) 38:43 (1%) 36:41 (2%) 32:37 (3%)	view
6	x 4	cloudtrail fastlylogs s3_elastic_index fastlylogs	read_cloudtrail.rb fastly_log_chopper.rb reindex_s3.rb parse_fastly_log.rb	36:42 (3%) 42:48 (1%) 40:46 (2%) 36:42 (3%)	view
6	x 4	cloudtrail fastlylogs s3_elastic_index fastlylogs	read_cloudtrail.rb fastly_log_chopper.rb reindex_s3.rb parse_fastly_log.rb	20:25 (3%) 26:31 (1%) 24:29 (2%) 20:25 (3%)	view
6	x 4	cloudtrail fastlylogs s3_elastic_index fastlylogs	read_cloudtrail.rb fastly_log_chopper.rb reindex_s3.rb parse_fastly_log.rb	30:35 (3%) 36:41 (1%) 34:39 (2%) 30:35 (3%)	view
6	x 4	cloudtrail fastlylogs s3_elastic_index fastlylogs	read_cloudtrail.rb fastly_log_chopper.rb reindex_s3.rb parse_fastly_log.rb	56:62 (3%) 66:73 (1%) 60:66 (2%) 55:61 (3%)	view
6	x 4	cloudtrail fastlylogs s3_elastic_index fastlylogs	read_cloudtrail.rb fastly_log_chopper.rb reindex_s3.rb parse_fastly_log.rb	35:41 (3%) 41:47 (1%) 39:45 (2%) 35:41 (3%)	view
6	x 3	cloudtrail s3_elastic_index fastlylogs	read_cloudtrail.rb reindex_s3.rb parse_fastly_log.rb	55:60 (3%) 59:64 (2%) 54:59 (3%)	view
6	x 3	cloudtrail s3_elastic_index fastlylogs	read_cloudtrail.rb reindex_s3.rb parse_fastly_log.rb	53:58 (3%) 57:62 (2%) 52:57 (3%)	view
6	x 3	ROOT ROOT ROOT	interactivepublisher.pl interactivepublisher.pl interactivepublisher.pl	86:92 (1%) 100:106 (1%) 161:167 (1%)	view
6	x 3	cloudtrail fastlylogs s3_elastic_index	read_cloudtrail.rb fastly_log_chopper.rb reindex_s3.rb	23:31 (3%) 29:37 (1%) 27:35 (2%)	view
6	x 3	cloudtrail s3_elastic_index fastlylogs	read_cloudtrail.rb reindex_s3.rb parse_fastly_log.rb	69:74 (3%) 80:85 (2%) 68:73 (3%)	view
6	x 3	cloudtrail s3_elastic_index fastlylogs	read_cloudtrail.rb reindex_s3.rb parse_fastly_log.rb	70:76 (3%) 81:87 (2%) 69:75 (3%)	view
6	x 3	cloudtrail s3_elastic_index fastlylogs	read_cloudtrail.rb reindex_s3.rb parse_fastly_log.rb	52:57 (3%) 56:61 (2%) 51:56 (3%)	view
6	x 3	cloudtrail fastlylogs s3_elastic_index	read_cloudtrail.rb fastly_log_chopper.rb reindex_s3.rb	27:34 (3%) 33:40 (1%) 31:38 (2%)	view
6	x 3	cloudtrail fastlylogs s3_elastic_index	read_cloudtrail.rb fastly_log_chopper.rb reindex_s3.rb	39:47 (3%) 45:53 (1%) 43:51 (2%)	view
6	x 3	cloudtrail fastlylogs s3_elastic_index	read_cloudtrail.rb fastly_log_chopper.rb reindex_s3.rb	25:33 (3%) 31:39 (1%) 29:37 (2%)	view
6	x 3	cloudtrail s3_elastic_index fastlylogs	read_cloudtrail.rb reindex_s3.rb parse_fastly_log.rb	54:59 (3%) 58:63 (2%) 53:58 (3%)	view
6	x 3	cloudtrail fastlylogs s3_elastic_index	read_cloudtrail.rb fastly_log_chopper.rb reindex_s3.rb	24:32 (3%) 30:38 (1%) 28:36 (2%)	view
6	x 3	cloudtrail fastlylogs s3_elastic_index	read_cloudtrail.rb fastly_log_chopper.rb reindex_s3.rb	41:48 (3%) 47:54 (1%) 45:52 (2%)	view
6	x 3	cloudtrail fastlylogs s3_elastic_index	read_cloudtrail.rb fastly_log_chopper.rb reindex_s3.rb	22:30 (3%) 28:36 (1%) 26:34 (2%)	view

Duplicated Units

The list of top 1 duplicated units.

See data for all 1 unit duplicate

Size	#	Folders	Files	Lines	Code
17	x 2	cloudtrail fastlylogs	read_cloudtrail.rb fastly_log_chopper.rb	107:130 343:366	view

About This Analysis

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

To learn more about duplications and techniques for eliminating duplication, Sokrates recommends the following resources:

Avoid Repetition, MartinFlower.com (IEEE Software article)
Refactoring duplicated code, sourcemaking.com
Beck Design Rules, MartinFowler.com
DRY (Don't Repeat Yourself) Principle, Wikipedia
The Cut-and-Paste Programming Software Development Anti-Pattern, sourcemaking.com
Code Smells / Duplicated Code, Jeff Atwood, codinghorror.com:

"Duplicated code is the bane of software development. Stamp out duplication whenever possible. You should always be on the lookout for more subtle cases of near-duplication, too. Don't Repeat Yourself!"

Reducing Coupling, MartinFlower.com (IEEE Software article):

"There are several ways to describe coupling, but it boils down to this: If changing one module in a program requires changing another module, then coupling exists. ... Duplication always implies coupling, because changing one piece of duplicate code implies changing the other."