apache / datafu - Duplication

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

20% duplication:

11,024 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
2,272 duplicated lines

271 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Longest Duplicates

The list of 50 longest duplicates.

See data for all 271 duplicates...

Size	#	Folders	Files	Lines	Code
105	x 2	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	DoubleVAR.java FloatVAR.java	107:260 (47%) 109:262 (47%)	view
73	x 2	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	DoubleVAR.java VAR.java	107:215 (33%) 124:232 (31%)	view
73	x 2	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	FloatVAR.java VAR.java	109:217 (33%) 124:232 (31%)	view
61	x 2	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	IntVAR.java LongVAR.java	116:204 (27%) 116:205 (27%)	view
51	x 2	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	IntVAR.java LongVAR.java	295:370 (23%) 296:371 (23%)	view
51	x 2	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	DoubleVAR.java FloatVAR.java	296:371 (23%) 298:373 (23%)	view
40	x 2	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	DoubleVAR.java VAR.java	316:371 (18%) 345:400 (17%)	view
40	x 2	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	FloatVAR.java VAR.java	318:373 (18%) 345:400 (17%)	view
37	x 2	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	FloatVAR.java LongVAR.java	116:166 (16%) 114:164 (16%)	view
37	x 2	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	DoubleVAR.java LongVAR.java	114:164 (16%) 114:164 (16%)	view
37	x 2	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	LongVAR.java VAR.java	114:164 (16%) 131:181 (16%)	view
36	x 2	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	FloatVAR.java IntVAR.java	118:166 (16%) 116:163 (16%)	view
36	x 2	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	DoubleVAR.java IntVAR.java	116:164 (16%) 116:163 (16%)	view
36	x 2	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	IntVAR.java VAR.java	116:163 (16%) 133:181 (15%)	view
31	x 2	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	FloatVAR.java VAR.java	219:262 (14%) 233:276 (13%)	view
31	x 2	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	DoubleVAR.java VAR.java	217:260 (14%) 233:276 (13%)	view
29	x 2	datafu-pig/src/main/java/datafu/pig/text/opennlp datafu-pig/src/main/java/datafu/pig/text/opennlp	TokenizeSimple.java TokenizeWhitespace.java	52:98 (93%) 53:99 (93%)	view
29	x 2	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	DoubleVAR.java VAR.java	46:93 (13%) 62:109 (12%)	view
29	x 2	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	FloatVAR.java VAR.java	46:93 (13%) 62:109 (12%)	view
29	x 2	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	IntVAR.java LongVAR.java	46:93 (13%) 46:93 (13%)	view
29	x 2	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	DoubleVAR.java FloatVAR.java	46:93 (13%) 46:93 (13%)	view
27	x 2	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	IntVAR.java LongVAR.java	221:259 (12%) 222:260 (12%)	view
24	x 2	datafu-pig/src/main/java/datafu/pig/bags datafu-pig/src/main/java/datafu/pig/bags	Enumerate.java ReverseEnumerate.java	109:150 (45%) 97:138 (55%)	view
22	x 2	datafu-hourglass/src/mai...afu/hourglass/mapreduce datafu-hourglass/src/mai...afu/hourglass/mapreduce	CollapsingCombiner.java CollapsingReducer.java	203:240 (27%) 278:315 (18%)	view
21	x 2	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	FloatVAR.java IntVAR.java	339:368 (9%) 336:365 (9%)	view
21	x 2	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	IntVAR.java VAR.java	336:365 (9%) 366:395 (9%)	view
21	x 2	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	DoubleVAR.java LongVAR.java	337:366 (9%) 337:366 (9%)	view
21	x 2	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	DoubleVAR.java IntVAR.java	337:366 (9%) 336:365 (9%)	view
21	x 2	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	FloatVAR.java LongVAR.java	339:368 (9%) 337:366 (9%)	view
21	x 2	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	LongVAR.java VAR.java	337:366 (9%) 366:395 (9%)	view
20	x 2	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	DoubleVAR.java VAR.java	262:291 (9%) 279:308 (8%)	view
19	x 2	datafu-pig/src/main/resources/datafu datafu-pig/src/main/resources/datafu	pig left_outer_join.pig sample_by_keys.pig	1:20 (51%) 1:20 (50%)	view
19	x 2	datafu-pig/src/main/resources/datafu datafu-pig/src/main/resources/datafu	pig dedup.pig left_outer_join.pig	1:20 (55%) 1:20 (51%)	view
19	x 2	datafu-pig/src/main/resources/datafu datafu-pig/src/main/resources/datafu	pig count_macros.pig diff_macros.pig	1:20 (50%) 1:20 (55%)	view
19	x 2	datafu-hourglass/src/mai...a/datafu/hourglass/jobs datafu-hourglass/src/mai...a/datafu/hourglass/jobs	PartitionCollapsingIncrementalJob.java PartitionPreservingIncrementalJob.java	60:96 (32%) 57:93 (40%)	view
19	x 2	datafu-pig/src/main/resources/datafu datafu-pig/src/main/resources/datafu	pig count_macros.pig left_outer_join.pig	1:20 (50%) 1:20 (51%)	view
19	x 2	datafu-pig/src/main/java/datafu/pig/text/opennlp datafu-pig/src/main/java/datafu/pig/text/opennlp	TokenizeME.java TokenizeSimple.java	93:123 (41%) 68:98 (61%)	view
19	x 2	datafu-pig/src/main/resources/datafu datafu-pig/src/main/resources/datafu	pig diff_macros.pig left_outer_join.pig	1:20 (55%) 1:20 (51%)	view
19	x 2	datafu-pig/src/main/resources/datafu datafu-pig/src/main/resources/datafu	pig count_macros.pig sample_by_keys.pig	1:20 (50%) 1:20 (50%)	view
19	x 2	datafu-pig/src/main/resources/datafu datafu-pig/src/main/resources/datafu	pig dedup.pig sample_by_keys.pig	1:20 (55%) 1:20 (50%)	view
19	x 2	datafu-pig/src/main/java/datafu/pig/text/opennlp datafu-pig/src/main/java/datafu/pig/text/opennlp	TokenizeME.java TokenizeWhitespace.java	93:123 (41%) 69:99 (61%)	view
19	x 2	datafu-pig/src/main/resources/datafu datafu-pig/src/main/resources/datafu	pig count_macros.pig dedup.pig	1:20 (50%) 1:20 (55%)	view
19	x 2	datafu-pig/src/main/resources/datafu datafu-pig/src/main/resources/datafu	pig dedup.pig diff_macros.pig	1:20 (55%) 1:20 (55%)	view
19	x 2	datafu-pig/src/main/resources/datafu datafu-pig/src/main/resources/datafu	pig diff_macros.pig sample_by_keys.pig	1:20 (55%) 1:20 (50%)	view
18	x 2	site/source/blog site/source/stylesheets	index.html.erb highlight.css.erb	4:21 (54%) 1:18 (94%)	view
18	x 2	site/source/layouts site/source/layouts	_header.erb blog.erb	1:18 (81%) 1:18 (46%)	view
18	x 2	site/source/blog site/source/layouts	index.html.erb _footer.erb	4:21 (54%) 1:18 (75%)	view
18	x 2	site/source/layouts site/source/stylesheets	blog.erb highlight.css.erb	1:18 (46%) 1:18 (94%)	view
18	x 2	site/source/layouts site/source/layouts	_docs_nav.erb blog.erb	1:18 (36%) 1:18 (46%)	view
18	x 2	site/source/layouts site/source/layouts	_footer.erb docs.erb	1:18 (75%) 1:18 (60%)	view

Duplicated Units

The list of top 24 duplicated units.

See data for all 24 unit duplicates...

Size	#	Folders	Files	Lines	Code
31	x 2	datafu-pig/src/main/java/datafu/pig/bags datafu-pig/src/main/java/datafu/pig/bags	ReverseEnumerate.java Enumerate.java	101:140 113:152	view
29	x 2	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	FloatVAR.java DoubleVAR.java	193:235 191:233	view
27	x 2	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	LongVAR.java IntVAR.java	321:354 320:353	view
27	x 3	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	VAR.java FloatVAR.java DoubleVAR.java	350:383 323:356 321:354	view
25	x 2	datafu-hourglass/src/mai...afu/hourglass/mapreduce datafu-hourglass/src/mai...afu/hourglass/mapreduce	CollapsingCombiner.java CollapsingReducer.java	215:242 290:317	view
23	x 2	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	LongVAR.java IntVAR.java	161:188 160:187	view
23	x 3	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	VAR.java FloatVAR.java DoubleVAR.java	178:205 163:190 161:188	view
20	x 4	datafu-pig/src/main/java/datafu/pig/text/opennlp datafu-pig/src/main/java/datafu/pig/text/opennlp datafu-pig/src/main/java/datafu/pig/text/opennlp datafu-pig/src/main/java/datafu/pig/text/opennlp	TokenizeSimple.java SentenceDetect.java TokenizeWhitespace.java TokenizeME.java	76:100 97:121 77:101 101:125	view
17	x 2	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	LongVAR.java IntVAR.java	49:72 49:72	view
17	x 3	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	VAR.java FloatVAR.java DoubleVAR.java	65:88 49:72 49:72	view
15	x 2	datafu-pig/src/main/java/datafu/pig/text/opennlp datafu-pig/src/main/java/datafu/pig/text/opennlp	TokenizeSimple.java TokenizeWhitespace.java	55:73 56:74	view
14	x 2	datafu-hourglass/src/mai...a/datafu/hourglass/jobs datafu-hourglass/src/mai...a/datafu/hourglass/jobs	PartitionPreservingExecutionPlanner.java PartitionCollapsingExecutionPlanner.java	212:227 138:153	view
11	x 2	datafu-hourglass/src/mai...a/datafu/hourglass/jobs datafu-hourglass/src/mai...a/datafu/hourglass/jobs	AbstractPartitionCollapsingIncrementa... AbstractPartitionPreservingIncrementa...	349:361 225:237	view
10	x 5	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	LongVAR.java VAR.java IntVAR.java FloatVAR.java DoubleVAR.java	144:156 161:173 144:155 146:158 144:156	view
9	x 2	datafu-hourglass/src/mai...atafu/hourglass/schemas datafu-hourglass/src/mai...atafu/hourglass/schemas	PartitionCollapsingSchemas.java PartitionPreservingSchemas.java	155:165 103:113	view
9	x 5	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	LongVAR.java VAR.java IntVAR.java FloatVAR.java DoubleVAR.java	235:246 251:262 234:245 237:248 235:246	view
8	x 2	datafu-pig/src/main/java/datafu/pig/bags datafu-pig/src/main/java/datafu/pig/bags	PrependToBag.java AppendToBag.java	67:76 61:70	view
7	x 2	datafu-hourglass/src/mai...atafu/hourglass/schemas datafu-hourglass/src/mai...atafu/hourglass/schemas	PartitionCollapsingSchemas.java PartitionPreservingSchemas.java	131:139 93:101	view
7	x 2	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	LongVAR.java IntVAR.java	364:372 363:371	view
7	x 3	datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats datafu-pig/src/main/java/datafu/pig/stats	VAR.java FloatVAR.java DoubleVAR.java	393:401 366:374 364:372	view
6	x 2	datafu-hourglass/src/mai...a/datafu/hourglass/jobs datafu-hourglass/src/mai...a/datafu/hourglass/jobs	PartitionPreservingIncrementalJob.java PartitionCollapsingIncrementalJob.java	171:178 212:219	view
6	x 2	datafu-hourglass/src/mai...atafu/hourglass/schemas datafu-hourglass/src/mai...atafu/hourglass/schemas	PartitionCollapsingSchemas.java PartitionPreservingSchemas.java	141:148 124:131	view
6	x 2	datafu-hourglass/src/mai...atafu/hourglass/schemas datafu-hourglass/src/mai...atafu/hourglass/schemas	PartitionCollapsingSchemas.java PartitionPreservingSchemas.java	188:195 146:153	view
6	x 2	datafu-pig/src/main/java/datafu/pig/hash/lsh datafu-pig/src/main/java/datafu/pig/hash/lsh	L1PStableHash.java L2PStableHash.java	167:174 168:175	view