Uber / RemoteShuffleServiceDuplication

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

8% duplication:

12,304 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
1,037 duplicated lines

100 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Longest Duplicates

The list of 50 longest duplicates.

See data for all 100 duplicates...

Size	#	Folders	Files	Lines	Code
50	x 2	src/main/java/com/uber/rss/clients src/main/java/com/uber/rss/clients	RecordSocketReadClient.java ShuffleDataSocketReadClient.java	44:111 (87%) 44:111 (87%)	view
44	x 2	src/main/java/com/uber/rss/messages src/main/java/com/uber/rss/messages	ConnectDownloadRequest.java ConnectDownloadRequestMessage.java	54:115 (69%) 58:119 (65%)	view
18	x 2	src/main/java/com/uber/rss/clients src/main/java/com/uber/rss/clients	MultiServerAsyncWriteClient.java MultiServerSyncWriteClient.java	368:391 (7%) 185:208 (15%)	view
16	x 2	src/main/java/com/uber/rss/clients src/main/java/com/uber/rss/clients	TaskByteArrayDataBlock.java TaskDataBlock.java	31:55 (45%) 31:55 (45%)	view
14	x 2	src/main/java/com/uber/rss/clients src/main/java/com/uber/rss/clients	TaskByteArrayDataBlock.java TaskDataBlock.java	57:74 (40%) 57:74 (40%)	view
13	x 2	src/main/java/com/uber/rss/clients src/main/java/com/uber/rss/clients	MultiServerAsyncWriteClient.java MultiServerSyncWriteClient.java	91:104 (5%) 66:79 (11%)	view
12	x 2	src/main/java/com/uber/rss/clients src/main/java/com/uber/rss/clients	MultiServerAsyncWriteClient.java MultiServerSyncWriteClient.java	297:315 (4%) 164:181 (10%)	view
11	x 2	src/main/java/com/uber/rss/clients src/main/java/com/uber/rss/clients	MultiServerAsyncWriteClient.java MultiServerSyncWriteClient.java	256:273 (4%) 132:149 (9%)	view
11	x 2	src/main/java/com/uber/rss/handlers src/main/java/com/uber/rss/handlers	NotifyChannelInboundHandler.java RegistryChannelInboundHandler.java	40:55 (33%) 41:56 (33%)	view
10	x 2	src/main/java/com/uber/rss/handlers src/main/java/com/uber/rss/handlers	NotifyChannelInboundHandler.java UploadChannelInboundHandler.java	64:77 (30%) 218:231 (7%)	view
10	x 2	src/main/java/com/uber/rss/metrics src/main/java/com/uber/rss/metrics	NotifyClientMetricsKey.java WriteClientMetricsKey.java	24:39 (38%) 24:39 (38%)	view
10	x 2	src/main/scala/org/apache/spark/shuffle src/main/scala/org/apache/spark/shuffle/rss	RssShuffleReader.scala BlockDownloaderPartitionRangeRecordIt...	34:43 (13%) 41:50 (7%)	view
10	x 2	src/main/java/com/uber/rss/handlers src/main/java/com/uber/rss/handlers	NotifyChannelInboundHandler.java RegistryChannelInboundHandler.java	64:77 (30%) 65:78 (30%)	view
10	x 2	src/main/java/com/uber/rss/handlers src/main/java/com/uber/rss/handlers	RegistryChannelInboundHandler.java UploadChannelInboundHandler.java	65:78 (30%) 218:231 (7%)	view
10	x 2	src/main/java/com/uber/rss/metrics src/main/java/com/uber/rss/metrics	ReadClientMetricsKey.java WriteClientMetricsKey.java	24:39 (38%) 24:39 (38%)	view
10	x 2	src/main/java/com/uber/rss/metrics src/main/java/com/uber/rss/metrics	NotifyClientMetricsKey.java ReadClientMetricsKey.java	24:39 (38%) 24:39 (38%)	view
10	x 2	src/main/scala/org/apache/spark/shuffle src/main/scala/org/apache/spark/shuffle	RssShuffleManager.scala RssShuffleManager.scala	200:214 (3%) 437:448 (3%)	view
9	x 2	src/main/java/com/uber/rss/execution src/main/java/com/uber/rss/execution	ShuffleExecutor.java ShuffleExecutor.java	397:410 (2%) 429:442 (2%)	view
8	x 2	src/main/java/com/uber/rss/clients src/main/java/com/uber/rss/clients	DataBlockSocketReadClient.java ShuffleDataSocketReadClient.java	285:295 (3%) 100:110 (14%)	view
8	x 2	src/main/java/com/uber/rss/common src/main/java/com/uber/rss/messages	AppShufflePartitionId.java ConnectDownloadRequest.java	46:59 (16%) 86:99 (12%)	view
8	x 2	src/main/java/com/uber/rss/clients src/main/java/com/uber/rss/clients	DataBlockSocketReadClient.java DataBlockSyncWriteClient.java	288:296 (3%) 181:189 (7%)	view
8	x 2	src/main/java/com/uber/rss/messages src/main/java/com/uber/rss/messages	GetDataAvailabilityRequest.java GetDataAvailabilityRequestMessage.java	32:43 (61%) 32:43 (61%)	view
8	x 2	src/main/java/com/uber/rss/clients src/main/java/com/uber/rss/clients	MultiServerAsyncWriteClient.java MultiServerSyncWriteClient.java	111:119 (3%) 81:89 (6%)	view
8	x 2	src/main/java/com/uber/rss/clients src/main/java/com/uber/rss/clients	DataBlockSocketReadClient.java RecordSocketReadClient.java	285:295 (3%) 100:110 (14%)	view
8	x 2	src/main/java/com/uber/rss/common src/main/java/com/uber/rss/common	AppMapId.java AppTaskAttemptId.java	39:52 (19%) 45:58 (14%)	view
8	x 2	src/main/java/com/uber/rss/metadata src/main/java/com/uber/rss/metadata	InMemoryServiceRegistry.java ZooKeeperServiceRegistry.java	48:61 (21%) 92:105 (5%)	view
8	x 2	src/main/java/com/uber/rss/common src/main/java/com/uber/rss/common	FilePathAndLength.java PartitionFilePathAndLength.java	28:39 (30%) 34:45 (25%)	view
8	x 2	src/main/java/com/uber/rss/metrics src/main/java/com/uber/rss/metrics	M3DummyScope.java M3DummyScope.java	56:65 (13%) 80:89 (13%)	view
8	x 2	src/main/java/com/uber/rss/clients src/main/java/com/uber/rss/clients	DataBlockSyncWriteClient.java NotifyClient.java	181:189 (7%) 89:97 (20%)	view
8	x 2	src/main/java/com/uber/rss/common src/main/java/com/uber/rss/messages	AppShufflePartitionId.java ConnectDownloadRequestMessage.java	46:59 (16%) 90:103 (11%)	view
8	x 2	src/main/java/com/uber/rss/messages src/main/java/com/uber/rss/messages	ConnectDownloadRequest.java ConnectDownloadRequestMessage.java	41:50 (12%) 45:54 (11%)	view
8	x 2	src/main/java/com/uber/rss/clients src/main/java/com/uber/rss/clients	ServerIdAwareSyncWriteClient.java ServerIdAwareSyncWriteClient.java	106:113 (8%) 116:123 (8%)	view
8	x 2	src/main/java/com/uber/rss/clients src/main/java/com/uber/rss/clients	DataBlockSocketReadClient.java NotifyClient.java	288:296 (3%) 89:97 (20%)	view
8	x 2	src/main/java/com/uber/rss/clients src/main/java/com/uber/rss/clients	DataBlockSocketReadClient.java DataBlockSocketReadClient.java	154:162 (3%) 221:229 (3%)	view
7	x 2	src/main/java/com/uber/rss/clients src/main/java/com/uber/rss/clients	MultiServerAsyncWriteClient.java MultiServerSyncWriteClient.java	125:135 (2%) 95:105 (6%)	view
7	x 2	src/main/java/com/uber/rss/metrics src/main/java/com/uber/rss/metrics	ReadClientMetricsKey.java WriteClientMetricsKey.java	41:51 (26%) 41:51 (26%)	view
7	x 2	src/main/java/com/uber/rss/clients src/main/java/com/uber/rss/clients	NotifyClient.java ShuffleDataSocketReadClient.java	89:96 (17%) 103:110 (12%)	view
7	x 2	src/main/scala/org/apache/spark/shuffle src/main/scala/org/apache/spark/shuffle	RssShuffleManager.scala RssShuffleManager.scala	239:245 (2%) 257:263 (2%)	view
7	x 2	src/main/java/com/uber/rss/metrics src/main/java/com/uber/rss/metrics	NotifyClientMetricsKey.java WriteClientMetricsKey.java	41:51 (26%) 41:51 (26%)	view
7	x 2	src/main/java/com/uber/rss/clients src/main/java/com/uber/rss/clients	DataBlockSyncWriteClient.java RecordSocketReadClient.java	181:188 (6%) 103:110 (12%)	view
7	x 2	src/main/java/com/uber/rss/tools src/main/java/com/uber/rss/tools	StreamServerStressToolLongRun.java StreamServerStressToolWrite64GB.java	109:116 (6%) 27:34 (33%)	view
7	x 2	src/main/java/com/uber/rss/clients src/main/java/com/uber/rss/clients	NotifyClient.java RecordSocketReadClient.java	89:96 (17%) 103:110 (12%)	view
7	x 2	src/main/java/com/uber/rss/metrics src/main/java/com/uber/rss/metrics	NotifyClientMetricsKey.java ReadClientMetricsKey.java	41:51 (26%) 41:51 (26%)	view
7	x 2	src/main/scala/org/apache/spark/shuffle/rss src/main/scala/org/apache/spark/shuffle/rss	BlockDownloaderPartitionRangeRecordIt... BlockDownloaderPartitionRecordIterato...	151:158 (5%) 149:155 (5%)	view
7	x 2	src/main/java/com/uber/rss/clients src/main/java/com/uber/rss/clients	DataBlockSyncWriteClient.java ShuffleDataSocketReadClient.java	181:188 (6%) 103:110 (12%)	view
6	x 2	src/main/java/com/uber/rss/clients src/main/java/com/uber/rss/clients	ServerIdAwareSyncWriteClient.java UnpooledWriteClientFactory.java	108:113 (6%) 37:42 (35%)	view
6	x 2	src/main/java/com/uber/rss/clients src/main/java/com/uber/rss/clients	RetriableSocketReadClient.java RetriableSocketReadClient.java	44:49 (8%) 54:59 (8%)	view
6	x 2	src/main/java/com/uber/rss/messages src/main/java/com/uber/rss/messages	ConnectDownloadRequest.java ConnectUploadRequest.java	82:91 (9%) 50:59 (17%)	view
6	x 2	src/main/java/com/uber/rss/messages src/main/java/com/uber/rss/messages	ConnectDownloadRequestMessage.java HeartbeatMessage.java	86:95 (8%) 54:63 (14%)	view
6	x 2	src/main/java/com/uber/rss/decoders src/main/java/com/uber/rss/decoders	StreamServerMessageDecoder.java StreamServerMessageDecoder.java	154:160 (2%) 180:186 (2%)	view

Duplicated Units

The list of top 11 duplicated units.

See data for all 11 unit duplicates...

Size	#	Folders	Files	Lines	Code
12	x 2	src/main/java/com/uber/rss/clients src/main/java/com/uber/rss/clients	RecordSocketReadClient.java ShuffleDataSocketReadClient.java	49:62 49:62	view
12	x 2	src/main/java/com/uber/rss/clients src/main/java/com/uber/rss/clients	RecordSocketReadClient.java ShuffleDataSocketReadClient.java	76:89 76:89	view
12	x 2	src/main/java/com/uber/rss/messages src/main/java/com/uber/rss/messages	ConnectDownloadRequest.java ConnectDownloadRequestMessage.java	67:80 71:84	view
9	x 3	src/main/java/com/uber/rss/clients src/main/java/com/uber/rss/clients src/main/java/com/uber/rss/clients	DataBlockSocketReadClient.java NotifyClient.java DataBlockSyncWriteClient.java	288:298 89:99 181:191	view
9	x 2	src/main/java/com/uber/rss/clients src/main/java/com/uber/rss/clients	RecordSocketReadClient.java ShuffleDataSocketReadClient.java	103:113 103:113	view
9	x 2	src/main/java/com/uber/rss/messages src/main/java/com/uber/rss/messages	ConnectDownloadRequest.java ConnectDownloadRequestMessage.java	55:65 59:69	view
8	x 2	src/main/java/com/uber/rss/clients src/main/java/com/uber/rss/clients	MultiServerAsyncWriteClient.java MultiServerSyncWriteClient.java	308:317 174:183	view
8	x 2	src/main/java/com/uber/rss/messages src/main/java/com/uber/rss/messages	ConnectDownloadRequest.java ConnectDownloadRequestMessage.java	107:116 111:120	view
7	x 2	src/main/java/com/uber/rss/clients src/main/java/com/uber/rss/clients	MultiServerAsyncWriteClient.java MultiServerSyncWriteClient.java	262:270 138:146	view
6	x 2	src/main/java/com/uber/rss/clients src/main/java/com/uber/rss/clients	MultiServerAsyncWriteClient.java MultiServerSyncWriteClient.java	368:375 185:192	view
6	x 2	src/main/java/com/uber/rss/clients src/main/java/com/uber/rss/clients	RecordSocketReadClient.java ShuffleDataSocketReadClient.java	65:73 65:73	view