GoogleCloudPlatform / gcs-connector-for-pytorch

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

31% duplication:

1,917 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
611 duplicated lines

31 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Longest Duplicates

The list of 31 longest duplicates.

See data for all 31 duplicates...

Size	#	Folders	Files	Lines	Code
46	x 2	kokoro kokoro	cfg pypi.cfg pypi_presubmit.cfg	15:71 (100%) 15:71 (100%)	view
42	x 2	kokoro kokoro	cfg presubmit_base.cfg pypi_presubmit.cfg	17:68 (82%) 17:68 (91%)	view
42	x 2	kokoro kokoro	cfg presubmit_base.cfg pypi.cfg	17:68 (82%) 17:68 (91%)	view
32	x 2	dataflux_pytorch/benchmark/checkpointing/simulated dataflux_pytorch/benchmark/checkpointing/simulated	benchmark.py llama2.py	14:51 (35%) 15:52 (35%)	view
16	x 2	dataflux_pytorch/benchmark/checkpointing/multinode dataflux_pytorch/benchmark/checkpointing/simulated	benchmark-deploy.yaml benchmark-deployment.yaml	54:69 (19%) 47:62 (19%)	view
14	x 2	dataflux_pytorch/benchmark/checkpointing/multinode dataflux_pytorch/benchmark/checkpointing/multinode	train.py train.py	140:153 (7%) 185:198 (7%)	view
13	x 2	dataflux_pytorch/benchmark/checkpointing/multinode dataflux_pytorch/benchmark/standalone_dataloader	benchmark-deploy.yaml deployment.yaml	57:69 (15%) 51:64 (14%)	view
13	x 2	dataflux_pytorch dataflux_pytorch	dataflux_iterable_dataset.py dataflux_mapstyle_dataset.py	96:126 (10%) 99:132 (10%)	view
13	x 2	dataflux_pytorch dataflux_pytorch	dataflux_iterable_dataset.py dataflux_mapstyle_dataset.py	28:72 (10%) 29:73 (10%)	view
13	x 2	dataflux_pytorch dataflux_pytorch	dataflux_iterable_dataset.py dataflux_mapstyle_dataset.py	200:217 (10%) 206:225 (10%)	view
13	x 2	dataflux_pytorch/benchmark/checkpointing/simulated dataflux_pytorch/benchmark/standalone_dataloader	benchmark-deployment.yaml deployment.yaml	50:62 (15%) 51:64 (14%)	view
11	x 2	dataflux_pytorch dataflux_pytorch	dataflux_iterable_dataset.py dataflux_mapstyle_dataset.py	73:83 (9%) 75:85 (8%)	view
11	x 2	dataflux_pytorch dataflux_pytorch	dataflux_iterable_dataset.py dataflux_mapstyle_dataset.py	187:198 (9%) 193:204 (8%)	view
10	x 2	dataflux_pytorch/benchmark/checkpointing/multinode dataflux_pytorch/benchmark/checkpointing/simulated	benchmark-deploy.yaml benchmark-deployment.yaml	33:46 (11%) 33:43 (12%)	view
9	x 2	dataflux_pytorch/benchmark/checkpointing/simulated dataflux_pytorch/benchmark/checkpointing/simulated	benchmark.py llama2.py	88:96 (10%) 90:98 (10%)	view
9	x 2	dataflux_pytorch/benchmark/standalone_dataloader dataflux_pytorch/benchmark/standalone_dataloader	standalone_dataloader.py standalone_dataloader.py	263:271 (4%) 289:297 (4%)	view
9	x 2	dataflux_pytorch/benchmark/checkpointing/multinode dataflux_pytorch/benchmark/checkpointing/simulated	benchmark-deploy.yaml benchmark-deployment.yaml	96:105 (10%) 93:102 (10%)	view
8	x 2	dataflux_pytorch/benchmark/checkpointing/multinode dataflux_pytorch/benchmark/checkpointing/simulated	benchmark-deploy.yaml benchmark-deployment.yaml	19:26 (9%) 19:26 (9%)	view
8	x 2	dataflux_pytorch/lightning dataflux_pytorch/lightning	gcs_filesystem.py gcs_filesystem.py	107:114 (8%) 122:129 (8%)	view
8	x 2	dataflux_pytorch/multipart_upload dataflux_pytorch/multipart_upload	multipart.py multipart.py	195:202 (6%) 296:303 (6%)	view
7	x 2	dataflux_pytorch dataflux_pytorch	dataflux_iterable_dataset.py dataflux_mapstyle_dataset.py	137:143 (5%) 145:151 (5%)	view
7	x 2	dataflux_pytorch/benchmark/standalone_dataloader dataflux_pytorch	standalone_dataloader.py dataflux_iterable_dataset.py	286:292 (3%) 178:184 (5%)	view
6	x 2	kokoro kokoro	cfg presubmit_base.cfg pypi_presubmit.cfg	71:77 (11%) 65:71 (13%)	view
6	x 2	dataflux_pytorch/benchmark/checkpointing/multinode dataflux_pytorch/benchmark/standalone_dataloader	benchmark-deploy.yaml deployment.yaml	21:26 (7%) 10:15 (6%)	view
6	x 2	kokoro kokoro	cfg presubmit_base.cfg pypi.cfg	71:77 (11%) 65:71 (13%)	view
6	x 2	dataflux_pytorch/benchmark/checkpointing/multinode dataflux_pytorch/benchmark/checkpointing/simulated	benchmark-deploy.yaml benchmark-deployment.yaml	85:90 (7%) 77:82 (7%)	view
6	x 2	dataflux_pytorch dataflux_pytorch	dataflux_iterable_dataset.py dataflux_mapstyle_dataset.py	145:152 (5%) 153:160 (4%)	view
6	x 2	dataflux_pytorch/benchmark/checkpointing/simulated dataflux_pytorch/benchmark/checkpointing/simulated	benchmark.py llama2.py	58:64 (6%) 59:65 (6%)	view
6	x 2	dataflux_pytorch/lightning dataflux_pytorch/lightning	dataflux_lightning_checkpoint.py dataflux_lightning_checkpoint.py	34:39 (8%) 93:98 (8%)	view
6	x 2	dataflux_pytorch dataflux_pytorch	dataflux_iterable_dataset.py dataflux_mapstyle_dataset.py	84:91 (5%) 87:94 (4%)	view
6	x 2	dataflux_pytorch/benchmark/checkpointing/simulated dataflux_pytorch/benchmark/standalone_dataloader	benchmark-deployment.yaml deployment.yaml	21:26 (7%) 10:15 (6%)	view