GoogleCloudPlatform / public-datasets-pipelines
Duplication

Places in code with 6 or more lines that are exactly the same.

Intro
Learn more...
Duplication Overall
system52% (54,138 lines)
dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...
Duplication per Extension
py51% (23,467 lines)
yaml49% (19,907 lines)
tf60% (10,551 lines)
jinja260% (213 lines)
Duplication per Component (primary)
datasets52% (53,832 lines)
templates60% (213 lines)
scripts8% (93 lines)
ROOT0% (0 lines)

Duplication Between Components (50+ lines)

G datasets datasets templates templates datasets--templates 8057

Download: SVG DOT (open online Graphviz editor)

Open 2D force graph... Open 3D force graph...

Show more details on duplication between components...
Longest Duplicates
The list of 50 longest duplicates.
See data for all 150,042 duplicates...
Size#FoldersFilesLinesCode
319 x 2 datasets/noaa/pipelines/noaa
datasets/noaa/pipelines/noaa
1501:1820 (10%)
2177:2496 (10%)
view
173 x 2 datasets/sunroof_solar/p...otential_by_censustract
datasets/sunroof_solar/p...otential_by_postal_code
62:234 (73%)
62:234 (73%)
view
166 x 2 datasets/libraries_io/pipelines/repositories
datasets/libraries_io/pipelines/repositories
125:290 (23%)
567:732 (23%)
view
164 x 2 datasets/libraries_io/pipelines/repositories
datasets/libraries_io/pipelines/repositories
130:294 (22%)
351:515 (22%)
view
161 x 2 datasets/libraries_io/pipelines/repositories
datasets/libraries_io/pipelines/repositories
351:511 (22%)
572:732 (22%)
view
156 x 2 datasets/fec/pipelines/opex_2016
datasets/fec/pipelines/opex_2018
66:221 (79%)
66:221 (79%)
view
156 x 2 datasets/fec/pipelines/opex_2018
datasets/fec/pipelines/opex_2020
66:221 (79%)
66:221 (79%)
view
156 x 2 datasets/fec/pipelines/opex_2016
datasets/fec/pipelines/opex_2020
66:221 (79%)
66:221 (79%)
view
150 x 2 datasets/deepmind/pipelines/alphafold
datasets/deepmind/pipelines/alphafold_v4
157:306 (50%)
151:300 (49%)
view
140 x 2 datasets/city_health_das...s/run_csv_transform_kub
datasets/covid19_google_...s/run_csv_transform_kub
275:430 (31%)
255:410 (34%)
view
139 x 2 datasets/fec/pipelines/c...ttee_contributions_2016
datasets/fec/pipelines/c...ttee_contributions_2020
66:205 (76%)
66:205 (76%)
view
139 x 2 datasets/fec/pipelines/c...ttee_contributions_2016
datasets/fec/pipelines/c...ttee_contributions_2018
66:205 (76%)
66:205 (76%)
view
139 x 2 datasets/fec/pipelines/c...ttee_contributions_2018
datasets/fec/pipelines/c...ttee_contributions_2020
66:205 (76%)
66:205 (76%)
view
134 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/individuals_ingest_2020
616:749 (8%)
786:919 (8%)
view
134 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/individuals_ingest_2020
616:749 (8%)
956:1089 (8%)
view
134 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/individuals_ingest_2020
786:919 (8%)
956:1089 (8%)
view
134 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/individuals_ingest_2020
786:919 (8%)
1465:1598 (8%)
view
134 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/individuals_ingest_2020
786:919 (8%)
1296:1429 (8%)
view
134 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/individuals_ingest_2020
786:919 (8%)
1126:1259 (8%)
view
134 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/individuals_ingest_2020
446:579 (8%)
616:749 (8%)
view
134 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/individuals_ingest_2020
446:579 (8%)
956:1089 (8%)
view
134 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/individuals_ingest_2020
446:579 (8%)
786:919 (8%)
view
134 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/individuals_ingest_2020
1126:1259 (8%)
1296:1429 (8%)
view
134 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/individuals_ingest_2020
1126:1259 (8%)
1465:1598 (8%)
view
134 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/individuals_ingest_2020
616:749 (8%)
1126:1259 (8%)
view
134 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/individuals_ingest_2020
616:749 (8%)
1296:1429 (8%)
view
134 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/individuals_ingest_2020
616:749 (8%)
1465:1598 (8%)
view
134 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/individuals_ingest_2020
446:579 (8%)
1126:1259 (8%)
view
134 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/individuals_ingest_2020
446:579 (8%)
1465:1598 (8%)
view
134 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/individuals_ingest_2020
446:579 (8%)
1296:1429 (8%)
view
134 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/individuals_ingest_2020
956:1089 (8%)
1126:1259 (8%)
view
134 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/individuals_ingest_2020
956:1089 (8%)
1296:1429 (8%)
view
134 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/individuals_ingest_2020
956:1089 (8%)
1465:1598 (8%)
view
134 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/individuals_ingest_2020
276:409 (8%)
1465:1598 (8%)
view
134 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/individuals_ingest_2020
276:409 (8%)
1126:1259 (8%)
view
134 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/individuals_ingest_2020
276:409 (8%)
1296:1429 (8%)
view
134 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/individuals_ingest_2020
276:409 (8%)
956:1089 (8%)
view
134 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/individuals_ingest_2020
276:409 (8%)
446:579 (8%)
view
134 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/individuals_ingest_2020
276:409 (8%)
616:749 (8%)
view
134 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/individuals_ingest_2020
276:409 (8%)
786:919 (8%)
view
134 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/individuals_ingest_2020
1296:1429 (8%)
1465:1598 (8%)
view
132 x 2 datasets/fec/pipelines/other_committee_tx_2018
datasets/fec/pipelines/other_committee_tx_2020
66:197 (76%)
101:232 (62%)
view
132 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/other_committee_tx_2020
108:239 (8%)
101:232 (62%)
view
132 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/other_committee_tx_2016
108:239 (8%)
66:197 (76%)
view
132 x 2 datasets/fec/pipelines/other_committee_tx_2016
datasets/fec/pipelines/other_committee_tx_2018
66:197 (76%)
66:197 (76%)
view
132 x 2 datasets/fec/pipelines/other_committee_tx_2016
datasets/fec/pipelines/other_committee_tx_2020
66:197 (76%)
101:232 (62%)
view
132 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/other_committee_tx_2018
108:239 (8%)
66:197 (76%)
view
129 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/other_committee_tx_2020
961:1089 (8%)
104:232 (61%)
view
129 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/other_committee_tx_2016
961:1089 (8%)
69:197 (74%)
view
129 x 2 datasets/fec/pipelines/individuals_ingest_2020
datasets/fec/pipelines/other_committee_tx_2018
1301:1429 (8%)
69:197 (74%)
view
Duplicated Units
The list of top 6 duplicated units.
See data for all 6 unit duplicates...
Size#FoldersFilesLinesCode
18 x 2 datasets/open_buildings_.../_images/run_script_kub
datasets/open_buildings/.../_images/run_script_kub
0:0 
0:0 
view
10 x 3 datasets/census_bureau_i...s/run_csv_transform_kub
datasets/census_bureau_i..._kub_midyear_population
datasets/census_bureau_i..._kub_country_names_area
0:0 
0:0 
0:0 
view
9 x 3 datasets/world_bank_intl...s/run_csv_transform_kub
datasets/world_bank_heal...s/run_csv_transform_kub
datasets/world_bank_intl...s/run_csv_transform_kub
0:0 
0:0 
0:0 
view
7 x 13 datasets/race_and_econom...s/run_csv_transform_kub
datasets/cms_medicare/pi...s/run_csv_transform_kub
datasets/irs_990/pipelin...s/run_csv_transform_kub
datasets/news_hatecrimes...s/run_csv_transform_kub
datasets/cdc_places/pipe...s/run_csv_transform_kub
datasets/san_francisco_t...s/run_csv_transform_kub
datasets/austin_bikeshar...s/run_csv_transform_kub
datasets/covid19_italy/p...s/run_csv_transform_kub
datasets/cfpb_complaints...s/run_csv_transform_kub
datasets/decennial_censu...s/run_csv_transform_kub
...
0:0 
0:0 
0:0 
0:0 
0:0 
0:0 
0:0 
0:0 
0:0 
0:0 
...
view
7 x 3 datasets/america_health_...s/run_csv_transform_kub
datasets/austin_waste/pi...s/run_csv_transform_kub
datasets/mnist/pipelines...s/run_csv_transform_kub
0:0 
0:0 
0:0 
view
6 x 2 datasets/google_cloud_re..._images/copy_bq_dataset
datasets/scalable_open_s...images/bq_data_transfer
0:0 
0:0 
view