duplicated block id: 1
size: 32 cleaned lines of code
in 2 files:
 - snakemake/profiles/slurm-csd3/submit.py (12:59)
 - snakemake/profiles/slurm-moz/submit.py (12:59)

duplicated block id: 2
size: 25 cleaned lines of code
in 2 files:
 - pipeline/common/datasets.py (51:86)
 - taskcluster/translations_taskgraph/util/dataset_helpers.py (15:50)

duplicated block id: 3
size: 22 cleaned lines of code
in 2 files:
 - snakemake/profiles/slurm-csd3/status.py (72:93)
 - snakemake/profiles/slurm-moz/status.py (43:64)

duplicated block id: 4
size: 18 cleaned lines of code
in 2 files:
 - taskcluster/translations_taskgraph/actions/train.py (271:288)
 - taskcluster/translations_taskgraph/actions/train.py (290:307)

duplicated block id: 5
size: 16 cleaned lines of code
in 2 files:
 - site/model-registry/model-registry.mjs (688:703)
 - site/model-registry/model-registry.mjs (742:757)

duplicated block id: 6
size: 15 cleaned lines of code
in 2 files:
 - snakemake/profiles/slurm-csd3/submit.py (65:83)
 - snakemake/profiles/slurm-moz/submit.py (60:78)

duplicated block id: 7
size: 14 cleaned lines of code
in 2 files:
 - utils/find_corpus.py (249:263)
 - utils/find_corpus.py (296:310)

duplicated block id: 8
size: 14 cleaned lines of code
in 2 files:
 - utils/find_corpus.py (202:216)
 - utils/find_corpus.py (249:263)

duplicated block id: 9
size: 14 cleaned lines of code
in 2 files:
 - utils/find_corpus.py (202:216)
 - utils/find_corpus.py (296:310)

duplicated block id: 10
size: 12 cleaned lines of code
in 2 files:
 - taskcluster/translations_taskgraph/transforms/from_datasets.py (105:118)
 - taskcluster/translations_taskgraph/transforms/from_datasets.py (173:186)

duplicated block id: 11
size: 12 cleaned lines of code
in 2 files:
 - taskcluster/translations_taskgraph/actions/train.py (206:217)
 - taskcluster/translations_taskgraph/actions/train.py (226:237)

duplicated block id: 12
size: 12 cleaned lines of code
in 2 files:
 - taskcluster/translations_taskgraph/actions/train.py (226:237)
 - taskcluster/translations_taskgraph/actions/train.py (244:255)

duplicated block id: 13
size: 12 cleaned lines of code
in 2 files:
 - utils/model_registry.py (176:190)
 - utils/model_registry.py (231:245)

duplicated block id: 14
size: 12 cleaned lines of code
in 2 files:
 - taskcluster/translations_taskgraph/actions/train.py (206:217)
 - taskcluster/translations_taskgraph/actions/train.py (244:255)

duplicated block id: 15
size: 12 cleaned lines of code
in 2 files:
 - snakemake/profiles/slurm-csd3/config.yaml (5:16)
 - snakemake/profiles/slurm-moz/config.yaml (5:16)

duplicated block id: 16
size: 10 cleaned lines of code
in 2 files:
 - utils/tb_log_parser.py (39:48)
 - utils/tb_log_parser.py (137:146)

duplicated block id: 17
size: 10 cleaned lines of code
in 2 files:
 - site/model-registry/model-registry.mjs (470:480)
 - site/model-registry/model-registry.mjs (607:617)

duplicated block id: 18
size: 10 cleaned lines of code
in 2 files:
 - tracking/translations_parser/utils.py (32:41)
 - tracking/translations_parser/utils.py (51:60)

duplicated block id: 19
size: 10 cleaned lines of code
in 2 files:
 - inference/scripts/build-wasm.py (41:50)
 - inference/scripts/test-wasm.py (39:48)

duplicated block id: 20
size: 9 cleaned lines of code
in 2 files:
 - tracking/translations_parser/cli/taskcluster.py (60:68)
 - tracking/translations_parser/cli/taskcluster_group.py (51:59)

duplicated block id: 21
size: 9 cleaned lines of code
in 2 files:
 - snakemake/profiles/local-container/config.yaml (9:18)
 - snakemake/profiles/local/config.yaml (7:20)

duplicated block id: 22
size: 9 cleaned lines of code
in 2 files:
 - snakemake/profiles/slurm-csd3/status.py (61:69)
 - snakemake/profiles/slurm-moz/status.py (31:39)

duplicated block id: 23
size: 9 cleaned lines of code
in 2 files:
 - snakemake/profiles/slurm-csd3/status.py (40:49)
 - snakemake/profiles/slurm-moz/status.py (19:28)

duplicated block id: 24
size: 8 cleaned lines of code
in 2 files:
 - pipeline/alignments/align.py (368:377)
 - pipeline/data/parallel_importer.py (229:238)

duplicated block id: 25
size: 8 cleaned lines of code
in 2 files:
 - taskcluster/translations_taskgraph/transforms/find_upstreams.py (112:123)
 - taskcluster/translations_taskgraph/transforms/find_upstreams.py (160:171)

duplicated block id: 26
size: 7 cleaned lines of code
in 2 files:
 - pipeline/train/train.py (398:408)
 - pipeline/translate/translate.py (83:93)

duplicated block id: 27
size: 7 cleaned lines of code
in 2 files:
 - snakemake/profiles/slurm-csd3/status.py (9:18)
 - snakemake/profiles/slurm-moz/status.py (9:18)

duplicated block id: 28
size: 7 cleaned lines of code
in 2 files:
 - utils/model_registry.py (903:910)
 - utils/model_registry.py (986:993)

duplicated block id: 29
size: 7 cleaned lines of code
in 2 files:
 - pipeline/clean/merge-mono.py (201:207)
 - pipeline/clean/merge-parallel.py (243:251)

duplicated block id: 30
size: 7 cleaned lines of code
in 2 files:
 - taskcluster/translations_taskgraph/transforms/cast_to.py (19:28)
 - taskcluster/translations_taskgraph/transforms/marian_args.py (22:31)

duplicated block id: 31
size: 7 cleaned lines of code
in 2 files:
 - site/model-registry/model-registry.mjs (716:722)
 - site/model-registry/model-registry.mjs (751:757)

duplicated block id: 32
size: 7 cleaned lines of code
in 2 files:
 - site/model-registry/model-registry.mjs (708:714)
 - site/model-registry/model-registry.mjs (777:783)

duplicated block id: 33
size: 7 cleaned lines of code
in 2 files:
 - site/model-registry/model-registry.mjs (725:731)
 - site/model-registry/model-registry.mjs (760:766)

duplicated block id: 34
size: 7 cleaned lines of code
in 2 files:
 - site/model-registry/model-registry.mjs (697:703)
 - site/model-registry/model-registry.mjs (785:791)

duplicated block id: 35
size: 7 cleaned lines of code
in 2 files:
 - taskcluster/translations_taskgraph/actions/train.py (75:85)
 - taskcluster/translations_taskgraph/actions/train.py (393:403)

duplicated block id: 36
size: 7 cleaned lines of code
in 2 files:
 - taskcluster/translations_taskgraph/actions/train.py (75:85)
 - taskcluster/translations_taskgraph/actions/train.py (405:414)

duplicated block id: 37
size: 7 cleaned lines of code
in 2 files:
 - taskcluster/translations_taskgraph/transforms/cached_tasks.py (36:45)
 - taskcluster/translations_taskgraph/transforms/cast_to.py (19:28)

duplicated block id: 38
size: 7 cleaned lines of code
in 2 files:
 - taskcluster/translations_taskgraph/transforms/cached_tasks.py (36:45)
 - taskcluster/translations_taskgraph/transforms/marian_args.py (22:31)

duplicated block id: 39
size: 7 cleaned lines of code
in 2 files:
 - pipeline/train/train.py (429:435)
 - pipeline/translate/translate.py (112:118)

duplicated block id: 40
size: 7 cleaned lines of code
in 2 files:
 - pipeline/train/train.py (440:446)
 - pipeline/translate/translate.py (117:123)

duplicated block id: 41
size: 7 cleaned lines of code
in 2 files:
 - site/model-registry/model-registry.mjs (697:703)
 - site/model-registry/model-registry.mjs (716:722)

duplicated block id: 42
size: 7 cleaned lines of code
in 2 files:
 - taskcluster/translations_taskgraph/actions/train.py (393:403)
 - taskcluster/translations_taskgraph/actions/train.py (405:414)

duplicated block id: 43
size: 7 cleaned lines of code
in 2 files:
 - site/model-registry/model-registry.mjs (461:467)
 - site/model-registry/model-registry.mjs (598:604)

duplicated block id: 44
size: 7 cleaned lines of code
in 2 files:
 - site/model-registry/model-registry.mjs (716:722)
 - site/model-registry/model-registry.mjs (785:791)

duplicated block id: 45
size: 7 cleaned lines of code
in 2 files:
 - site/model-registry/model-registry.mjs (751:757)
 - site/model-registry/model-registry.mjs (785:791)

duplicated block id: 46
size: 7 cleaned lines of code
in 2 files:
 - site/model-registry/model-registry.mjs (733:739)
 - site/model-registry/model-registry.mjs (768:774)

duplicated block id: 47
size: 6 cleaned lines of code
in 2 files:
 - snakemake/profiles/local-container/config.yaml (1:6)
 - snakemake/profiles/local/config.yaml (1:6)

duplicated block id: 48
size: 6 cleaned lines of code
in 2 files:
 - utils/find_corpus.py (192:198)
 - utils/find_corpus.py (285:292)

duplicated block id: 49
size: 6 cleaned lines of code
in 2 files:
 - tracking/translations_parser/cli/experiments.py (86:91)
 - tracking/translations_parser/cli/taskcluster_group.py (89:94)

duplicated block id: 50
size: 6 cleaned lines of code
in 2 files:
 - pipeline/train/train.py (474:479)
 - pipeline/translate/translate.py (135:140)

duplicated block id: 51
size: 6 cleaned lines of code
in 2 files:
 - utils/find_corpus.py (233:239)
 - utils/find_corpus.py (285:292)

duplicated block id: 52
size: 6 cleaned lines of code
in 2 files:
 - taskcluster/translations_taskgraph/transforms/find_upstreams.py (188:194)
 - taskcluster/translations_taskgraph/transforms/from_datasets.py (171:176)

duplicated block id: 53
size: 6 cleaned lines of code
in 2 files:
 - pipeline/clean/merge-parallel.py (227:233)
 - pipeline/train/train.py (401:408)

duplicated block id: 54
size: 6 cleaned lines of code
in 2 files:
 - pipeline/clean/merge-parallel.py (227:233)
 - pipeline/translate/translate.py (86:93)

duplicated block id: 55
size: 6 cleaned lines of code
in 2 files:
 - site/firefox-models/index.html (72:77)
 - site/firefox-models/index.html (79:84)

duplicated block id: 56
size: 6 cleaned lines of code
in 2 files:
 - pipeline/translate/extract_best.py (49:55)
 - pipeline/translate/extract_best.py (101:107)

duplicated block id: 57
size: 6 cleaned lines of code
in 2 files:
 - taskcluster/scripts/pipeline/train_taskcluster.py (14:19)
 - taskcluster/translations_taskgraph/transforms/training_continuation.py (7:12)

duplicated block id: 58
size: 6 cleaned lines of code
in 2 files:
 - pipeline/translate/translate.py (17:22)
 - pipeline/translate/translate_ctranslate2.py (20:25)

duplicated block id: 59
size: 6 cleaned lines of code
in 2 files:
 - taskcluster/translations_taskgraph/transforms/find_upstreams.py (31:36)
 - taskcluster/translations_taskgraph/transforms/find_upstreams.py (46:51)

duplicated block id: 60
size: 6 cleaned lines of code
in 2 files:
 - taskcluster/translations_taskgraph/transforms/from_datasets.py (77:82)
 - taskcluster/translations_taskgraph/transforms/from_datasets.py (130:135)

duplicated block id: 61
size: 6 cleaned lines of code
in 2 files:
 - taskcluster/translations_taskgraph/transforms/from_datasets.py (84:89)
 - taskcluster/translations_taskgraph/transforms/from_datasets.py (145:150)

duplicated block id: 62
size: 6 cleaned lines of code
in 2 files:
 - pipeline/translate/translate.py (86:93)
 - utils/local_remote_settings/__main__.py (548:555)

duplicated block id: 63
size: 6 cleaned lines of code
in 2 files:
 - utils/model_registry.py (963:968)
 - utils/model_registry.py (1000:1005)

duplicated block id: 64
size: 6 cleaned lines of code
in 2 files:
 - utils/find_corpus.py (192:198)
 - utils/find_corpus.py (233:239)

duplicated block id: 65
size: 6 cleaned lines of code
in 2 files:
 - utils/preflight_check.py (532:537)
 - utils/preflight_check.py (544:549)

duplicated block id: 66
size: 6 cleaned lines of code
in 2 files:
 - tracking/translations_parser/data.py (104:109)
 - tracking/translations_parser/data.py (133:138)

duplicated block id: 67
size: 6 cleaned lines of code
in 2 files:
 - pipeline/clean/merge-parallel.py (227:233)
 - utils/local_remote_settings/__main__.py (548:555)

duplicated block id: 68
size: 6 cleaned lines of code
in 2 files:
 - pipeline/train/train.py (401:408)
 - utils/local_remote_settings/__main__.py (548:555)