G

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

15% duplication:

84,546 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
13,145 duplicated lines

5,011 duplicates

Duplication per Extension

Duplication per Component (primary)

Duplication Between Components (50+ lines)

Download: SVG DOT (open online Graphviz editor)

From Component --> To Component	Duplicated Lines	File Pairs	Details
parlai/agents (1%) --> projects/light_whoami (5%)	290	6 file pairs	details...
parlai/core (<1%) --> parlai/tasks (<1%)	202	11 file pairs	details...
parlai/agents (<1%) --> parlai/core (<1%)	176	10 file pairs	details...
parlai/core (<1%) --> projects/light_whoami (2%)	169	5 file pairs	details...
parlai/agents (<1%) --> projects/blenderbot2 (5%)	156	6 file pairs	details...
parlai/agents (<1%) --> parlai/tasks (<1%)	132	2 file pairs	details...
parlai/core (<1%) --> projects/style_gen (15%)	86	2 file pairs	details...
parlai/crowdsourcing (<1%) --> projects/safety_recipes (27%)	70	7 file pairs	details...
projects/blenderbot2 (2%) --> projects/msc (9%)	62	2 file pairs	details...
parlai/agents (<1%) --> projects/style_gen (10%)	61	2 file pairs	details...
parlai/chat_service (1%) --> parlai/scripts (<1%)	56	1 file pair	details...
parlai/tasks (<1%) --> projects/light_whoami (<1%)	54	4 file pairs	details...
parlai/tasks (<1%) --> projects/wizard_of_wikipedia (3%)	54	4 file pairs	details...
parlai/agents (<1%) --> projects/params_vs_compute (18%)	52	1 file pair	details...

Open 3D force graph...

Show more details on duplication between components...

Longest Duplicates

The list of 20 longest duplicates.

See data for all 5,011 duplicates...

Size	#	Folders	Files	Lines	Code
261	x 2	website/static/css website/static/css	home-theme.css parlai.css	544:804 (32%) 673:933 (28%)	view
174	x 2	website/static/css website/static/css	home-theme.css parlai.css	8:181 (21%) 92:265 (19%)	view
141	x 2	parlai/crowdsourcing/pro...val/frontend/components parlai/crowdsourcing/tas...hat/frontend/components	chat_app_with_onboarding.jsx chat_app_with_onboarding.jsx	22:177 (69%) 22:177 (69%)	view
95	x 2	website/static/css website/static/css	home-theme.css parlai.css	260:354 (11%) 344:438 (10%)	view
90	x 2	website/static/css website/static/css	home-theme.css parlai.css	345:434 (11%) 446:535 (9%)	view
75	x 2	website/static/css website/static/css	home-theme.css parlai.css	183:257 (9%) 267:341 (8%)	view
71	x 2	website/static/css website/static/css	home-theme.css parlai.css	456:526 (8%) 583:653 (7%)	view
67	x 2	parlai/crowdsourcing/pro...val/frontend/components parlai/crowdsourcing/tas...hat/frontend/components	response_panes.jsx response_panes.jsx	125:201 (39%) 181:257 (31%)	view
60	x 2	parlai/tasks/light_dialog parlai/tasks/light_dialog_wild	builder.py builder.py	16:86 (18%) 131:201 (9%)	view
60	x 2	parlai/agents/ir_baseline parlai/tasks/wizard_of_wikipedia	ir_baseline.py agents.py	89:148 (28%) 961:1020 (6%)	view
48	x 2	parlai/tasks/light_dialog_wild parlai/tasks/light_dialog_wild	agents.py agents.py	209:257 (11%) 313:361 (11%)	view
45	x 2	parlai/tasks/multiwoz_v20 parlai/tasks/multiwoz_v21	agents.py agents.py	50:101 (80%) 50:101 (80%)	view
42	x 2	parlai/tasks/light_dialog parlai/tasks/light_dialog_wild	builder.py builder.py	160:201 (12%) 270:311 (6%)	view
36	x 2	parlai/crowdsourcing/pro...val/frontend/components parlai/crowdsourcing/tas...hat/frontend/components	chat_app_with_onboarding.jsx chat_app_with_onboarding.jsx	189:230 (17%) 189:230 (17%)	view
35	x 2	parlai/tasks/light_dialog parlai/tasks/light_dialog_wild	agents.py agents.py	238:272 (14%) 283:317 (8%)	view
35	x 2	parlai/tasks/light_dialog parlai/tasks/light_dialog_wild	agents.py agents.py	171:205 (14%) 179:213 (8%)	view
34	x 2	parlai/tasks/light_dialog parlai/tasks/light_dialog_wild	agents.py agents.py	82:115 (13%) 56:89 (8%)	view
34	x 2	parlai/chat_service/core parlai/chat_service/services/websocket	chat_service_manager.py websocket_manager.py	698:737 (8%) 97:136 (21%)	view
31	x 2	parlai/crowdsourcing/pro...val/frontend/components parlai/crowdsourcing/tas...hat/frontend/components	message.jsx message.jsx	150:180 (20%) 53:83 (41%)	view
30	x 2	parlai/agents/transformer/modules projects/light_whoami/agents	decoder.py expanded_attention.py	113:148 (10%) 826:870 (3%)	view

Duplicated Units

The list of top 20 duplicated units.

See data for all 43 unit duplicates...

Size	#	Folders	Files	Lines	Code
28	x 2	parlai/tasks/light_dialog parlai/tasks/light_dialog_wild	builder.py builder.py	0:0 0:0	view
23	x 2	parlai/tasks/multiwoz_v20 parlai/tasks/multiwoz_v21	agents.py agents.py	0:0 0:0	view
20	x 2	parlai/tasks/light_dialog parlai/tasks/light_dialog_wild	builder.py builder.py	0:0 0:0	view
16	x 2	projects/blenderbot2/agents projects/blenderbot2/agents	modules.py modules.py	0:0 0:0	view
15	x 2	parlai/tasks/light_dialog parlai/tasks/light_dialog_wild	builder.py builder.py	0:0 0:0	view
23	x 2	parlai/tasks/vqa_v1 parlai/tasks/flickr30k	agents.py agents.py	0:0 0:0	view
13	x 2	parlai/tasks/light_dialog parlai/tasks/light_dialog_wild	builder.py builder.py	0:0 0:0	view
12	x 2	parlai/tasks/squad2 parlai/tasks/squad2	agents.py agents.py	0:0 0:0	view
15	x 2	parlai/tasks/casino parlai/tasks/dealnodeal	agents.py agents.py	0:0 0:0	view
12	x 2	parlai/agents/bert_ranker parlai/agents/bert_ranker	bi_encoder_ranker.py cross_encoder_ranker.py	0:0 0:0	view
10	x 2	parlai/tasks/squad2 parlai/tasks/squad	agents.py agents.py	0:0 0:0	view
10	x 2	parlai/tasks/taskmaster parlai/tasks/taskmaster2	agents.py agents.py	0:0 0:0	view
10	x 2	parlai/tasks/taskmaster parlai/tasks/taskmaster2	agents.py agents.py	0:0 0:0	view
10	x 2	parlai/tasks/multiwoz_v20 parlai/tasks/multiwoz_v21	agents.py agents.py	0:0 0:0	view
12	x 2	projects/style_gen parlai/core	classifier.py torch_classifier_agent.py	0:0 0:0	view
12	x 2	projects/light_whoami/agents parlai/core	poly_return_weights.py torch_generator_agent.py	0:0 0:0	view
13	x 2	parlai/tasks/qacnn parlai/tasks/qadailymail	build.py build.py	0:0 0:0	view
9	x 2	parlai/tasks/squad2 parlai/tasks/squad2	agents.py agents.py	0:0 0:0	view
9	x 2	parlai/tasks/squad parlai/tasks/squad	agents.py agents.py	0:0 0:0	view
8	x 2	projects/safety_bench/model_wrappers projects/safety_bench/model_wrappers	gpt_wrappers.py gpt_wrappers.py	0:0 0:0	view