Duplication · guardian/oz-2023-facebook-political-ads-scraper

Duplication Overall

70% duplication:

7,099 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
4,970 duplicated lines

11,545 duplicates

dependency graphs: 2D graph | 2D graph (with duplicates)... | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Duplication Between Components (50+ lines)

flowchart TB
    n0["analysis"]
    n1["ROOT"]
    n0 ---|"373"| n1
    classDef default fill:#00b2ee,stroke:#ffffff,color:#000000;
    classDef highlighted fill:#00b2ee,stroke:#ffffff,color:#000000;
    linkStyle 0 stroke:#DC143CFF,stroke-width:10px

Download: Mermaid (.mmd) (open online Mermaid editor)

2D force graph | 3D force graph

Show more details on duplication between components...

From Component --> To Component	Duplicated Lines	File Pairs	Details
analysis (24%) --> ROOT (23%)	373	2 file pairs	details...

Longest Duplicates

The list of 50 longest duplicates.

See data for all 11,545 duplicates...

Size	#	Folders	Files	Lines	Code
353	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	1775:2127 (6%) 1818:2170 (6%)	view
310	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	1775:2084 (5%) 1861:2170 (5%)	view
267	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	1775:2041 (5%) 1904:2170 (5%)	view
224	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	1775:1998 (4%) 1947:2170 (4%)	view
181	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	1775:1955 (3%) 1990:2170 (3%)	view
138	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	1775:1912 (2%) 2033:2170 (2%)	view
138	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	3107:3244 (2%) 3150:3287 (2%)	view
138	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	4414:4551 (2%) 4457:4594 (2%)	view
130	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	1644:1773 (2%) 1817:1946 (2%)	view
130	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	1644:1773 (2%) 1860:1989 (2%)	view
130	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	1644:1773 (2%) 1903:2032 (2%)	view
130	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	1644:1773 (2%) 1946:2075 (2%)	view
130	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	1644:1773 (2%) 1989:2118 (2%)	view
130	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	1644:1773 (2%) 2032:2161 (2%)	view
129	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	1645:1773 (2%) 1775:1903 (2%)	view
129	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	2756:2884 (2%) 2799:2927 (2%)	view
96	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	1644:1739 (1%) 2075:2170 (1%)	view
95	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	1775:1869 (1%) 2076:2170 (1%)	view
95	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	3107:3201 (1%) 3193:3287 (1%)	view
95	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	4414:4508 (1%) 4500:4594 (1%)	view
87	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	1644:1730 (1%) 1687:1773 (1%)	view
86	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	1688:1773 (1%) 1775:1860 (1%)	view
86	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	2756:2841 (1%) 2842:2927 (1%)	view
62	x 2	ROOT ROOT	ad-scraper.py political-ads.py	10:94 (42%) 10:94 (41%)	view
62	x 2	analysis ROOT	classify.py classify.py	142:207 (26%) 127:192 (25%)	view
61	x 2	ROOT ROOT	ad-scraper.py voice-ads.py	10:86 (41%) 10:85 (42%)	view
61	x 2	ROOT ROOT	political-ads.py voice-ads.py	10:86 (40%) 10:85 (42%)	view
53	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	1644:1696 (1%) 2118:2170 (1%)	view
52	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	1775:1826 (<1%) 2119:2170 (<1%)	view
52	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	3107:3158 (<1%) 3236:3287 (<1%)	view
52	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	3497:3548 (<1%) 3540:3591 (<1%)	view
52	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	4414:4465 (<1%) 4543:4594 (<1%)	view
50	x 2	ROOT ROOT	ad-scraper.py political-ads.py	95:159 (34%) 96:160 (33%)	view
48	x 2	analysis ROOT	classify.py classify.py	54:140 (20%) 40:126 (20%)	view
45	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	1107:1151 (<1%) 1150:1194 (<1%)	view
45	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	4109:4153 (<1%) 4152:4196 (<1%)	view
44	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	709:752 (<1%) 752:795 (<1%)	view
44	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	1644:1687 (<1%) 1730:1773 (<1%)	view
43	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	1422:1464 (<1%) 1465:1507 (<1%)	view
43	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	1731:1773 (<1%) 1775:1817 (<1%)	view
43	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	2248:2290 (<1%) 2292:2334 (<1%)	view
43	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	2384:2426 (<1%) 2427:2469 (<1%)	view
43	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	2615:2657 (<1%) 2658:2700 (<1%)	view
43	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	2756:2798 (<1%) 2885:2927 (<1%)	view
43	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	3716:3758 (<1%) 3759:3801 (<1%)	view
43	x 2	classify_model/old classify_model/old	zero_shot.ipynb zero_shot.ipynb	4283:4325 (<1%) 4326:4368 (<1%)	view
32	x 2	classify_model/old classify_model/old	yes_no_zero_shot.ipynb zero_shot.ipynb	135:166 (19%) 5185:5216 (<1%)	view
31	x 2	ROOT ROOT	ad-scraper.py voice-ads.py	125:159 (21%) 123:158 (21%)	view
31	x 2	ROOT ROOT	political-ads.py voice-ads.py	126:160 (20%) 123:158 (21%)	view
29	x 2	classify_model/old classify_model/old	sent_sim.ipynb yes_no_zero_shot.ipynb	42:70 (21%) 36:64 (17%)	view

Most Frequent Duplicates

The list of 50 most frequently found duplicates.

Size	#	Folders	Files	Lines	Code
6	x 92	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1076:1081 (<1%) 1121:1126 (<1%) 1164:1169 (<1%) 1207:1212 (<1%) 1251:1256 (<1%) 1295:1300 (<1%) 1339:1344 (<1%) 1388:1393 (<1%) 1436:1441 (<1%) 1479:1484 (<1%) ...	view
6	x 92	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1090:1095 (<1%) 1135:1140 (<1%) 1178:1183 (<1%) 1221:1226 (<1%) 1265:1270 (<1%) 1309:1314 (<1%) 1353:1358 (<1%) 1402:1407 (<1%) 1450:1455 (<1%) 1493:1498 (<1%) ...	view
6	x 92	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1078:1083 (<1%) 1123:1128 (<1%) 1166:1171 (<1%) 1209:1214 (<1%) 1253:1258 (<1%) 1297:1302 (<1%) 1341:1346 (<1%) 1390:1395 (<1%) 1438:1443 (<1%) 1481:1486 (<1%) ...	view
6	x 92	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1091:1096 (<1%) 1136:1141 (<1%) 1179:1184 (<1%) 1222:1227 (<1%) 1266:1271 (<1%) 1310:1315 (<1%) 1354:1359 (<1%) 1403:1408 (<1%) 1451:1456 (<1%) 1494:1499 (<1%) ...	view
6	x 92	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1089:1094 (<1%) 1134:1139 (<1%) 1177:1182 (<1%) 1220:1225 (<1%) 1264:1269 (<1%) 1308:1313 (<1%) 1352:1357 (<1%) 1401:1406 (<1%) 1449:1454 (<1%) 1492:1497 (<1%) ...	view
6	x 92	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1077:1082 (<1%) 1122:1127 (<1%) 1165:1170 (<1%) 1208:1213 (<1%) 1252:1257 (<1%) 1296:1301 (<1%) 1340:1345 (<1%) 1389:1394 (<1%) 1437:1442 (<1%) 1480:1485 (<1%) ...	view
6	x 77	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1196:1201 (<1%) 1240:1245 (<1%) 1284:1289 (<1%) 1377:1382 (<1%) 1425:1430 (<1%) 1468:1473 (<1%) 1519:1524 (<1%) 1562:1567 (<1%) 1605:1610 (<1%) 1648:1653 (<1%) ...	view
6	x 77	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1195:1200 (<1%) 1239:1244 (<1%) 1283:1288 (<1%) 1376:1381 (<1%) 1424:1429 (<1%) 1467:1472 (<1%) 1518:1523 (<1%) 1561:1566 (<1%) 1604:1609 (<1%) 1647:1652 (<1%) ...	view
6	x 76	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1193:1198 (<1%) 1237:1242 (<1%) 1281:1286 (<1%) 1374:1379 (<1%) 1422:1427 (<1%) 1465:1470 (<1%) 1516:1521 (<1%) 1559:1564 (<1%) 1602:1607 (<1%) 1645:1650 (<1%) ...	view
6	x 76	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1194:1199 (<1%) 1238:1243 (<1%) 1282:1287 (<1%) 1375:1380 (<1%) 1423:1428 (<1%) 1466:1471 (<1%) 1517:1522 (<1%) 1560:1565 (<1%) 1603:1608 (<1%) 1646:1651 (<1%) ...	view
6	x 39	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1192:1197 (<1%) 1464:1469 (<1%) 1558:1563 (<1%) 1601:1606 (<1%) 1644:1649 (<1%) 1687:1692 (<1%) 1730:1735 (<1%) 1817:1822 (<1%) 1860:1865 (<1%) 1903:1908 (<1%) ...	view
6	x 17	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	328:333 (<1%) 373:378 (<1%) 416:421 (<1%) 460:465 (<1%) 503:508 (<1%) 548:553 (<1%) 592:597 (<1%) 638:643 (<1%) 681:686 (<1%) 724:729 (<1%) ...	view
6	x 17	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	342:347 (<1%) 387:392 (<1%) 430:435 (<1%) 474:479 (<1%) 517:522 (<1%) 562:567 (<1%) 606:611 (<1%) 652:657 (<1%) 695:700 (<1%) 738:743 (<1%) ...	view
6	x 17	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	341:346 (<1%) 386:391 (<1%) 429:434 (<1%) 473:478 (<1%) 516:521 (<1%) 561:566 (<1%) 605:610 (<1%) 651:656 (<1%) 694:699 (<1%) 737:742 (<1%) ...	view
6	x 17	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	343:348 (<1%) 388:393 (<1%) 431:436 (<1%) 475:480 (<1%) 518:523 (<1%) 563:568 (<1%) 607:612 (<1%) 653:658 (<1%) 696:701 (<1%) 739:744 (<1%) ...	view
6	x 17	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	329:334 (<1%) 374:379 (<1%) 417:422 (<1%) 461:466 (<1%) 504:509 (<1%) 549:554 (<1%) 593:598 (<1%) 639:644 (<1%) 682:687 (<1%) 725:730 (<1%) ...	view
6	x 17	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	330:335 (<1%) 375:380 (<1%) 418:423 (<1%) 462:467 (<1%) 505:510 (<1%) 550:555 (<1%) 594:599 (<1%) 640:645 (<1%) 683:688 (<1%) 726:731 (<1%) ...	view
6	x 15	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1063:1068 (<1%) 1108:1113 (<1%) 1151:1156 (<1%) 1326:1331 (<1%) 2478:2483 (<1%) 2616:2621 (<1%) 2659:2664 (<1%) 2973:2978 (<1%) 3717:3722 (<1%) 3760:3765 (<1%) ...	view
6	x 15	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1064:1069 (<1%) 1109:1114 (<1%) 1152:1157 (<1%) 1327:1332 (<1%) 2479:2484 (<1%) 2617:2622 (<1%) 2660:2665 (<1%) 2974:2979 (<1%) 3718:3723 (<1%) 3761:3766 (<1%) ...	view
6	x 15	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1062:1067 (<1%) 1107:1112 (<1%) 1150:1155 (<1%) 1325:1330 (<1%) 2477:2482 (<1%) 2615:2620 (<1%) 2658:2663 (<1%) 2972:2977 (<1%) 3716:3721 (<1%) 3759:3764 (<1%) ...	view
6	x 15	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1065:1070 (<1%) 1110:1115 (<1%) 1153:1158 (<1%) 1328:1333 (<1%) 2480:2485 (<1%) 2618:2623 (<1%) 2661:2666 (<1%) 2975:2980 (<1%) 3719:3724 (<1%) 3762:3767 (<1%) ...	view
6	x 12	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1671:1676 (<1%) 1714:1719 (<1%) 1757:1762 (<1%) 1801:1806 (<1%) 1844:1849 (<1%) 1887:1892 (<1%) 1930:1935 (<1%) 1973:1978 (<1%) 2016:2021 (<1%) 2059:2064 (<1%) ...	view
6	x 12	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1656:1661 (<1%) 1699:1704 (<1%) 1742:1747 (<1%) 1786:1791 (<1%) 1829:1834 (<1%) 1872:1877 (<1%) 1915:1920 (<1%) 1958:1963 (<1%) 2001:2006 (<1%) 2044:2049 (<1%) ...	view
6	x 12	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1653:1658 (<1%) 1696:1701 (<1%) 1739:1744 (<1%) 1783:1788 (<1%) 1826:1831 (<1%) 1869:1874 (<1%) 1912:1917 (<1%) 1955:1960 (<1%) 1998:2003 (<1%) 2041:2046 (<1%) ...	view
6	x 12	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1649:1654 (<1%) 1692:1697 (<1%) 1735:1740 (<1%) 1779:1784 (<1%) 1822:1827 (<1%) 1865:1870 (<1%) 1908:1913 (<1%) 1951:1956 (<1%) 1994:1999 (<1%) 2037:2042 (<1%) ...	view
6	x 12	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1650:1655 (<1%) 1693:1698 (<1%) 1736:1741 (<1%) 1780:1785 (<1%) 1823:1828 (<1%) 1866:1871 (<1%) 1909:1914 (<1%) 1952:1957 (<1%) 1995:2000 (<1%) 2038:2043 (<1%) ...	view
6	x 12	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1663:1668 (<1%) 1706:1711 (<1%) 1749:1754 (<1%) 1793:1798 (<1%) 1836:1841 (<1%) 1879:1884 (<1%) 1922:1927 (<1%) 1965:1970 (<1%) 2008:2013 (<1%) 2051:2056 (<1%) ...	view
6	x 12	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1669:1674 (<1%) 1712:1717 (<1%) 1755:1760 (<1%) 1799:1804 (<1%) 1842:1847 (<1%) 1885:1890 (<1%) 1928:1933 (<1%) 1971:1976 (<1%) 2014:2019 (<1%) 2057:2062 (<1%) ...	view
6	x 12	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1657:1662 (<1%) 1700:1705 (<1%) 1743:1748 (<1%) 1787:1792 (<1%) 1830:1835 (<1%) 1873:1878 (<1%) 1916:1921 (<1%) 1959:1964 (<1%) 2002:2007 (<1%) 2045:2050 (<1%) ...	view
6	x 12	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1665:1670 (<1%) 1708:1713 (<1%) 1751:1756 (<1%) 1795:1800 (<1%) 1838:1843 (<1%) 1881:1886 (<1%) 1924:1929 (<1%) 1967:1972 (<1%) 2010:2015 (<1%) 2053:2058 (<1%) ...	view
6	x 12	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1676:1681 (<1%) 1719:1724 (<1%) 1762:1767 (<1%) 1806:1811 (<1%) 1849:1854 (<1%) 1892:1897 (<1%) 1935:1940 (<1%) 1978:1983 (<1%) 2021:2026 (<1%) 2064:2069 (<1%) ...	view
6	x 12	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1681:1686 (<1%) 1724:1729 (<1%) 1767:1772 (<1%) 1811:1816 (<1%) 1854:1859 (<1%) 1897:1902 (<1%) 1940:1945 (<1%) 1983:1988 (<1%) 2026:2031 (<1%) 2069:2074 (<1%) ...	view
6	x 12	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1666:1671 (<1%) 1709:1714 (<1%) 1752:1757 (<1%) 1796:1801 (<1%) 1839:1844 (<1%) 1882:1887 (<1%) 1925:1930 (<1%) 1968:1973 (<1%) 2011:2016 (<1%) 2054:2059 (<1%) ...	view
6	x 12	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1667:1672 (<1%) 1710:1715 (<1%) 1753:1758 (<1%) 1797:1802 (<1%) 1840:1845 (<1%) 1883:1888 (<1%) 1926:1931 (<1%) 1969:1974 (<1%) 2012:2017 (<1%) 2055:2060 (<1%) ...	view
6	x 12	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1658:1663 (<1%) 1701:1706 (<1%) 1744:1749 (<1%) 1788:1793 (<1%) 1831:1836 (<1%) 1874:1879 (<1%) 1917:1922 (<1%) 1960:1965 (<1%) 2003:2008 (<1%) 2046:2051 (<1%) ...	view
6	x 12	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1664:1669 (<1%) 1707:1712 (<1%) 1750:1755 (<1%) 1794:1799 (<1%) 1837:1842 (<1%) 1880:1885 (<1%) 1923:1928 (<1%) 1966:1971 (<1%) 2009:2014 (<1%) 2052:2057 (<1%) ...	view
6	x 12	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1670:1675 (<1%) 1713:1718 (<1%) 1756:1761 (<1%) 1800:1805 (<1%) 1843:1848 (<1%) 1886:1891 (<1%) 1929:1934 (<1%) 1972:1977 (<1%) 2015:2020 (<1%) 2058:2063 (<1%) ...	view
6	x 12	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1651:1656 (<1%) 1694:1699 (<1%) 1737:1742 (<1%) 1781:1786 (<1%) 1824:1829 (<1%) 1867:1872 (<1%) 1910:1915 (<1%) 1953:1958 (<1%) 1996:2001 (<1%) 2039:2044 (<1%) ...	view
6	x 12	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1682:1687 (<1%) 1725:1730 (<1%) 1768:1773 (<1%) 1812:1817 (<1%) 1855:1860 (<1%) 1898:1903 (<1%) 1941:1946 (<1%) 1984:1989 (<1%) 2027:2032 (<1%) 2070:2075 (<1%) ...	view
6	x 12	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1652:1657 (<1%) 1695:1700 (<1%) 1738:1743 (<1%) 1782:1787 (<1%) 1825:1830 (<1%) 1868:1873 (<1%) 1911:1916 (<1%) 1954:1959 (<1%) 1997:2002 (<1%) 2040:2045 (<1%) ...	view
6	x 12	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1679:1684 (<1%) 1722:1727 (<1%) 1765:1770 (<1%) 1809:1814 (<1%) 1852:1857 (<1%) 1895:1900 (<1%) 1938:1943 (<1%) 1981:1986 (<1%) 2024:2029 (<1%) 2067:2072 (<1%) ...	view
6	x 12	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1668:1673 (<1%) 1711:1716 (<1%) 1754:1759 (<1%) 1798:1803 (<1%) 1841:1846 (<1%) 1884:1889 (<1%) 1927:1932 (<1%) 1970:1975 (<1%) 2013:2018 (<1%) 2056:2061 (<1%) ...	view
6	x 12	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1680:1685 (<1%) 1723:1728 (<1%) 1766:1771 (<1%) 1810:1815 (<1%) 1853:1858 (<1%) 1896:1901 (<1%) 1939:1944 (<1%) 1982:1987 (<1%) 2025:2030 (<1%) 2068:2073 (<1%) ...	view
6	x 12	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1675:1680 (<1%) 1718:1723 (<1%) 1761:1766 (<1%) 1805:1810 (<1%) 1848:1853 (<1%) 1891:1896 (<1%) 1934:1939 (<1%) 1977:1982 (<1%) 2020:2025 (<1%) 2063:2068 (<1%) ...	view
6	x 12	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1654:1659 (<1%) 1697:1702 (<1%) 1740:1745 (<1%) 1784:1789 (<1%) 1827:1832 (<1%) 1870:1875 (<1%) 1913:1918 (<1%) 1956:1961 (<1%) 1999:2004 (<1%) 2042:2047 (<1%) ...	view
6	x 12	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1678:1683 (<1%) 1721:1726 (<1%) 1764:1769 (<1%) 1808:1813 (<1%) 1851:1856 (<1%) 1894:1899 (<1%) 1937:1942 (<1%) 1980:1985 (<1%) 2023:2028 (<1%) 2066:2071 (<1%) ...	view
6	x 12	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1677:1682 (<1%) 1720:1725 (<1%) 1763:1768 (<1%) 1807:1812 (<1%) 1850:1855 (<1%) 1893:1898 (<1%) 1936:1941 (<1%) 1979:1984 (<1%) 2022:2027 (<1%) 2065:2070 (<1%) ...	view
6	x 12	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1655:1660 (<1%) 1698:1703 (<1%) 1741:1746 (<1%) 1785:1790 (<1%) 1828:1833 (<1%) 1871:1876 (<1%) 1914:1919 (<1%) 1957:1962 (<1%) 2000:2005 (<1%) 2043:2048 (<1%) ...	view
6	x 12	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1662:1667 (<1%) 1705:1710 (<1%) 1748:1753 (<1%) 1792:1797 (<1%) 1835:1840 (<1%) 1878:1883 (<1%) 1921:1926 (<1%) 1964:1969 (<1%) 2007:2012 (<1%) 2050:2055 (<1%) ...	view
6	x 11	classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old classify_model/old ...	zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb zero_shot.ipynb ...	1686:1691 (<1%) 1729:1734 (<1%) 1816:1821 (<1%) 1859:1864 (<1%) 1902:1907 (<1%) 1945:1950 (<1%) 1988:1993 (<1%) 2031:2036 (<1%) 2074:2079 (<1%) 2117:2122 (<1%) ...	view

Duplicated Units

The list of top 6 duplicated units.

See data for all 6 unit duplicates...

Size	#	Folders	Files	Lines	Code
23	x 3	ROOT ROOT ROOT	ad-scraper.py political-ads.py voice-ads.py	20:46 20:46 19:45	view
29	x 2	ROOT ROOT	ad-scraper.py political-ads.py	167:207 168:208	view
12	x 2	analysis ROOT	classify.py classify.py	111:133 97:119	view
7	x 3	ROOT ROOT ROOT	ad-scraper.py political-ads.py voice-ads.py	72:80 72:80 71:79	view
7	x 2	analysis analysis	allAnalysis.py classifyAds.py	16:24 39:47	view
8	x 2	analysis ROOT	classify.py classify.py	94:108 80:94	view

About This Analysis

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

To learn more about duplications and techniques for eliminating duplication, Sokrates recommends the following resources:

Avoid Repetition, MartinFlower.com (IEEE Software article)
Refactoring duplicated code, sourcemaking.com
Beck Design Rules, MartinFowler.com
DRY (Don't Repeat Yourself) Principle, Wikipedia
The Cut-and-Paste Programming Software Development Anti-Pattern, sourcemaking.com
Code Smells / Duplicated Code, Jeff Atwood, codinghorror.com:

"Duplicated code is the bane of software development. Stamp out duplication whenever possible. You should always be on the lookout for more subtle cases of near-duplication, too. Don't Repeat Yourself!"

Reducing Coupling, MartinFlower.com (IEEE Software article):

"There are several ways to describe coupling, but it boils down to this: If changing one module in a program requires changing another module, then coupling exists. ... Duplication always implies coupling, because changing one piece of duplicate code implies changing the other."