mozilla / bigquery-etl

Intro

File size measurements show the distribution of size of files.
Files are classified in four categories based on their size (lines of code): 1-100 (very small files), 101-200 (small files), 201-500 (medium size files), 501-1000 (long files), 1001+(very long files).
It is a good practice to keep files small. Long files may become "bloaters", code that have increased to such gargantuan proportions that they are hard to work with.

Learn more...

File Size Overall

There are 4,708 files with 252,454 lines of code.

22 very long files (42,065 lines of code)
29 long files (19,120 lines of code)
169 medium size files (50,776 lines of codeclsfd_ftr_w_mp_ins)
324 small files (45,709 lines of code)
4,164 very small files (94,784 lines of code)

Legend:

1001+

501-1000

201-500

101-200

1-100

explore: grouped by folders | grouped by size | sunburst | 3D view

File Size per Extension

1001+

501-1000

201-500

101-200

1-100

File Size per Logical Decomposition

primary

1001+

501-1000

201-500

101-200

1-100

Longest Files (Top 50)

File	# lines	# units
templating.yaml in sql_generators/use_counters/templates	2584	-
schema.yaml in sql/moz-fx-data-shared-prod/telemetry_derived/clients_last_seen_v2	2542	-
schema.yaml in sql/moz-fx-data-shared-prod/telemetry_derived/clients_last_seen_v1	2397	-
schema.yaml in sql/moz-fx-data-shared-prod/telemetry_derived/clients_daily_joined_v1	2384	-
schema.yaml in sql/moz-fx-data-shared-prod/telemetry_derived/clients_first_seen_v1	2350	-
schema.yaml in sql/moz-fx-data-shared-prod/telemetry_derived/clients_daily_v6	2348	-
query.sql in sql/moz-fx-data-shared-prod/telemetry_derived/clients_daily_keyed_histogram_aggregates_v1	2283	-
query.sql in sql/glam-fenix-dev/glam_etl/org_mozilla_fenix__clients_daily_scalar_aggregates_metrics_v1	2187	-
query.sql in sql/glam-fenix-dev/glam_etl/org_mozilla_fenix_nightly__clients_daily_scalar_aggregates_metrics_v1	2187	-
query.sql in sql/glam-fenix-dev/glam_etl/org_mozilla_fennec_aurora__clients_daily_scalar_aggregates_metrics_v1	2187	-
dags.yaml in root	2168	-
schema.yaml in sql/moz-fx-data-shared-prod/telemetry_derived/clients_last_seen_joined_v1	2151	-
query.sql in sql/glam-fenix-dev/glam_etl/org_mozilla_fenix__clients_daily_scalar_aggregates_migration_v1	1973	-
query.py in bigquery_etl/cli	1961	30
query.sql in sql/moz-fx-data-shared-prod/telemetry_derived/clients_daily_v6	1675	-
schema.yaml in sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_auth_events_v1	1451	-
view.sql in sql/moz-fx-data-shared-prod/telemetry/rocket_android_events_v1	1406	-
schema.yaml in sql/moz-fx-data-shared-prod/firefox_accounts_derived/nonprod_fxa_gcp_stderr_events_v1	1361	-
schema.yaml in sql/moz-fx-data-shared-prod/firefox_accounts_derived/docker_fxa_customs_sanitized_v2	1137	-
templating.yaml in sql_generators/feature_usage	1136	-
schema.yaml in sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_gcp_stderr_events_v1	1134	-
schema.yaml in sql/moz-fx-data-shared-prod/sumo_ga_derived/ga3_events_v1	1063	-
monitoring.py in bigquery_etl/cli	917	11
query.sql in sql/moz-fx-data-shared-prod/fenix_derived/feature_usage_events_v1	871	-
config.py in bigquery_etl/shredder	852	16
aliases.yaml in sql_generators/country_code_lookup	828	-
shredder_mitigation.py in bigquery_etl/backfill	779	13
tokenizer.py in bigquery_etl/format_sql	739	4
schema.yaml in sql/moz-fx-data-shared-prod/firefox_accounts_derived/nonprod_fxa_auth_events_v1	727	-
query.sql in sql/moz-fx-data-shared-prod/telemetry_derived/clients_first_seen_v3	723	-
delete.py in bigquery_etl/shredder	710	14
backfill.py in bigquery_etl/cli	709	11
query.sql in sql/glam-fenix-dev/glam_etl/org_mozilla_fenix__clients_daily_histogram_aggregates_metrics_v1	692	-
query.sql in sql/glam-fenix-dev/glam_etl/org_mozilla_fenix_nightly__clients_daily_histogram_aggregates_metrics_v1	682	-
query.sql in sql/glam-fenix-dev/glam_etl/org_mozilla_fennec_aurora__clients_daily_histogram_aggregates_metrics_v1	682	-
schema.yaml in sql/moz-fx-data-shared-prod/sumo_ga_derived/ga4_events_v1	677	-
schema.yaml in sql/moz-fx-data-shared-prod/subscription_platform_derived/stripe_subscriptions_history_v2	622	-
query.sql in sql/moz-fx-data-shared-prod/telemetry_derived/newtab_visits_v1	615	-
query.sql in sql/moz-fx-data-shared-prod/fenix_derived/firefox_android_clients_v1	607	-
schema.yaml in sql/moz-fx-data-shared-prod/firefox_accounts_derived/docker_fxa_customs_sanitized_v1	596	-
query.sql in sql/moz-fx-data-shared-prod/telemetry_derived/clients_first_seen_v2	596	-
query.sql in sql/glam-fenix-dev/glam_etl/firefox_desktop__clients_daily_scalar_aggregates_metrics_v1	590	-
schema.yaml in sql/moz-fx-data-shared-prod/subscription_platform_derived/stripe_subscriptions_v2	580	-
query.sql in sql/moz-fx-data-shared-prod/search_derived/search_dau_aggregates_v1	574	-
query.sql in sql/moz-fx-data-shared-prod/telemetry_derived/clients_last_seen_v2	568	-
script.sql in sql/moz-fx-data-shared-prod/mozilla_org_derived/ga_sessions_v2	562	-
task.py in bigquery_etl/query_scheduling	545	31
schema.yaml in sql/moz-fx-data-shared-prod/fenix_derived/feature_usage_events_v1	538	-
query.sql in sql/moz-fx-data-shared-prod/telemetry_derived/clients_daily_histogram_aggregates_gpu_v1	515	-
query.sql in sql/moz-fx-data-shared-prod/internet_outages/global_outages_v2	513	-

Files With Most Units (Top 50)

File	# lines	# units
task.py in bigquery_etl/query_scheduling	545	31
query.py in bigquery_etl/cli	1961	30
parse_metadata.py in bigquery_etl/metadata	355	22
__init__.py in bigquery_etl/view	335	22
dag.py in bigquery_etl/query_scheduling	197	20
__init__.py in bigquery_etl/schema	313	20
parse_routine.py in bigquery_etl/routine	255	18
dryrun.py in bigquery_etl	474	18
config.py in bigquery_etl/shredder	852	16
utils.py in bigquery_etl/backfill	274	15
standard_args.py in bigquery_etl/util	148	15
common.py in sql_generators/glean_usage	403	14
delete.py in bigquery_etl/shredder	710	14
parse.py in bigquery_etl/backfill	155	14
shredder_mitigation.py in bigquery_etl/backfill	779	13
utils.py in bigquery_etl/cli	194	13
survey.py in bigquery_etl/alchemer	155	12
validate_metadata.py in bigquery_etl/metadata	349	12
dag_collection.py in bigquery_etl/query_scheduling	123	11
validate.py in bigquery_etl/backfill	95	11
view.py in bigquery_etl/cli	337	11
backfill.py in bigquery_etl/cli	709	11
monitoring.py in bigquery_etl/cli	917	11
query.py in sql/moz-fx-data-shared-prod/microsoft_derived/app_acquisitions_v1	197	11
query.py in sql/moz-fx-data-shared-prod/microsoft_derived/app_installs_v1	174	11
formatter.py in bigquery_etl/format_sql	249	10
routine.py in bigquery_etl/pytest_plugin	75	10
common.py in bigquery_etl/util	247	10
query.py in sql/moz-fx-data-shared-prod/microsoft_derived/app_conversions_v1	170	10
__init__.py in sql_generators/events_daily	111	9
utils.py in bigquery_etl/query_scheduling	66	9
routine.py in bigquery_etl/cli	331	9
publish_gcs_metadata.py in bigquery_etl/public_data	176	9
query.py in sql/moz-fx-data-shared-prod/firefox_ios_derived/app_store_choice_screen_selection_v1	190	9
query.py in sql/moz-fx-data-shared-prod/firefox_ios_derived/app_store_choice_screen_engagement_v1	182	9
clients_daily_scalar_aggregates_v1.sql.py in sql/moz-fx-data-shared-prod/telemetry_derived	261	9
cli.py in bigquery_etl/glam	138	8
dependency.py in bigquery_etl	207	8
formatters.py in bigquery_etl/query_scheduling	45	8
sql.py in bigquery_etl/pytest_plugin	173	8
check.py in bigquery_etl/cli	207	8
stage.py in bigquery_etl/cli	482	8
publish_json.py in bigquery_etl/public_data	253	8
query.py in sql/moz-fx-data-shared-prod/telemetry_dev_cycle_external/experiments_stats_v1	130	8
query.py in sql/moz-fx-data-shared-prod/monitoring_derived/shredder_targets_v1	251	8
conftest.py in root	64	7
models.py in bigquery_etl/glam	120	7
copy_deduplicate.py in bigquery_etl	327	7
query.py in sql/moz-fx-data-shared-prod/cloudflare_derived/device_usage_v1	346	7
query.py in sql/moz-fx-data-shared-prod/adjust_derived/adjust_deliverables_v1	220	7

Files With Long Lines (Top 50)

There are 238 files with lines longer than 120 characters. In total, there are 680 long lines.

File	# lines	# units	# long lines
udf.sql in sql/moz-fx-data-shared-prod/udf/main_summary_scalars	353	-	174
view.sql in sql/moz-fx-data-shared-prod/telemetry/first_shutdown_summary_v4	415	-	57
query.sql in sql/moz-fx-data-shared-prod/telemetry_derived/clients_daily_v6	1675	-	34
udf.sql in sql/moz-fx-data-shared-prod/udf_js/main_summary_active_addons	154	-	14
schema.yaml in sql/moz-fx-data-shared-prod/telemetry_derived/clients_first_seen_28_days_later_v3	261	-	12
schema.yaml in sql/moz-fx-data-shared-prod/telemetry_derived/clients_first_seen_v2	190	-	12
schema.yaml in sql/moz-fx-data-shared-prod/telemetry_derived/clients_first_seen_v3	259	-	12
schema.yaml in sql/moz-fx-data-shared-prod/monitoring_derived/jobs_by_organization_v1	192	-	8
schema.yaml in sql/moz-fx-data-shared-prod/mozilla_org_derived/gclid_conversions_v2	37	-	7
schema.yaml in sql/moz-fx-data-shared-prod/mozilla_org_derived/gclid_conversions_v3	37	-	7
monitor_mozilla_accounts_funnels.toml in sql_generators/funnels/configs	79	-	6
schema.yaml in sql/moz-fx-data-shared-prod/mozilla_org_derived/ga_sessions_v1	127	-	6
udf.sql in sql/moz-fx-data-shared-prod/udf_js/parse_sponsored_interaction	120	-	6
schema.yaml in sql/moz-fx-data-shared-prod/telemetry_derived/active_users_rollup_shredder_v1	57	-	6
monitor_dashboard_user_journey_funnels.toml in sql_generators/funnels/configs	43	-	5
metadata.yaml in sql/moz-fx-data-shared-prod/internet_outages/global_outages_v2	67	-	5
schema.yaml in sql/moz-fx-data-shared-prod/mozilla_org_derived/ga_sessions_v2	143	-	5
pwd_reset_funnels_by_service.toml in sql_generators/funnels/configs	68	-	4
usage_reporting_clients_daily_v1.query.sql.jinja in sql_generators/usage_reporting/templates	65	-	4
monitoring.py in bigquery_etl/cli	917	11	4
udf.sql in sql/mozfun/marketing/parse_campaign_name	265	-	4
stored_procedure.sql in sql/mozfun/utils/diff_query_schemas	58	-	4
schema.yaml in sql/moz-fx-data-shared-prod/mozilla_org_derived/gclid_conversions_v1	25	-	4
schema.yaml in sql/moz-fx-data-shared-prod/mozilla_org_derived/ga_clients_v1	96	-	4
schema.yaml in sql/moz-fx-data-shared-prod/mozilla_org_derived/ga_clients_v2	96	-	4
query.py in sql/moz-fx-data-shared-prod/cloudflare_derived/os_usage_v1	304	4	4
udf.sql in sql/moz-fx-data-shared-prod/udf_js/decode_uri_attribution	153	-	4
schema.yaml in sql/moz-fx-data-shared-prod/telemetry_derived/newtab_interactions_hourly_v1	56	-	4
schema.yaml in sql/moz-fx-data-shared-prod/mozilla_vpn_derived/site_metrics_summary_v2	85	-	4
desktop_checks.sql in sql_generators/active_users_aggregates_v3/templates	189	-	3
desktop_checks.sql in sql_generators/active_users_aggregates_v4/templates	189	-	3
metadata.yaml in sql_generators/urlbar_events/templates	27	-	3
dags.yaml in root	2168	-	3
schema.yaml in sql/moz-fx-data-shared-prod/internet_outages/global_outages_v2	96	-	3
view.sql in sql/moz-fx-data-shared-prod/monitoring/table_storage_trends	182	-	3
query.sql in sql/moz-fx-data-shared-prod/telemetry_derived/newtab_clients_daily_v1	369	-	3
funnel.sql in sql_generators/funnels/templates	169	-	2
reg_engagement_funnel.toml in sql_generators/funnels/configs	144	-	2
login_funnels_by_service.toml in sql_generators/funnels/configs	253	-	2
fenix_checks.sql in sql_generators/active_users_aggregates_v3/templates	154	-	2
mobile_checks.sql in sql_generators/active_users_aggregates_v3/templates	85	-	2
query.sql in sql_generators/events_daily/templates/event_types_history_v1	437	-	2
fenix_checks.sql in sql_generators/active_users_aggregates_v4/templates	154	-	2
mobile_checks.sql in sql_generators/active_users_aggregates_v4/templates	85	-	2
attribution_clients.query.sql in sql_generators/mobile_kpi_support_metrics/templates	298	-	2
usage_reporting_active_users_aggregates.view.sql.jinja in sql_generators/usage_reporting/templates	23	-	2
usage_reporting_active_users.view.sql.jinja in sql_generators/usage_reporting/templates	69	-	2
validate_metadata.py in bigquery_etl/metadata	349	12	2
atob.js in sql/mozfun/lib	40	4	2
query.sql in sql/moz-fx-data-shared-prod/search_terms_derived/suggest_impression_sanitized_v2	43	-	2