-
Notifications
You must be signed in to change notification settings - Fork 124
DSRE-1779 Add FxA monitoring query #6511
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
fdadaa4
Add FxA monitoring query
akkomar c29ea82
Add schema
akkomar d89e24a
Fix schema
akkomar 0c1ca59
Use time travel to avoid losing data during backfills
akkomar cb9478e
update time travel comment
akkomar f0f0abc
Update sql/moz-fx-data-shared-prod/accounts_backend_derived/monitorin…
akkomar 293280c
Update sql/moz-fx-data-shared-prod/accounts_backend_derived/monitorin…
akkomar 544d68c
Merge branch 'main' into wclouser_fxa_db_counts
akkomar 4d98bdc
s/@as_of_date,/@as_of_date+1,/g
akkomar File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
19 changes: 19 additions & 0 deletions
19
sql/moz-fx-data-shared-prod/accounts_backend_derived/monitoring_db_counts_v1/metadata.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| friendly_name: FxA DB Counts Monitoring | ||
| description: |- | ||
| Simple aggregation of counts of records in the FxA DB tables. | ||
| Enables to identify trends within accounts data. E.g. "How many | ||
| inactive accounts are there?" | ||
| owners: | ||
| - [email protected] | ||
| labels: | ||
| incremental: true | ||
| owner1: [email protected] | ||
| scheduling: | ||
| dag_name: bqetl_accounts_derived | ||
| date_partition_parameter: as_of_date | ||
| bigquery: | ||
| time_partitioning: | ||
| type: day | ||
| field: as_of_date | ||
| require_partition_filter: false | ||
| expiration_days: null |
300 changes: 300 additions & 0 deletions
300
sql/moz-fx-data-shared-prod/accounts_backend_derived/monitoring_db_counts_v1/query.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,300 @@ | ||
| WITH table_counts AS ( | ||
| SELECT | ||
| 'account_customers' AS table_name, | ||
| COUNT(*) AS total_rows | ||
| FROM | ||
| `moz-fx-data-shared-prod.accounts_db_external.fxa_account_customers_v1` FOR SYSTEM_TIME AS OF TIMESTAMP( | ||
| @as_of_date + 1, | ||
| 'UTC' | ||
| ) | ||
| UNION ALL | ||
| SELECT | ||
| 'account_groups' AS table_name, | ||
| COUNT(*) AS total_rows | ||
| FROM | ||
| `moz-fx-data-shared-prod.accounts_db_external.fxa_account_groups_v1` FOR SYSTEM_TIME AS OF TIMESTAMP( | ||
| @as_of_date + 1, | ||
| 'UTC' | ||
| ) | ||
| UNION ALL | ||
| SELECT | ||
| 'account_reset_tokens' AS table_name, | ||
| COUNT(*) AS total_rows | ||
| FROM | ||
| `moz-fx-data-shared-prod.accounts_db_external.fxa_account_reset_tokens_v1` FOR SYSTEM_TIME AS OF TIMESTAMP( | ||
| @as_of_date + 1, | ||
| 'UTC' | ||
| ) | ||
| UNION ALL | ||
| SELECT | ||
| 'accounts' AS table_name, | ||
| COUNT(*) AS total_rows | ||
| FROM | ||
| `moz-fx-data-shared-prod.accounts_db_external.fxa_accounts_v1` FOR SYSTEM_TIME AS OF TIMESTAMP( | ||
| @as_of_date + 1, | ||
| 'UTC' | ||
| ) | ||
| UNION ALL | ||
| SELECT | ||
| 'carts' AS table_name, | ||
| COUNT(*) AS total_rows | ||
| FROM | ||
| `moz-fx-data-shared-prod.accounts_db_external.fxa_carts_v1` FOR SYSTEM_TIME AS OF TIMESTAMP( | ||
| @as_of_date + 1, | ||
| 'UTC' | ||
| ) | ||
| UNION ALL | ||
| SELECT | ||
| 'device_commands' AS table_name, | ||
| COUNT(*) AS total_rows | ||
| FROM | ||
| `moz-fx-data-shared-prod.accounts_db_external.fxa_device_commands_v1` FOR SYSTEM_TIME AS OF TIMESTAMP( | ||
| @as_of_date + 1, | ||
| 'UTC' | ||
| ) | ||
| UNION ALL | ||
| SELECT | ||
| 'devices' AS table_name, | ||
| COUNT(*) AS total_rows | ||
| FROM | ||
| `moz-fx-data-shared-prod.accounts_db_external.fxa_devices_v1` FOR SYSTEM_TIME AS OF TIMESTAMP( | ||
| @as_of_date + 1, | ||
| 'UTC' | ||
| ) | ||
| UNION ALL | ||
| SELECT | ||
| 'email_bounces' AS table_name, | ||
| COUNT(*) AS total_rows | ||
| FROM | ||
| `moz-fx-data-shared-prod.accounts_db_external.fxa_email_bounces_v1` FOR SYSTEM_TIME AS OF TIMESTAMP( | ||
| @as_of_date + 1, | ||
| 'UTC' | ||
| ) | ||
| UNION ALL | ||
| SELECT | ||
| 'emails' AS table_name, | ||
| COUNT(*) AS total_rows | ||
| FROM | ||
| `moz-fx-data-shared-prod.accounts_db_external.fxa_emails_v1` FOR SYSTEM_TIME AS OF TIMESTAMP( | ||
| @as_of_date + 1, | ||
| 'UTC' | ||
| ) | ||
| UNION ALL | ||
| SELECT | ||
| 'linked_accounts' AS table_name, | ||
| COUNT(*) AS total_rows | ||
| FROM | ||
| `moz-fx-data-shared-prod.accounts_db_external.fxa_linked_accounts_v1` FOR SYSTEM_TIME AS OF TIMESTAMP( | ||
| @as_of_date + 1, | ||
| 'UTC' | ||
| ) | ||
| UNION ALL | ||
| SELECT | ||
| 'oauth_codes' AS table_name, | ||
| COUNT(*) AS total_rows | ||
| FROM | ||
| `moz-fx-data-shared-prod.accounts_db_external.fxa_oauth_codes_v1` FOR SYSTEM_TIME AS OF TIMESTAMP( | ||
| @as_of_date + 1, | ||
| 'UTC' | ||
| ) | ||
| UNION ALL | ||
| SELECT | ||
| 'oauth_refresh_tokens' AS table_name, | ||
| COUNT(*) AS total_rows | ||
| FROM | ||
| `moz-fx-data-shared-prod.accounts_db_external.fxa_oauth_refresh_tokens_v1` FOR SYSTEM_TIME AS OF TIMESTAMP( | ||
| @as_of_date + 1, | ||
| 'UTC' | ||
| ) | ||
| UNION ALL | ||
| SELECT | ||
| 'oauth_tokens' AS table_name, | ||
| COUNT(*) AS total_rows | ||
| FROM | ||
| `moz-fx-data-shared-prod.accounts_db_external.fxa_oauth_tokens_v1` FOR SYSTEM_TIME AS OF TIMESTAMP( | ||
| @as_of_date + 1, | ||
| 'UTC' | ||
| ) | ||
| UNION ALL | ||
| SELECT | ||
| 'password_change_tokens' AS table_name, | ||
| COUNT(*) AS total_rows | ||
| FROM | ||
| `moz-fx-data-shared-prod.accounts_db_external.fxa_password_change_tokens_v1` FOR SYSTEM_TIME AS OF TIMESTAMP( | ||
| @as_of_date + 1, | ||
| 'UTC' | ||
| ) | ||
| UNION ALL | ||
| SELECT | ||
| 'password_forgot_tokens' AS table_name, | ||
| COUNT(*) AS total_rows | ||
| FROM | ||
| `moz-fx-data-shared-prod.accounts_db_external.fxa_password_forgot_tokens_v1` FOR SYSTEM_TIME AS OF TIMESTAMP( | ||
| @as_of_date + 1, | ||
| 'UTC' | ||
| ) | ||
| UNION ALL | ||
| SELECT | ||
| 'paypal_customers' AS table_name, | ||
| COUNT(*) AS total_rows | ||
| FROM | ||
| `moz-fx-data-shared-prod.accounts_db_external.fxa_paypal_customers_v1` FOR SYSTEM_TIME AS OF TIMESTAMP( | ||
| @as_of_date + 1, | ||
| 'UTC' | ||
| ) | ||
| UNION ALL | ||
| SELECT | ||
| 'recovery_codes' AS table_name, | ||
| COUNT(*) AS total_rows | ||
| FROM | ||
| `moz-fx-data-shared-prod.accounts_db_external.fxa_recovery_codes_v1` FOR SYSTEM_TIME AS OF TIMESTAMP( | ||
| @as_of_date + 1, | ||
| 'UTC' | ||
| ) | ||
| UNION ALL | ||
| SELECT | ||
| 'security_events' AS table_name, | ||
| COUNT(*) AS total_rows | ||
| FROM | ||
| `moz-fx-data-shared-prod.accounts_db_external.fxa_security_events_v1` FOR SYSTEM_TIME AS OF TIMESTAMP( | ||
| @as_of_date + 1, | ||
| 'UTC' | ||
| ) | ||
| UNION ALL | ||
| SELECT | ||
| 'sent_emails' AS table_name, | ||
| COUNT(*) AS total_rows | ||
| FROM | ||
| `moz-fx-data-shared-prod.accounts_db_external.fxa_sent_emails_v1` FOR SYSTEM_TIME AS OF TIMESTAMP( | ||
| @as_of_date + 1, | ||
| 'UTC' | ||
| ) | ||
| UNION ALL | ||
| SELECT | ||
| 'session_tokens' AS table_name, | ||
| COUNT(*) AS total_rows | ||
| FROM | ||
| `moz-fx-data-shared-prod.accounts_db_external.fxa_session_tokens_v1` FOR SYSTEM_TIME AS OF TIMESTAMP( | ||
| @as_of_date + 1, | ||
| 'UTC' | ||
| ) | ||
| UNION ALL | ||
| SELECT | ||
| 'signin_codes' AS table_name, | ||
| COUNT(*) AS total_rows | ||
| FROM | ||
| `moz-fx-data-shared-prod.accounts_db_external.fxa_signin_codes_v1` FOR SYSTEM_TIME AS OF TIMESTAMP( | ||
| @as_of_date + 1, | ||
| 'UTC' | ||
| ) | ||
| UNION ALL | ||
| SELECT | ||
| 'totp' AS table_name, | ||
| COUNT(*) AS total_rows | ||
| FROM | ||
| `moz-fx-data-shared-prod.accounts_db_external.fxa_totp_v1` FOR SYSTEM_TIME AS OF TIMESTAMP( | ||
| @as_of_date + 1, | ||
| 'UTC' | ||
| ) | ||
| UNION ALL | ||
| SELECT | ||
| 'unblock_codes' AS table_name, | ||
| COUNT(*) AS total_rows | ||
| FROM | ||
| `moz-fx-data-shared-prod.accounts_db_external.fxa_unblock_codes_v1` FOR SYSTEM_TIME AS OF TIMESTAMP( | ||
| @as_of_date + 1, | ||
| 'UTC' | ||
| ) | ||
| UNION ALL | ||
| SELECT | ||
| 'unverified_tokens' AS table_name, | ||
| COUNT(*) AS total_rows | ||
| FROM | ||
| `moz-fx-data-shared-prod.accounts_db_external.fxa_unverified_tokens_v1` FOR SYSTEM_TIME AS OF TIMESTAMP( | ||
| @as_of_date + 1, | ||
| 'UTC' | ||
| ) | ||
| UNION ALL | ||
| SELECT | ||
| 'verification_reminders' AS table_name, | ||
| COUNT(*) AS total_rows | ||
| FROM | ||
| `moz-fx-data-shared-prod.accounts_db_external.fxa_verification_reminders_v1` FOR SYSTEM_TIME AS OF TIMESTAMP( | ||
| @as_of_date + 1, | ||
| 'UTC' | ||
| ) | ||
| UNION ALL | ||
| ( | ||
| SELECT | ||
| "accounts_with_secondary_emails" AS table_name, | ||
| COUNT( | ||
| DISTINCT `moz-fx-data-shared-prod.accounts_db_external.fxa_accounts_v1`.uid | ||
| ) AS total_rows | ||
| FROM | ||
| `moz-fx-data-shared-prod.accounts_db_external.fxa_accounts_v1` FOR SYSTEM_TIME AS OF TIMESTAMP( | ||
| @as_of_date + 1, | ||
| 'UTC' | ||
| ) | ||
| JOIN | ||
| `moz-fx-data-shared-prod.accounts_db_external.fxa_emails_v1` FOR SYSTEM_TIME AS OF TIMESTAMP( | ||
| @as_of_date + 1, | ||
| 'UTC' | ||
| ) | ||
| ON `moz-fx-data-shared-prod.accounts_db_external.fxa_accounts_v1`.uid = `moz-fx-data-shared-prod.accounts_db_external.fxa_emails_v1`.uid | ||
| WHERE | ||
| `moz-fx-data-shared-prod.accounts_db_external.fxa_emails_v1`.isPrimary = FALSE | ||
| ) | ||
| UNION ALL | ||
| ( | ||
| SELECT | ||
| "accounts_with_unverified_emails" AS table_name, | ||
| COUNT( | ||
| DISTINCT `moz-fx-data-shared-prod.accounts_db_external.fxa_accounts_v1`.uid | ||
| ) AS total_rows | ||
| FROM | ||
| `moz-fx-data-shared-prod.accounts_db_external.fxa_accounts_v1` FOR SYSTEM_TIME AS OF TIMESTAMP( | ||
| @as_of_date + 1, | ||
| 'UTC' | ||
| ) | ||
| JOIN | ||
| `moz-fx-data-shared-prod.accounts_db_external.fxa_emails_v1` FOR SYSTEM_TIME AS OF TIMESTAMP( | ||
| @as_of_date + 1, | ||
| 'UTC' | ||
| ) | ||
| ON `moz-fx-data-shared-prod.accounts_db_external.fxa_accounts_v1`.uid = `moz-fx-data-shared-prod.accounts_db_external.fxa_emails_v1`.uid | ||
| WHERE | ||
| `moz-fx-data-shared-prod.accounts_db_external.fxa_emails_v1`.isVerified = FALSE | ||
| ) | ||
| UNION ALL | ||
| ( | ||
| SELECT | ||
| "accounts_linked_to_google" AS table_name, | ||
| COUNT(uid) AS total_rows | ||
| FROM | ||
| `moz-fx-data-shared-prod.accounts_db_external.fxa_linked_accounts_v1` FOR SYSTEM_TIME AS OF TIMESTAMP( | ||
| @as_of_date + 1, | ||
| 'UTC' | ||
| ) | ||
| WHERE | ||
| providerId = 1 -- see LinkedAccountProviderIds at https://github.com/mozilla/fxa/blob/main/packages/fxa-settings/src/lib/types.ts | ||
| ) | ||
| UNION ALL | ||
| ( | ||
| SELECT | ||
| "accounts_linked_to_apple" AS table_name, | ||
| COUNT(uid) AS total_rows | ||
| FROM | ||
| `moz-fx-data-shared-prod.accounts_db_external.fxa_linked_accounts_v1` FOR SYSTEM_TIME AS OF TIMESTAMP( | ||
| @as_of_date + 1, | ||
| 'UTC' | ||
| ) | ||
| WHERE | ||
| providerId = 2 -- see LinkedAccountProviderIds at https://github.com/mozilla/fxa/blob/main/packages/fxa-settings/src/lib/types.ts | ||
| ) | ||
| ) | ||
| SELECT | ||
| @as_of_date AS as_of_date, | ||
| table_name, | ||
| total_rows | ||
| FROM | ||
| table_counts | ||
10 changes: 10 additions & 0 deletions
10
sql/moz-fx-data-shared-prod/accounts_backend_derived/monitoring_db_counts_v1/schema.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| fields: | ||
| - name: as_of_date | ||
| type: DATE | ||
| mode: NULLABLE | ||
| - name: table_name | ||
| type: STRING | ||
| mode: NULLABLE | ||
| - name: total_rows | ||
| type: INTEGER | ||
| mode: NULLABLE |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are there cases where
uidis null in this table?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unlikely, IIUC this query has been running for a while in a custom environment.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've run it by hand a few times but not "awhile". A NULL there would be a bug in the system. There shouldn't be any.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Usually the only reason to use
COUNT(expression)rather thanCOUNT(*)is if the expression might be null and shouldn't be counted in that case.Or if it was intended to count distinct
uidvalues then it should beCOUNT(DISTINCT uid).