Dryrun caching #8486
Merged
Dryrun caching #8486
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
scholtzan
force-pushed
the
query-schema-generation-dryrun-caching
branch
from
e4ec61f to
e5f7754
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Collaborator
Author
|
dryrun failing due to unrelated error in |
scholtzan
force-pushed
the
query-schema-generation-dryrun-caching
branch
from
af4028d to
8932af9
Compare
This comment has been minimized.
This comment has been minimized.
scholtzan
force-pushed
the
query-schema-generation-dryrun-caching
branch
2 times, most recently
from
7f0b230 to
163789c
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Collaborator
Author
|
I won't merge this before next week. |
wwyc
approved these changes
Nov 27, 2025
BenWu
reviewed
Nov 27, 2025
Contributor
BenWu
left a comment
BenWu
left a comment
There was a problem hiding this comment.
It looks like the test failure is due to reset-stage-env not having a dependency on test_routines so stage udfs are getting deleted
bigquery_etl/dryrun.py
Outdated
| try: | ||
| # write to temporary file first, then atomically rename | ||
| # this prevents race conditions where readers get partial files | ||
| temp_file = Path(str(cache_file) + f".tmp.{os.getpid()}") |
Contributor
There was a problem hiding this comment.
I think there's a possible edge case if this is called from a thread pool where the threads have the same PID an extra random string might be better.
Suggested change
| temp_file = Path(str(cache_file) + f".tmp.{os.getpid()}") | |
| temp_file = Path(str(cache_file) + f".tmp.{os.getpid()}.{os.urandom(4).hex()}") |
bigquery_etl/dryrun.py
Outdated
| table_cache_key = hashlib.sha256(table_identifier.encode()).hexdigest() | ||
| cache_file = cache_dir / f"table_metadata_{table_cache_key}.pkl" | ||
|
|
||
| try: |
Contributor
There was a problem hiding this comment.
The cache read and write code is almost identical between the metadata and the dry run result. Moving the common code into a common function would make this easier to maintain
scholtzan
force-pushed
the
query-schema-generation-dryrun-caching
branch
from
7fbd6bb to
63b77e5
Compare
Integration report for "Address feedback"
|
sean-rose
reviewed
Dec 12, 2025
Comment on lines
+2303
to
+2305
| project=project_name, | ||
| dataset=dataset_name, | ||
| table=table_name, |
Contributor
There was a problem hiding this comment.
It turns out this particular change is causing problems with deploying ETLs to stage:
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds a cache to
DryRunwhich caches query and table schemas for a configurable amount of time and as long as the query content is the same. This reduces the schema generation time slightly which is executed before table artifact deploys (locally it saved 3 to 4 minutes).Reviewer, please follow this checklist