Skip to content

fix(deps): support pandas 3.0 (prerelease)#998

Merged
tswast merged 4 commits intomainfrom
tswast-pandas-3
Mar 2, 2026
Merged

fix(deps): support pandas 3.0 (prerelease)#998
tswast merged 4 commits intomainfrom
tswast-pandas-3

Conversation

@tswast
Copy link
Contributor

@tswast tswast commented Dec 9, 2025

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> 🦕

@tswast tswast requested a review from a team as a code owner December 9, 2025 20:45
@tswast tswast requested review from a team, TrevorBergeron and shollyman December 9, 2025 20:45
@product-auto-label product-auto-label bot added size: xs Pull request size is extra small. api: bigquery Issues related to the googleapis/python-bigquery-pandas API. labels Dec 9, 2025
@gemini-code-assist
Copy link
Contributor

Summary of Changes

Hello @tswast, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request primarily focuses on updating the project's dependency configuration to support the upcoming pandas 3.0 release. This involves removing an outdated version constraint for pandas and making minor, targeted adjustments to existing test cases to ensure compatibility with changes in pandas' handling of missing data and specific BigQuery date parsing behaviors.

Highlights

  • Dependency Update: The upper version constraint for the pandas dependency in setup.py has been removed, enabling support for pandas 3.0 (prerelease) and future versions.
  • Test Case Adjustment (Missing Values): A test case in test_to_gbq.py was updated to use pandas.NA instead of None for representing missing values in a pandas.Series of object dtype, aligning with modern pandas practices.
  • Test Case Adjustment (Datetime): A datetime test value was changed from datetime(1, 1, 1) to datetime(1000, 1, 1) in test_to_gbq.py to accommodate a BigQuery CSV loader requirement for TIMESTAMP formatting, specifically regarding leading zeros.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the pandas dependency to allow for version 3.0 by removing the upper version constraint. The accompanying changes in the system tests, which involve updating the representation of missing values to pandas.NA and adjusting a datetime test value, appear to be necessary for compatibility with the newer pandas version. Overall, the changes look good, but I have one suggestion to clarify a comment in the test suite for better maintainability.

Comment on lines +347 to +350
# CSV loader in BigQuery currently requires leading 0s
# for TIMESTAMP but not DATETIME. See internal issue
# b/467399807.
datetime.datetime(1000, 1, 1),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The comment on lines 347-349 is a bit confusing. It states that the CSV loader's requirement for leading zeros applies to TIMESTAMP but not DATETIME. However, this change is made to datetime_col, which is defined as a DATETIME column in the table schema on line 374. Could you please clarify why this change is needed for a DATETIME column, or update the comment for future reference? This would help maintainers understand the reason for this specific test value.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct. I use year >= 1000 so that no leading zeros are required.

@tswast
Copy link
Contributor Author

tswast commented Dec 9, 2025

Local tests pass with pandas 3.0.0rc0.

Details
(venv) ➜  python-bigquery-pandas git:(tswast-pandas-3) ✗ pytest tests/
============================================================================================================== test session starts ==============================================================================================================
platform linux -- Python 3.13.5, pytest-9.0.1, pluggy-1.6.0
rootdir: /home/swast/src/github.com/googleapis/python-bigquery-pandas
collected 339 items                                                                                                                                                                                                                             

tests/system/test_auth.py ....                                                                                                                                                                                                            [  1%]
tests/system/test_gbq.py ........................................s......s...................                                                                                                                 [ 20%]
tests/system/test_read_gbq.py ssssssssss....                                                                                                                                                                 [ 25%]
tests/system/test_read_gbq_with_bqstorage.py ...                                                                                                                                                             [ 25%]
tests/system/test_sample.py ....                                                                                                                                                                             [ 27%]
tests/system/test_to_gbq.py .....s..........s......s.                                                                                                                                                        [ 34%]
tests/unit/schema/test_pandas_to_bigquery.py .....s                                                                                                                                                          [ 36%]
tests/unit/schema/test_pyarrow_to_bigquery.py ............                                                                                                                                                   [ 39%]
tests/unit/test_auth.py ....                                                                                                                                                                                 [ 41%]
tests/unit/test_context.py ...                                                                                                                                                                               [ 41%]
tests/unit/test_core_sample.py ......................                                                                                                                                                        [ 48%]
tests/unit/test_features.py ...........                                                                                                                                                                      [ 51%]
tests/unit/test_gbq.py ...........................................................................                                                                                                           [ 73%]
tests/unit/test_load.py ........................                                                                                                                                                             [ 80%]
tests/unit/test_query.py ...................                                                                                                                                                                 [ 86%]
tests/unit/test_schema.py .......................                                                                                                                                                            [ 93%]
tests/unit/test_timestamp.py ...                                                                                                                                                                             [ 94%]
tests/unit/test_to_gbq.py ....................                                                                                                                                                               [100%]

================================================================================================= warnings summary =================================================================================================
tests/system/test_auth.py::test_get_credentials_user_credentials_with_reauth
  /home/swast/src/github.com/googleapis/python-bigquery-pandas/venv/lib/python3.13/site-packages/google_auth_oauthlib/helpers.py:150: DeprecationWarning: datetime.datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.fromtimestamp(timestamp, datetime.UTC).
    credentials.expiry = datetime.datetime.utcfromtimestamp(session.token["expires_at"])

tests/system/test_gbq.py: 224 warnings
tests/system/test_read_gbq.py: 3 warnings
tests/system/test_read_gbq_with_bqstorage.py: 3 warnings
tests/system/test_sample.py: 4 warnings
tests/system/test_to_gbq.py: 108 warnings
tests/unit/test_context.py: 6 warnings
tests/unit/test_gbq.py: 63 warnings
tests/unit/test_query.py: 6 warnings
tests/unit/test_to_gbq.py: 24 warnings
  /home/swast/src/github.com/googleapis/python-bigquery-pandas/pandas_gbq/gbq_connector.py:308: PendingDeprecationWarning: In a future major release, the default delimiter will be changed to a `/` in accordance with RFC9110.
    user_agent = create_user_agent(

tests/system/test_gbq.py: 30 warnings
tests/system/test_read_gbq.py: 4 warnings
tests/system/test_read_gbq_with_bqstorage.py: 3 warnings
tests/system/test_sample.py: 4 warnings
tests/system/test_to_gbq.py: 22 warnings
tests/unit/test_context.py: 6 warnings
tests/unit/test_gbq.py: 20 warnings
  /home/swast/src/github.com/googleapis/python-bigquery-pandas/pandas_gbq/gbq_connector.py:134: PendingDeprecationWarning: In a future major release, the default delimiter will be changed to a `/` in accordance with RFC9110.
    user_agent = create_user_agent(

tests/system/test_gbq.py: 28 warnings
tests/system/test_read_gbq.py: 4 warnings
tests/system/test_read_gbq_with_bqstorage.py: 3 warnings
tests/system/test_to_gbq.py: 22 warnings
  /home/swast/src/github.com/googleapis/python-bigquery-pandas/venv/lib/python3.13/site-packages/google/cloud/bigquery/table.py:2689: UserWarning: A progress bar was requested, but there was an error loading the tqdm library. Please install tqdm to use the progress bar functionality.
    record_batch = self.to_arrow(

tests/system/test_gbq.py::TestToGBQIntegration::test_upload_data
  /home/swast/src/github.com/googleapis/python-bigquery-pandas/tests/system/test_gbq.py:677: DeprecationWarning: chunksize is ignored when using api_method='load_parquet'
    gbq.to_gbq(

tests/system/test_gbq.py::TestToGBQIntegration::test_upload_data_if_table_exists_fail
  /home/swast/src/github.com/googleapis/python-bigquery-pandas/tests/system/test_gbq.py:735: DeprecationWarning: chunksize is ignored when using api_method='load_parquet'
    gbq.to_gbq(

tests/system/test_gbq.py::TestToGBQIntegration::test_upload_data_if_table_exists_append
  /home/swast/src/github.com/googleapis/python-bigquery-pandas/tests/system/test_gbq.py:769: DeprecationWarning: chunksize is ignored when using api_method='load_parquet'
    gbq.to_gbq(

tests/system/test_gbq.py::TestToGBQIntegration::test_upload_subset_columns_if_table_exists_append
  /home/swast/src/github.com/googleapis/python-bigquery-pandas/tests/system/test_gbq.py:815: DeprecationWarning: chunksize is ignored when using api_method='load_parquet'
    gbq.to_gbq(

tests/system/test_gbq.py::TestToGBQIntegration::test_upload_data_if_table_exists_replace
  /home/swast/src/github.com/googleapis/python-bigquery-pandas/tests/system/test_gbq.py:847: FutureWarning: generate_bq_schema is deprecated and will be removed in a future version
    schema_new = gbq.generate_bq_schema(df_different_schema)

tests/system/test_gbq.py::TestToGBQIntegration::test_upload_data_if_table_exists_replace
  /home/swast/src/github.com/googleapis/python-bigquery-pandas/tests/system/test_gbq.py:850: DeprecationWarning: chunksize is ignored when using api_method='load_parquet'
    gbq.to_gbq(

tests/system/test_gbq.py::TestToGBQIntegration::test_upload_data_if_table_exists_replace
  /home/swast/src/github.com/googleapis/python-bigquery-pandas/tests/system/test_gbq.py:874: FutureWarning: generate_bq_schema is deprecated and will be removed in a future version
    schema_returned = gbq.generate_bq_schema(df_new)

tests/system/test_gbq.py::TestToGBQIntegration::test_upload_data_flexible_column_order
  /home/swast/src/github.com/googleapis/python-bigquery-pandas/tests/system/test_gbq.py:980: DeprecationWarning: chunksize is ignored when using api_method='load_parquet'
    gbq.to_gbq(

tests/unit/test_schema.py::test_generate_bq_schema[default-type-fails-pyarrow-conversion]
  /home/swast/src/github.com/googleapis/python-bigquery-pandas/pandas_gbq/schema/pandas_to_bigquery.py:159: UserWarning: Could not determine the type of columns: col1
    warnings.warn(msg)

tests/unit/test_to_gbq.py::test_create_user_agent[test_user_agent/2.0.42-False-test_user_agent/2.0.42 pandas-3.0.0rc0]
tests/unit/test_to_gbq.py::test_create_user_agent[None-False-pandas-3.0.0rc0]
  /home/swast/src/github.com/googleapis/python-bigquery-pandas/tests/unit/test_to_gbq.py:211: PendingDeprecationWarning: In a future major release, the default delimiter will be changed to a `/` in accordance with RFC9110.
    result = create_user_agent(user_agent, rfc9110_delimiter)

tests/unit/test_to_gbq.py::test_create_user_agent_vscode
  /home/swast/src/github.com/googleapis/python-bigquery-pandas/tests/unit/test_to_gbq.py:219: PendingDeprecationWarning: In a future major release, the default delimiter will be changed to a `/` in accordance with RFC9110.
    assert create_user_agent() == f"pandas-{pd.__version__} vscode"

tests/unit/test_to_gbq.py::test_create_user_agent_vscode_plugin
  /home/swast/src/github.com/googleapis/python-bigquery-pandas/tests/unit/test_to_gbq.py:243: PendingDeprecationWarning: In a future major release, the default delimiter will be changed to a `/` in accordance with RFC9110.
    create_user_agent()

tests/unit/test_to_gbq.py::test_create_user_agent_jupyter
  /home/swast/src/github.com/googleapis/python-bigquery-pandas/tests/unit/test_to_gbq.py:252: PendingDeprecationWarning: In a future major release, the default delimiter will be changed to a `/` in accordance with RFC9110.
    assert create_user_agent() == f"pandas-{pd.__version__} jupyter"

tests/unit/test_to_gbq.py::test_create_user_agent_jupyter_extension
  /home/swast/src/github.com/googleapis/python-bigquery-pandas/tests/unit/test_to_gbq.py:271: PendingDeprecationWarning: In a future major release, the default delimiter will be changed to a `/` in accordance with RFC9110.
    create_user_agent()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

@GarrettWu GarrettWu removed their assignment Dec 12, 2025
@product-auto-label product-auto-label bot added size: s Pull request size is small. and removed size: xs Pull request size is extra small. labels Jan 5, 2026
@product-auto-label product-auto-label bot added size: xs Pull request size is extra small. and removed size: s Pull request size is small. labels Jan 5, 2026
@tswast tswast requested a review from a team as a code owner February 24, 2026 20:34
@chalmerlowe
Copy link
Contributor

📢 Migration Notice: 📢 This library is moving to the google-cloud-python monorepo soon.

We kept this PR open due to recent activity. We would like to finalize this PR so it can be merged if it is critical.

If we don't hear from the PR author, we will close this PR in the next few days. The PR can then be re-opened in the monorepo once the migration is complete and work can continue there.

@tswast tswast merged commit 4ea2d57 into main Mar 2, 2026
28 of 29 checks passed
@tswast tswast deleted the tswast-pandas-3 branch March 2, 2026 21:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: bigquery Issues related to the googleapis/python-bigquery-pandas API. size: xs Pull request size is extra small.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants