fix(pydantic-ai): Stop capturing internal exceptions

[alexander-alderman-webb]

Description

Stop capturing broad exceptions, that bubble up to top-level run functions, at the tool call level for Pydantic AI. Only capture ToolRetryError at the tool call level, and mark these exceptions as handled.

Issues

Closes #5232

Reminders

• Please add tests to validate your changes, and lint your code using tox -e linters.
• Add GH Issue ID & Linear ID (if applicable)
• PR title should use conventional commit style (feat:, fix:, ref:, meta:)
• For external contributors: CONTRIBUTING.md, Sentry SDK development docs, Discord community

[sentrivana]

I wonder if removing the exception capturing altogether is the way to go. If a user has errors happening in their tool, even if they don't necessarily bubble all the way up, they'll lose visibility into them. If e.g. a tool is consistently providing erroneous output, the dev might want to know about it and fix it, even if the whole LLM chain manages to recover.

What we could consider:

• still capturing other exceptions, just not ToolRetryError (that wouldn't fix the example I talk about above, but it'd at least make sure we don't stop capturing other potentially important exceptions)
• see which of the exceptions captured here will be handled by Pydantic AI internally and capture those with handled=True
• add an integration-level option to turn tool exceptions on/off

Since I have little experience in this type of development though, might be this is also fine as is and no one cares about recoverable errors in their tools. 😅

[alexander-alderman-webb]

good points, I did some more investigation.

• Other errors bubble all the way up, so if we catch them here we're catching them multiple times. At least on the code path in

_agent_graph.process_tool_calls()
-> _agent_graph._call_tools()
-> _agent_graph._call_tool()
-> ToolManager.handle_call()
-> ToolManager._call_tool()`

A few places call process_tool_calls(), but only ToolRetryError is handled, and experimentally, other errors bubble up to the top-level run methods.

• I'm open to this. Tool authors can raise ModelRetry to trigger a ToolRetryError, or ToolRetryError can result from a validation failure. These are documented, and the only two ways to raise ToolRetryError in ToolManager._call_tool(). See https://ai.pydantic.dev/tools-advanced/#tool-retries.
• We should add the option if we capture ToolRetryError.

[sentrivana]

Thanks for investigating!

I'm leaning towards the following:

• removing the broad except Exception and replacing it with except ToolRetryError, to prevent double capture of other errors
• gating the actual ToolRetryError capture behind an option (unsure about the default)
• if the option is on and we end up capturing a ToolRetryError, capturing it with handled=True

WDYT?

[alexander-alderman-webb]

I agree, was already working on this approach.

I set the option to True, since this is useful information in my view.

[sentrivana]

Please also add to https://docs.sentry.io/platforms/python/integrations/pydantic-ai/#options 🙏🏻