Built-in Tools Support

[singankit]

Fixes # 2585

Document for Responses API with built-in tools proposal : https://github.com/singankit/semantic_convention_proposal/blob/main/semantic-convention-responses-api.md

Prototype
OpenAI Agents Sample

Changes

Please provide a brief description of the changes here.

Important

Pull requests acceptance are subject to the triage process as described in Issue and PR Triage Management.
PRs that do not follow the guidance above, may be automatically rejected and closed.

Merge requirement checklist

• CONTRIBUTING.md guidelines followed.
• Change log entry added, according to the guidelines in When to add a changelog entry.
  • If your PR does not need a change log, start the PR title with [chore]
• Links to the prototypes or existing instrumentations (when adding or changing conventions)

[singankit]

https://github.com/open-telemetry/semantic-conventions/issues/2585

[lmolkova]

I think we need to document the schema, not just the example, so that we know how to record different types of tools.

In this proposal shape tool call part would have to accommodate file search, code_interpreter and everything else without clear idea on what the format of each of them and which property is used as a discriminator.

[singankit]

@lmolkova Are you suggesting having a discriminator that can distinguish which tool is called and based on the tool call schema can be inferred. For example, in below example , tool is code_interpreter and "code_interpreter" property is present only when tool_type is code_interpreter.

# Code Interpreter tool call part
{
    "type": "tool_call"
    "id": "id",
    "name": "code_interpreter",
    "tool_type": "code_interpreter", # discriminator
    "code_interpreter": {
        "arguments": {
          "code": "import random\n\n# Generate a random number\nrandom_number = random.randint(1, 100)\n\n# Execute some operation with the random number (e.g., squaring it)\nresult = random_number ** 2\n\nrandom_number, result",
          "container_id": "cntr_690bdbfed8688190884efd4c7ae6435b0db1f006442e8941",
        }
     }
}

For Function Tool Call

{
    "role": "assistant",
    "parts": [
      {
        "type": "tool_call",
        "id": "call_VSPygqKTWdrhaFErNvMV18Yl",
        "tool_type": "function",
       "function": {
             "name": "get_weather",
             "arguments": {
                  "location": "Paris"
             }
        },
       
      }
    ],
    "finish_reason": "tool_call"
  }

[lmolkova]

Yeah, I'm suggesting to add a formal json schema that specifies how to record built-in tools and use some discriminator to separate them.

[singankit]

Have updated the PR including schema for code interpreter and making tool call request/response polymorphic to allow any tool to be supported (just need tool call/response schema to be added)

[DylanRussell]

Is it worthwhile to maintain a formal schema for each built in tool ?

Gemini has 6 or 7 built in tools. This would require us to maintain 6 or 7 schemas here for the response.. Each model has it's own set of built-in tools that probably overlap a little but not completely, and I think it'll be kind of a pain to converge on a shared schema for each of them..

Or can we just use a generic catch-all Any type instead of doing that?

Also for Gemini there is no ToolCallRequest for built-in tools. Instead built-in tools are enabled, and you can only tell whether the model used them by checking what the model returned in the response.

[singankit]

Thanks @DylanRussell for the feedback. Having Any for any unknown tools makes sense to me. This proposal supports including tool schemas for different tools even provider specific tools if needed. Having ToolCallRequest Part provided information about tool being called. However, if it captures the actual arguments tool is called can probably be optional.

[lmolkova]

let's also update the python code

[lmolkova]

should this be provider-specific or generic? are code_interpreter tools are similar ?

[aabmass]

What do you think of flattening the changes into a new part type instead of extending the type: "tool_call" Part here?