response_schema is not working when using typing.TypedDict

[mwigh]

Description of the bug:

When using response_schema in generate_content the response schema is not respected if the response_schema is set using a <class 'typing_extensions._TypedDictMeta'> object

Actual vs expected behavior:

I expect the response schema to be respected, since according to the documentation it should:
https://ai.google.dev/gemini-api/docs/structured-output?lang=python#generate-json

The issue is probably that the fields are not set as "required".
Can that be done somehow?

Any other information you'd like to share?

Code where I explicit set the schema but also explicit asks it to not respect the schema (but it still should, according to documentation):

import typing_extensions as typing
import google.generativeai as genai
import os

class Recipe(typing.TypedDict):
    recipe_name: str
    ingredients: list[str]

genai.configure(api_key=os.environ["API_KEY"])
model = genai.GenerativeModel("gemini-1.5-pro-latest")
result = model.generate_content(
    "List one popular cookie recipe. Response should be a JSON string on format {receipt_name: str}. There should only be one key in the response.",
    generation_config=genai.GenerationConfig(
        response_mime_type="application/json", response_schema=Recipe
    ),
)
print(result.text)

[Hamza-nabil]

This issue seems related to #541

[Gunand3043]

@MarkDaoust ,there is an issue with passing text prompt schema and model configuration schema at the same time, Please find the gist here for reference .Thanks!

[mwigh]

@Gunand3043
The problem is probably that GenerationConfig does not set the fields as required.

Your: "Example 2: (working) - Supply a schema through model configuration"
is not always working for all cases, that is how I found this issue

[kinto-b]

Yep there's a reprex of the Example 2 type failing here: #541 (comment)

[HNx1]

EDIT: Edited to reflect below conversation

Hi, just wanted to share an example: https://colab.research.google.com/drive/1ZdyrENeHWh30Ijws0o8l2Od5M4i56HjD?usp=sharing

In this case, I tested with both the python SDK and the REST API. The REST API call works all the time, and the python SDK does not work.

I tested with both gemini pro and flash and neither produced structured outputs in the sdk, but did with the rest api.

Extra test if excluding response_schema as suggested:
This now works 100% of the time, on both the sdk and the rest api. .

[mwigh]

I only mean that it seems that it is loosing required field when creating the response schema, and that code is also in this SDK.

See this commit that fixes it:
mwigh@f30c36d