[Bug]: Opik and uvloop

[khoatavbis]

I got this error socket.gaierror: [Errno -3] Temporary failure in name resolution when running FastAPI app
I think Opik is interfering with uvloop when I tried to initiate my db connection (I am not sure)
the workaround turns out to be very simple, comment out the from opik import track, opik_context line, start the app and uncomment the line again.
So please help, is there another similar way to fix this behavior?

[Nimrod007]

Hi @khoatavbis! 👋

To help us fix this, could you share:

1. Full error traceback (the complete error message)
2. Python & package versions: python --version && pip show opik uvloop
3. Test this: Does OPIK_SENTRY_ENABLE=false python your_app.py work?

Thanks!

[khoatavbis]

Thank you @Nimrod007, for the reply

1. the error log (sorry it looks messy)

ERROR:    Traceback (most recent call last):
  File "/home/khoa/ai_platform/.venv/lib/python3.12/site-packages/starlette/routing.py", line 677, in lifespan
    async with self.lifespan_context(app) as maybe_state:
  File "/usr/local/lib/python3.12/contextlib.py", line 204, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/khoa/ai_platform/backend/main.py", line 31, in lifespan
    async with engine.begin() as conn:
  File "/usr/local/lib/python3.12/contextlib.py", line 204, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/khoa/ai_platform/.venv/lib/python3.12/site-packages/sqlalchemy/ext/asyncio/engine.py", line 1061, in begin
    async with conn:
  File "/home/khoa/ai_platform/.venv/lib/python3.12/site-packages/sqlalchemy/ext/asyncio/base.py", line 125, in __aenter__
    return await self.start(is_ctxmanager=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khoa/ai_platform/.venv/lib/python3.12/site-packages/sqlalchemy/ext/asyncio/engine.py", line 270, in start
    await greenlet_spawn(self.sync_engine.connect)
  File "/home/khoa/ai_platform/.venv/lib/python3.12/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 190, in greenlet_spawn
    result = context.throw(*sys.exc_info())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khoa/ai_platform/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 3268, in connect
    return self._connection_cls(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khoa/ai_platform/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 145, in __init__
    self._dbapi_connection = engine.raw_connection()
                             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khoa/ai_platform/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 3292, in raw_connection
    return self.pool.connect()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/khoa/ai_platform/.venv/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 452, in connect
    return _ConnectionFairy._checkout(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khoa/ai_platform/.venv/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 1269, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khoa/ai_platform/.venv/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 716, in checkout
    rec = pool._do_get()
          ^^^^^^^^^^^^^^
  File "/home/khoa/ai_platform/.venv/lib/python3.12/site-packages/sqlalchemy/pool/impl.py", line 169, in _do_get
    with util.safe_reraise():
  File "/home/khoa/ai_platform/.venv/lib/python3.12/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/home/khoa/ai_platform/.venv/lib/python3.12/site-packages/sqlalchemy/pool/impl.py", line 167, in _do_get
    return self._create_connection()
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khoa/ai_platform/.venv/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 393, in _create_connection
    return _ConnectionRecord(self)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khoa/ai_platform/.venv/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 678, in __init__
    self.__connect()
  File "/home/khoa/ai_platform/.venv/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 902, in __connect
    with util.safe_reraise():
  File "/home/khoa/ai_platform/.venv/lib/python3.12/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/home/khoa/ai_platform/.venv/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 898, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khoa/ai_platform/.venv/lib/python3.12/site-packages/sqlalchemy/engine/create.py", line 637, in connect
    return dialect.connect(*cargs, **cparams)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khoa/ai_platform/.venv/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 616, in connect
    return self.loaded_dbapi.connect(*cargs, **cparams)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khoa/ai_platform/.venv/lib/python3.12/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 936, in connect
    await_only(creator_fn(*arg, **kw)),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khoa/ai_platform/.venv/lib/python3.12/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 125, in await_only
    return current.driver.switch(awaitable)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khoa/ai_platform/.venv/lib/python3.12/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 185, in greenlet_spawn
    value = await result
            ^^^^^^^^^^^^
  File "/home/khoa/ai_platform/.venv/lib/python3.12/site-packages/asyncpg/connection.py", line 2443, in connect
    return await connect_utils._connect(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khoa/ai_platform/.venv/lib/python3.12/site-packages/asyncpg/connect_utils.py", line 1249, in _connect
    raise last_error or exceptions.TargetServerAttributeNotMatched(
  File "/home/khoa/ai_platform/.venv/lib/python3.12/site-packages/asyncpg/connect_utils.py", line 1218, in _connect
    conn = await _connect_addr(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/khoa/ai_platform/.venv/lib/python3.12/site-packages/asyncpg/connect_utils.py", line 1054, in _connect_addr
    return await __connect_addr(params, True, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khoa/ai_platform/.venv/lib/python3.12/site-packages/asyncpg/connect_utils.py", line 1099, in __connect_addr
    tr, pr = await connector
             ^^^^^^^^^^^^^^^
  File "/home/khoa/ai_platform/.venv/lib/python3.12/site-packages/asyncpg/connect_utils.py", line 969, in _create_ssl_connection
    tr, pr = await loop.create_connection(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1982, in create_connection
socket.gaierror: [Errno -3] Temporary failure in name resolution

ERROR:    Application startup failed. Exiting.

2. My env

Python 3.12.0
Name: opik
Version: 1.9.31
...
Requires: boto3-stubs, click, httpx, jinja2, litellm, openai, pydantic, pydantic-settings, pytest, rapidfuzz, rich, sentry_sdk, tenacity, tqdm, uuid6
---
Name: uvloop
Version: 0.22.1
...

3. export OPIK_SENTRY_ENABLE=false does not work for me

However, I found a workaround (based on ChatGPT :)), they said opik tracking is interfering with my initiation of db connection), so I do a lazy import of from opik import track, opik_context in my module and it worked and I can track simple things normally like trace/span. Well certainly I hope we can come up with a cleaner solution

[khoatavbis]

@Nimrod007, sorry but do you come up with any clean fix?

[Nimrod007]

Hey @khoatavbis , were still taking a look will update you about this soon

[YarivHashaiComet]

Jira Ticket Created: OPIK-3415

[khoatavbis]

Hi @Nimrod007, I think I found the reason for the error. Somehow Opik manages to read my .env file, thus changing the "host" in my DB_URL from "postgres" (docker prod env) to "localhost" (my local dev env). This is also the reason for the behavior I described at the beginning. So I guess no need to fix it, it will be fine in prod and keep this in mind for local development