BLDX-434 | Migrate to msgspec.Struct models

[Aryamanz29]

✨ Description

https://linear.app/atlan-epd/issue/BLDX-434/plan-productionization-of-msgspecstruct-models

🧩 Type of change

Select all that apply:

• 🚀 New feature (non-breaking change that adds functionality)
• 🐛 Bug fix (non-breaking change that fixes an issue) — please include tests! Refer testing-toolkit 🧪
• 🔄 Refactor (code change that neither fixes a bug nor adds a feature)
• 🧹 Maintenance (chores, cleanup, minor improvements)
• 💥 Breaking change (fix or feature that may break existing functionality)
• 📦 Dependency upgrade/downgrade
• 📚 Documentation updates

---

📋 Checklist

• My code follows the project’s style guidelines
• I’ve performed a self-review of my code
• I’ve added comments in tricky or complex areas
• I’ve updated the documentation as needed
• There are no new warnings from my changes
• I’ve added tests to cover my changes
• All new and existing tests pass locally

[fyzanshaik-atlan]

@greptile review

[greptile-apps]

Too many files changed for review. (650 files found, 500 file limit)

[fyzanshaik-atlan]

@greptile re-review

[fyzanshaik-atlan]

@claude /review

[fyzanshaik-atlan]

@claude /review

[claude]

Claude encountered an error —— View job

---

I'll analyze this and get back to you.

[Aryamanz29]

@claude /review

[claude]

Claude encountered an error —— View job

---

I'll analyze this and get back to you.

[fyzanshaik-atlan]

Code Review

This PR migrates the pyatlan SDK's internal model layer from Pydantic BaseModel to msgspec.Struct, introducing a parallel pyatlan_v9 package with its own client, model, and test trees. The approach runs both model systems side-by-side, bridging them via a custom _is_model_instance helper and a registered Pydantic encoder for msgspec structs. The change is marked as breaking and covers ~990 files.

Confidence Score: 2/5

• This PR is a draft and not yet intended for merge — reviewed in that context
• Checked for correctness bugs, security, standards compliance, CI integration, and packaging
• Score deducted for two infrastructure-level blockers: missing runtime dependency declaration and absent CI coverage for the new test suite, plus a confirmed logic bug in the audit model

Important Files Changed

File	Change	Risk
pyproject.toml	Modified	High — runtime dependency missing
.github/workflows/pyatlan-pr.yaml	Modified	High — v9 tests not wired
pyatlan/validate.py	Added	Medium — cross-boundary type matching
pyatlan_v9/client/atlan.py	Added	High — core client entry point
pyatlan_v9/model/audit.py	Added	Medium — logic bug in sort fallback
pyatlan_v9/model/typedef.py	Added	Medium — module-level mutable state
pyatlan_v9/client/transport.py	Added	Medium — custom HTTP retry transport
tests_v9/unit/	Added	Medium — not run in CI

Change Flow

sequenceDiagram
    participant User as User Code
    participant Client as AtlanClient (v9)
    participant SubClient as Sub-client (e.g. AssetClient)
    participant Validate as validate_arguments
    participant Bridge as _is_model_instance
    participant Msgspec as msgspec.Struct model
    participant HTTP as httpx + PyatlanTransport
    participant Legacy as Legacy Pydantic layer

    User->>Client: AtlanClient(base_url, api_key)
    Client->>Client: __post_init__ (env fallbacks, session init)
    User->>SubClient: client.assets.get_by_guid(guid)
    SubClient->>Validate: @validate_arguments checks type
    Validate->>Bridge: _is_model_instance(value, expected_type)
    Bridge-->>Validate: match by MRO name
    SubClient->>HTTP: _call_api(endpoint, request_obj)
    HTTP->>Legacy: deserialize response (Pydantic)
    Legacy-->>SubClient: legacy Asset object
    SubClient->>Msgspec: (v9 path) msgspec.convert(raw, TargetStruct)
    Msgspec-->>User: typed v9 model instance

Loading

Findings

#	Severity	File	Issue
1	Critical	pyproject.toml:29-41	msgspec is not declared as a runtime dependency. Any installation of the package will produce ModuleNotFoundError when importing pyatlan_v9.
2	Critical	.github/workflows/pyatlan-pr.yaml:117	The CI workflow only runs pytest tests/unit (legacy). The tests_v9/unit suite — 1,852 tests — is never executed on PRs. Regressions in v9 code will not be caught.
3	Warning	pyatlan_v9/model/audit.py:92,117,149	Logic bug in sort fallback: sort=sort if LATEST_FIRST else []. LATEST_FIRST is a non-empty list and is always truthy, so the condition always takes the left branch. The empty-list fallback is dead code. Intended expression is likely sort if sort else LATEST_FIRST.
4	Warning	.github/workflows/pyatlan-pr.yaml:109-110	QA checks (ruff-format, ruff-lint, mypy) are commented out. One of the commits is explicitly named [temp/ci] Skip QA checks. These must be restored before merge.
5	Warning	pyatlan_v9/model/typedef.py:448,557,675	_OPTIONS_PARENT_MAP: dict[int, Any] = {} is a module-level mutable dict keyed by id(self). Object IDs are reused after garbage collection, and there is no cleanup path, causing unbounded growth in long-running processes. This is also not thread-safe.
6	Warning	pyatlan/validate.py:91,110	_is_model_instance generic fallback matches any class whose MRO contains a name equal to expected_type.__name__. A user-defined class named Asset or Query will pass validation for a pyatlan.model.assets.core.Asset expected type, silently accepting the wrong object.

[fyzanshaik-atlan]

Warning — logic bug: sort fallback is dead code

The condition sort if LATEST_FIRST else [] tests LATEST_FIRST, not sort. LATEST_FIRST is defined on line 33 as a non-empty list, so it is always truthy. The empty-list branch is unreachable and the expression is equivalent to sort unconditionally.

The same pattern appears on lines 117 and 149.

If the intent is to fall back to LATEST_FIRST when the caller passes an empty list:

Suggested change

	sort=sort if LATEST_FIRST else [],
	sort=sort if sort else LATEST_FIRST,

If the intent is always to use whatever was passed (with LATEST_FIRST as the default parameter), the expression can simply be:

Suggested change

	sort=sort if LATEST_FIRST else [],
	sort=sort,

[fyzanshaik-atlan]

Warning — class name collision: type check matches by name only

The generic fallback on this line accepts any object whose MRO contains a class whose __name__ equals expected_type.__name__. This is not a true isinstance check — it will return True for a user-defined class named Asset or Query when the expected type is pyatlan.model.assets.core.Asset.

The earlier msgspec-specific branch (line 62) and Pydantic branch (line 68) have the same property, but the generic fallback extends this to all classes, making it the riskiest path.

For the SDK's internal usage this is a pragmatic bridge pattern, but it breaks the behavioral contract of isinstance for any caller using this function as a validation gate.

[fyzanshaik-atlan]

Warning — QA checks disabled

Ruff formatting, ruff lint, and mypy are commented out. The [temp/ci] Skip QA checks commit confirms this is intentional during development. These must be re-enabled before the PR leaves draft status — without them, type errors and style violations across the 990-file migration accumulate silently.

[fyzanshaik-atlan]

Critical — v9 test suite never executed in CI

This job runs only pytest tests/unit (the legacy Pydantic test suite). The entire tests_v9/unit suite — 1,852 tests covering the new msgspec client, models, and transport — is not invoked anywhere in this workflow. Any regression in the new v9 code will go undetected on PRs. A separate step (or an additional argument to this run) for tests_v9/unit is required before this PR can leave draft status.

[fyzanshaik-atlan]

Warning — module-level mutable state: potential memory leak

_OPTIONS_PARENT_MAP is a module-level dict keyed by id(self). Python object IDs are reused after garbage collection: an AttributeDef.Options instance can be collected, a new unrelated object can receive the same ID, and _OPTIONS_PARENT_MAP will then point that new object to a stale parent. There is no eviction or cleanup path for this dict.

In any long-running process (server, integration test loop, batch job) this dict grows unbounded and holds references that prevent GC of the parent AttributeDef objects. It is also not thread-safe: two threads constructing AttributeDef objects concurrently can race on insertion.

Consider using weakref.WeakKeyDictionary with the Options instance as the key instead.

[fyzanshaik-atlan]

Critical — msgspec is not declared as a runtime dependency

pyatlan_v9 is built entirely on msgspec, but msgspec does not appear in the dependencies array in pyproject.toml. Any user who installs the package with pip install pyatlan will get ModuleNotFoundError: No module named 'msgspec' the first time they import anything from pyatlan_v9.

msgspec must be added to pyproject.toml [project] dependencies pinned to the minor version in use.