-
Notifications
You must be signed in to change notification settings - Fork 3k
ci: add strict-no-cover to detect unnecessary coverage pragmas #1897
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
78d6633 to
edefce3
Compare
| uv run --frozen --no-sync coverage report | ||
| - name: Check for unnecessary no cover pragmas | ||
| if: runner.os != 'Windows' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Kludex without this all the windows pipelines fail to even run, seems like a strict-no-cover bug? Example failure from previous CI run: https://github.com/modelcontextprotocol/python-sdk/actions/runs/21070443439/job/60598385217
|
Is this working already? All the pragmas are actually being used? |
not sure, will test |
|
@Kludex Found two issues that prevented 1. Coverage 7.13.0 changed Fix: pin 2.
Fix: remove I also added a test |
464a9ff to
e942316
Compare
Code reviewNo issues found. Checked for bugs and CLAUDE.md compliance. |
f0c0b6e to
ab6650c
Compare
tests/shared/test_streamable_http.py
Outdated
| assert isinstance(captured_notifications[0], types.LoggingMessageNotification) # pragma: lax no cover | ||
| assert captured_notifications[0].params.data == "Second notification after lock" # pragma: lax no cover |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why are those needed?
tests/shared/test_streamable_http.py
Outdated
| # Now resume the session with the same mcp-session-id and protocol version | ||
| headers: dict[str, Any] = {} # pragma: no cover | ||
| if captured_session_id: # pragma: no cover | ||
| headers: dict[str, Any] = {} # pragma: lax no cover |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same... all those in this test actually
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems like on 3.11 there's a bug with the tg.cancel_scope.cancel() causing lines after it to not be marked as covered. I'll get claude to write up some details
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These are needed because of a coverage + anyio bug on Python 3.11 specifically.
What happens: when tg.cancel_scope.cancel() fires, anyio internally raises Cancelled to unwind the scope. On Python 3.11, this breaks coverage's sys.settrace-based line tracking — lines after the async with block exits are executed but not recorded by coverage. On 3.12+, coverage uses sys.monitoring (PEP 669) instead, which is not affected.
Confirmed locally — the lines ARE executed (inserting assert False at line 1306 fails the test), but coverage does not record them on 3.11.
Minimal repro (Python 3.11, coverage 7.12):
@pytest.mark.anyio
async def test_after_cancel():
async with anyio.create_task_group() as tg:
tg.start_soon(worker)
tg.cancel_scope.cancel()
x = 1 # MISS — not tracked by coverage
assert x == 1
@pytest.mark.anyio
async def test_after_normal_exit():
async with anyio.create_task_group() as tg:
tg.start_soon(worker)
x = 2 # HIT — tracked normally
assert x == 2On 3.13, both tests show all lines as HIT. The only difference is cancel_scope.cancel().
Add strict-no-cover from pydantic to CI pipeline. This tool identifies `# pragma: no cover` comments on lines that are actually covered by tests, helping keep coverage pragmas accurate. Changes: - Add strict-no-cover as dev dependency (installed from git) - Add `pragma: lax no cover` to coverage exclude_lines for partial coverage - Add CI step to run strict-no-cover after coverage report (Linux only due to a Windows bug in the tool) - Pin coverage to <=7.13 (7.13.1 introduced a regression that breaks strict-no-cover by changing how source paths are reported in .coverage databases) - Remove relative_files=true from [tool.coverage.run] as it interferes with strict-no-cover path matching
Remove `# pragma: no cover` from lines that are actually covered by tests, as detected by strict-no-cover. For lines with partial coverage across platforms or Python versions, use the appropriate pragma: - `# pragma: lax no cover` for platform-specific code (Windows skip markers, sys.platform branches, OS-specific exception handlers) - `# pragma: no branch` for branches where the exit path is never taken (async for loops, anyio.fail_after blocks, finally clauses)
- Move pragma to except line in streamable_http.py per Kludex suggestion - Remove unnecessary lax no cover from test_streamable_http.py assertions (these lines are consistently covered across all CI configurations)
The session ID and protocol version are always set after initialization, so assert their presence rather than silently skipping the headers.
This handler may be triggered in some CI configurations, so use lax to avoid strict-no-cover failures while still excluding from coverage.
Move assertions inside ClientSession blocks where possible. Lines between the two connection phases (clearing state, setting up headers) are not tracked by coverage on Python 3.11 due to a sys.settrace interaction with cancel scopes — mark them with pragma: lax no cover.
287e9f1 to
101fddd
Compare
Lines 1307-1309 (post-cancel assertions) are not reached on Python 3.14 due to cancel scope behavior. The ClientSession async with on line 1327 has an untaken exit branch. Mark accordingly with lax no cover and no branch pragmas.
Add strict-no-cover from pydantic to the CI pipeline. This tool identifies
# pragma: no covercomments on lines that are actually covered by tests, helping keep coverage pragmas accurate and preventing them from accumulating as dead annotations.Motivation
Over time,
# pragma: no coverannotations accumulate on lines that are actually exercised by tests — the original reason for the pragma may no longer apply (e.g., code was refactored, tests were added). These stale pragmas hide real coverage data and make it harder to identify genuinely uncovered code.Changes
CI integration (
e66af17)strict-no-coveras a dev dependency (installed from git)coverage[toml]to>=7.10.7,<=7.13because 7.13.1 introduced a regression in how source paths are reported in.coveragedatabases, breaking strict-no-coverrelative_files = truefrom[tool.coverage.run]as it interferes with strict-no-cover path matchingpragma: lax no coverandpragma: no branchtoexclude_linesin coverage configPragma cleanup (
f0c0b6e)# pragma: no coverfrom ~150 lines that are actually covered by tests# pragma: lax no coverfor platform-specific code (Windowsskipifmarkers,sys.platformbranches, OS-specific exception handlers)# pragma: no branchfor branches where the exit path is never taken (async forloops,anyio.fail_afterblocks,finallyclauses)Pragma guide
# pragma: no cover# pragma: lax no cover# pragma: no branchasync for,with fail_after(), loops that always breakHow it works
After
coverage run -m pytestandcoverage combine, the CI runsstrict-no-coverwhich:.coveragedatabase to find which lines were actually executed# pragma: no coverannotations# pragma: lax no coverlines are excluded from this check, making them safe for code that is covered in some CI configurations but not others.