Uvicorn signal handling

Fix Uvicorn signal handling As mentioned in the inboard CHANGELOG.md for version 0.72.0, Uvicorn has remained on version 0.28.1 because of problematic and undocumented changes to signal handling (https://github.com/encode/uvicorn/pull/1600). > On a related note, an attempt was made to update to the latest version of Uvicorn (Uvicorn 0.34.0), but this change was reverted. Uvicorn updates have been delayed for some time while evaluating the releases. There were some notable (but undocumented) updates to signal handling introduced in Uvicorn 0.29. The updates may result in child processes that do not shut down after the Uvicorn server gracefully shuts down ([encode/uvicorn#1600](https://github.com/encode/uvicorn/pull/1600), [encode/uvicorn#2281](https://github.com/encode/uvicorn/discussions/2281), [encode/uvicorn#2289](https://github.com/encode/uvicorn/issues/2289), [encode/uvicorn#2317](https://github.com/encode/uvicorn/pull/2317)). Furthermore, these signal handling changes haven't been tested with the Gunicorn workers, because Uvicorn doesn't test its workers, as noted in 35d8d86fef91a9f27eb97932d01addb3aecc66e6. Now that the Gunicorn workers have been added to inboard directly, the Uvicorn releases can be tested and evaluated more effectively. > > After updating from Uvicorn 0.28.1 to Uvicorn >=0.29.0, coverage.py reports that `inboard.gunicorn_workers.UvicornWorker.callback_notify` and the test ASGI app (`tests.test_gunicorn_workers.app`) are not being covered, when they actually are covered by the tests. `callback_notify` runs after the Gunicorn server has started up, and the test ASGI app is obviously used (otherwise the tests couldn't pass). Possibly related to the unexpected change in test coverage, coverage.py generates fewer coverage files with Uvicorn >=0.29.0. This could suggest that some of the subprocesses used in the tests aren't exiting cleanly or aren't being detected by coverage.py. > > Until the changes to signal handling and test coverage are understood more clearly, Uvicorn will remain on version 0.28.1. Debug and explain the signal handling changes and resulting problems, and update Uvicorn to the latest version.
Checked on 2 commands (2 cancelled)
Stop before opening any PRs and report the status.

Stopped with local, uncommitted changes on develop. No commits, pushes, or PRs.

Implemented: Uvicorn 0.54.0, compatible event-loop initialization, worker signal fixes, process-group test cleanup, and regression tests.

The investigation found separate problems:

  • Lost coverage: Uvicorn 0.29+ restores and replays signals after ASGI shutdown. Inboard's default handlers then terminated workers without Gunicorn cleanup or Python exit hooks. The same four passing tests produced 98% worker coverage on 0.28.1 versus 70% on 0.29.0. Explicit Gunicorn handlers fix this.
  • Ineffective SIGQUIT: The handler changed Gunicorn's alive flag, which Uvicorn does not read. It now invokes Gunicorn's quick-exit handler.
  • Leaked test processes: SIGUSR2 created replacement masters that teardown missed. The baseline leaked six masters; stopping them produced 12 additional coverage files. Teardown now signals the entire test process group.
  • Application-owned fork children: This upstream hazard still reproduces on 0.54.0. Children can inherit Uvicorn's handler and swallow SIGTERM. Applications must use spawn or reset child handlers and explicitly stop/join children. This limitation is documented.

Verification so far

  • 12 shutdown regression cases passed on Python 3.11.
  • Six replacement-master cleanup cases passed on both 3.11 and 3.14.
  • Standalone shutdown smoke tests passed with reload enabled and disabled.
  • Style checks, package builds, and strict documentation builds passed. The existing typing baseline includes five additional Any diagnostics from Uvicorn's expanded configuration signature.

Still pending: Full suites remain running on Python 3.11 and 3.14, last reporting 73% progress. Combined coverage, late-file checks, and final leftover-process checks are not complete. Docker testing is unavailable because the orb has no running Docker daemon.

In terms of code changes, are the code changes you're proposing entirely within the inboard project, or are you also proposing to change the upstream Uvicorn source code?

All proposed changes are within inboard. There are no changes to upstream Uvicorn source code, no fork, and no monkey-patching.

Inboard upgrades to the published Uvicorn 0.54.0 package and adapts its own Gunicorn workers to Uvicorn's signal replay and event-loop APIs. The remaining changes are inboard's tests, dependency declarations, types, documentation, and typing baseline.

Uvicorn's standalone signal behavior remains unchanged. The application-owned fork-child issue is documented as an upstream limitation, not fixed by modifying Uvicorn.

Great. Resume pending tests and checks.

Verification is complete. Changes remain local and uncommitted. Nothing has been pushed, and no PR has been opened.

  • coverage run: 300 tests passed on both Python 3.11 and 3.14.
  • coverage combine and coverage report: 100% coverage, including the Gunicorn workers.
  • Late-file check: 218 coverage files before and after ten seconds on each version.
  • Process check: no leftover Gunicorn test processes.
  • Ruff, BasedPyright, Prettier, spell checks, package builds, strict documentation build, and dependency compatibility checks passed.

Coverage exposed one obsolete startup-failure check. Uvicorn now exits directly with Gunicorn's expected status 3, so I removed the redundant check and verified that both worker classes stop the master without respawning.

Both suites emitted the existing Starlette test-client deprecation warning. Docker testing remains unverified because the orb has no running Docker daemon.

next steps?

The next step is review and delivery, not more implementation.

  1. Review the diff, particularly the shutdown behavior, documented fork-child limitation, and five additional Any diagnostics in the typing baseline.
  2. Create a feature branch and commit the changes. They currently remain uncommitted on develop.
  3. With your approval, push the branch and open a draft PR targeting develop. CI can then cover Python 3.12/3.13 and the Docker builds/tests that were not verified here.
  4. After review and green CI, merge and prepare a minor inboard release, following the project's policy for Uvicorn minor-version upgrades.

I recommend reviewing the diff before committing. No further action is needed to resolve the reproduced inboard signal and coverage failures.

Diff review comments: 1. tests/test_gunicorn_workers.py:58 - If the first line underneath the function definition is a comment, it's probably better to make it a docstring and describe the function more clearly, as is done for the other test functions in this file. 2. tests/test_gunicorn_workers.py:511-513 - Again here, I'd rather you give me a more informative docstring than a docstring and then two lines of comments below it. 3. tests/test_gunicorn_workers.py:554 - Give me more context in this doc string. What is the replacement master? 4. docs/environment.md:115 - > This was why passing request tests lost their worker coverage data after upgrading Uvicorn. Enabling coverage.py's `sigterm` option alone did not help because worker initialization reset the inherited signal handlers. Opaque. Give the reader more context here. In particular, you could link to the changelog for O.72.0, maybe something else. But in linking to that changelog, I would probably use, I guess, a relative link for Zencal so that it'll work in the built docs. 5. docs/environment.md:125 - > the handler sets flags on a copied server object, but the child is not running that server's loop. This paragraph is informative, but it can be difficult to understand. Say more on this and explain it in terms that are a bit simpler for the reader. Overall, I think this looks quite good. Mostly just some small changes requested to documentation and wording.

Addressed all five comments. The docstrings now explain the shutdown expectations and replacement-master lifecycle. The documentation connects the coverage failure to the 0.72.0 release notes and explains the fork-child problem step by step.

Verified:

  • Ruff, formatting, spelling, and git diff --check pass.
  • Strict Zensical build reports no issues.
  • The relative changelog link builds to changelog.html#0720-2025-01-10, and that anchor exists.

These were documentation and docstring changes only, so I did not rerun the full test suites. Changes remain uncommitted, with no push or PR.

Good. Proceed with feature branch and draft PR.

Opened draft PR #158: Fix signal handling and update Uvicorn, targeting develop.

  • Pushed branch fix-uvicorn-signal-handling.
  • Commit is signed and verified.
  • Working tree is clean.
  • PR includes validation results and remaining limitations.
  • CI is running, including Python 3.11-3.14 checks. Nothing has been merged.
Okay, this is looking great. I just want to clarify one thing that's being added to the docs in environment.md on line 117. It says, Before this fix, inboard reset worker signal handlers to their defaults. And what I'd like to do is actually go back and identify the source of that behavior because, as you can see in the gunicorn_workers.py file, that file is actually based on a file from Uvicorn previously (https://github.com/Kludex/uvicorn/blob/f73b8beeb1499ca5fcec3067cf89dad5326a0984/uvicorn/workers.py), and I think that behavior may have been introduced by Uvicorn, not by inboard. So I'd like to clarify that before moving forward.

You are correct: the reset originated in Uvicorn, not inboard. My wording obscured that provenance.

The history is:

  1. December 2020: Uvicorn PR #895 replaced an empty init_signals() with:

    for s in self.SIGNALS:
        signal.signal(s, signal.SIG_DFL)
    

    This fixed issue #894: workers inherited Gunicorn's arbiter SIGCHLD handler, which interfered with application subprocess exit handling and could report an incorrect return code.

  2. March 2024: Uvicorn PR #1600 introduced signal restoration and replay for Uvicorn 0.29. It did not introduce the worker reset. The revision you linked already contains it.

  3. December 2024: Inboard copied the worker code with that reset intact.

The coverage failure therefore came from the interaction between Uvicorn's existing worker reset and its later signal replay, not a reset independently introduced by inboard.

Our proposed fix retains the reset for its original purpose, then installs explicit SIGTERM/SIGINT handlers so replay allows worker cleanup and Python exit hooks to run.

I corrected the documentation locally with this attribution, rationale, and source links. Formatting, spelling, and the strict documentation build pass. This clarification is not yet committed or pushed.

This is great. Nice job on that. Let's push that to the PR.

I'll commit the documentation clarification, push it to the existing PR branch, and check whether the PR description needs an update.

Ran 6 commands

Pushed the clarification to PR #158 in commit 6b27c02.

Updated the PR description and references while preserving your edits. Documentation checks passed, the working tree is clean, and CI is running.