Skip to content

[3.4] Release the list each push notification is sized from, release 3.4.2 - #251

Merged
uglide merged 2 commits into
v3.4from
im/backport-239-v3.4
Sep 22, 2026
Merged

uglide merged 2 commits into
v3.4from
im/backport-239-v3.4

Conversation

@uglide

@uglide uglide commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Backport of #239 to the v3.4 branch, plus version bump to 3.4.2.

The leak

PushNotificationType_New sizes a new PushNotification by splicing a freshly built list into it:

int res = PyList_SetSlice(obj, PY_SSIZE_T_MAX, PY_SSIZE_T_MAX, PyList_New(size));

PyList_SetSlice increments a reference for each element it copies, but it does not steal a reference to the list it is handed. Nothing ever drops it, so every RESP3 push notification strands one list object for the lifetime of the process.

Under RESP3 this is unbounded and grows with message volume rather than with the number of connections, since push notifications carry pub/sub deliveries and client-side caching invalidations. A subscriber holding one long-lived reader accumulates the leak indefinitely.

The fix holds the list in a local and releases it after the splice. It also checks PyList_New for failure — on 3.4.1 a NULL list is PyList_SetSlice's signal to delete the slice, and deleting the empty range at the end of a list is a no-op returning 0, so the existing res == -1 check could not fire on allocation failure and the parser went on to write size elements past the end of a list with no item array. The companion guard in tryParentize returns early on a NULL reply so that failure is reported as REDIS_ERR_OOM.

What was left out

Commit 9b16075 also bumped .github/workflows/integration.yaml to 3.15.0-rc.2 and dev_requirements.txt to memray==1.20.0. Those hunks are not in this backport — v3.4 keeps its existing pins (3.15.0-beta.1, memray==1.19.2). If the 3.15 jobs fail here it is that pre-existing pin, not this change.

src/reader.c and tests/ are byte-identical between v3.4 and 9b16075's parent, so the cherry-pick is exact: git diff HEAD 9b16075 -- src/reader.c tests/test_reader.py is empty.

Verification

Locally on CPython 3.11.15:

  • Extension builds clean as 3.4.2, both normally and with build_ext -UNDEBUG.
  • Full suite passes, 65/65 (64 at 3.4.1 plus the new regression test).
  • test_push_notification_releases_backing_list fails against unpatched 3.4.1 — measured delta of exactly 512 stranded lists for 512 notifications — and passes after the fix.
  • Direct reproducer from Release the list each push notification is sized from #239: 100,000 four-element notifications strand 0 lists (was 100,000).
  • Re-run under gc.set_threshold(1, 1, 1) for notification sizes 0, 1, 4 and 64 — all 0. The empty notification matters there because it takes the ob_item == NULL path.
  • setup.py sdist / bdist_wheel produce hiredis-3.4.2.tar.gz and a 3.4.2 wheel; hiredis.__version__ reports 3.4.2.

🤖 Generated with Claude Code


Note

Medium Risk
Fixes a production memory leak in the C extension’s RESP3 parser with small behavioral changes on allocation failure; low chance of regressions for successful parses.

Overview
Release 3.4.2 backports a fix for unbounded memory growth when parsing RESP3 push notifications (pub/sub, client-side cache invalidations, etc.).

PushNotificationType_New used PyList_SetSlice with a freshly allocated list but never released that list—PyList_SetSlice borrows it, so each push notification leaked one Python list for the process lifetime. The change keeps the sizing list in a local, Py_DECREFs it after the splice, and bails out cleanly if PyList_New fails (avoiding a prior path where a failed allocation could still look successful and corrupt the notification list). tryParentize now returns early on a NULL child so OOM propagates as REDIS_ERR_OOM.

A CPython-only regression test counts live list objects via gc to ensure repeated push parses do not strand backing lists.

Reviewed by Cursor Bugbot for commit 52930bb. Bugbot is set up for automated code reviews on this repo. Configure here.

mmick78 and others added 2 commits September 22, 2026 11:09
Backport of #239 to the v3.4 branch. The Python 3.15 CI hunk from that
commit is left out, so this branch keeps its existing CI pins.

PushNotificationType_New sizes the new notification by splicing a freshly
built list into it, but PyList_SetSlice only borrows the items it is handed.
The reference was never dropped, so every RESP3 push notification stranded
one list object for the lifetime of the process.

Under RESP3 this is unbounded: push notifications carry pub/sub deliveries
and client-side caching invalidations, so the leak grows with message volume
rather than with the number of connections. Feeding 200k notifications to one
reader strands 200k lists and roughly 20MB.

Dropping the reference also removes an allocation from the path, which makes
parsing a small notification about 10% faster.

Also guards tryParentize against a NULL object. Every create* callback can
hand it a NULL when allocation fails, and the MAP branch would dereference
that via PyDict_SetItem and Py_DECREF. Returning NULL up front makes hiredis
report REDIS_ERR_OOM instead of crashing.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Vladyslav Vildanov <117659936+vladvildanov@users.noreply.github.com>
Co-authored-by: Igor Malinovskiy <igor.malinovskiy@redis.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@uglide
uglide merged commit b1b2951 into v3.4 Sep 22, 2026
38 of 40 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants