Repository navigation
[3.4] Release the list each push notification is sized from, release 3.4.2 - #251
Merged
Merged
Conversation
Backport of #239 to the v3.4 branch. The Python 3.15 CI hunk from that commit is left out, so this branch keeps its existing CI pins. PushNotificationType_New sizes the new notification by splicing a freshly built list into it, but PyList_SetSlice only borrows the items it is handed. The reference was never dropped, so every RESP3 push notification stranded one list object for the lifetime of the process. Under RESP3 this is unbounded: push notifications carry pub/sub deliveries and client-side caching invalidations, so the leak grows with message volume rather than with the number of connections. Feeding 200k notifications to one reader strands 200k lists and roughly 20MB. Dropping the reference also removes an allocation from the path, which makes parsing a small notification about 10% faster. Also guards tryParentize against a NULL object. Every create* callback can hand it a NULL when allocation fails, and the MAP branch would dereference that via PyDict_SetItem and Py_DECREF. Returning NULL up front makes hiredis report REDIS_ERR_OOM instead of crashing. Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Vladyslav Vildanov <117659936+vladvildanov@users.noreply.github.com> Co-authored-by: Igor Malinovskiy <igor.malinovskiy@redis.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #239 to the
v3.4branch, plus version bump to 3.4.2.The leak
PushNotificationType_Newsizes a newPushNotificationby splicing a freshly built list into it:PyList_SetSliceincrements a reference for each element it copies, but it does not steal a reference to the list it is handed. Nothing ever drops it, so every RESP3 push notification strands one list object for the lifetime of the process.Under RESP3 this is unbounded and grows with message volume rather than with the number of connections, since push notifications carry pub/sub deliveries and client-side caching invalidations. A subscriber holding one long-lived reader accumulates the leak indefinitely.
The fix holds the list in a local and releases it after the splice. It also checks
PyList_Newfor failure — on 3.4.1 aNULLlist isPyList_SetSlice's signal to delete the slice, and deleting the empty range at the end of a list is a no-op returning0, so the existingres == -1check could not fire on allocation failure and the parser went on to writesizeelements past the end of a list with no item array. The companion guard intryParentizereturns early on aNULLreply so that failure is reported asREDIS_ERR_OOM.What was left out
Commit 9b16075 also bumped
.github/workflows/integration.yamlto3.15.0-rc.2anddev_requirements.txttomemray==1.20.0. Those hunks are not in this backport —v3.4keeps its existing pins (3.15.0-beta.1,memray==1.19.2). If the 3.15 jobs fail here it is that pre-existing pin, not this change.src/reader.candtests/are byte-identical betweenv3.4and 9b16075's parent, so the cherry-pick is exact:git diff HEAD 9b16075 -- src/reader.c tests/test_reader.pyis empty.Verification
Locally on CPython 3.11.15:
build_ext -UNDEBUG.test_push_notification_releases_backing_listfails against unpatched 3.4.1 — measured delta of exactly 512 stranded lists for 512 notifications — and passes after the fix.gc.set_threshold(1, 1, 1)for notification sizes 0, 1, 4 and 64 — all 0. The empty notification matters there because it takes theob_item == NULLpath.setup.py sdist/bdist_wheelproducehiredis-3.4.2.tar.gzand a 3.4.2 wheel;hiredis.__version__reports3.4.2.🤖 Generated with Claude Code
Note
Medium Risk
Fixes a production memory leak in the C extension’s RESP3 parser with small behavioral changes on allocation failure; low chance of regressions for successful parses.
Overview
Release 3.4.2 backports a fix for unbounded memory growth when parsing RESP3 push notifications (pub/sub, client-side cache invalidations, etc.).
PushNotificationType_NewusedPyList_SetSlicewith a freshly allocated list but never released that list—PyList_SetSliceborrows it, so each push notification leaked one Python list for the process lifetime. The change keeps the sizing list in a local,Py_DECREFs it after the splice, and bails out cleanly ifPyList_Newfails (avoiding a prior path where a failed allocation could still look successful and corrupt the notification list).tryParentizenow returns early on aNULLchild so OOM propagates asREDIS_ERR_OOM.A CPython-only regression test counts live
listobjects viagcto ensure repeated push parses do not strand backing lists.Reviewed by Cursor Bugbot for commit 52930bb. Bugbot is set up for automated code reviews on this repo. Configure here.