Skip to content

fix/move orphan check from connection to worker #19

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jan 21, 2025

Conversation

stanley2058
Copy link
Collaborator

@stanley2058 stanley2058 commented Jan 21, 2025

Move the orphan check and recovery to the worker since a large stream count would make Redis halt.

The current process is as follows:

  1. When adding a message to a stream, if it exists and its length is greater than 100, the stream would be put into a checklist.
  2. When the workers are performing a persisting operation, the one with the smallest ID (sorted by alphabet) is elected as the leader and performs the recovery operation.
    • We only want one worker to do this to avoid race conditions.
  3. All workers would maintain a list of streams they are processing in a Redis set to speed up the lookup. The set would also have a TTL to avoid zombie entry if the worker exited.

@Yukaii
Copy link
Member

Yukaii commented Jan 21, 2025

Also getting error when running tests

  @y/redis/api/worker: adding 0 orphans back to worker +104ms
  ClientClosedError: The client is closed
      at Commander._RedisClient_sendCommand (/y-socketio-redis/node_modules/.pnpm/@[email protected]/node_modules/@redis/client/dist/lib/client/index.js:510:31)
      at Commander.commandsExecutor (/y-socketio-redis/node_modules/.pnpm/@[email protected]/node_modules/@redis/client/dist/lib/client/index.js:190:154)
      at BaseClass.<computed> [as expire] (/y-socketio-redis/node_modules/.pnpm/@[email protected]/node_modules/@redis/client/dist/lib/commander.js:8:29)
      at Api.consumeWorkerQueue (file:///y-socketio-redis/src/api.js:288:22)
      at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
      at async file:///y-socketio-redis/src/api.js:369:37

@Yukaii Yukaii merged commit cbd2657 into main Jan 21, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants