Skip to content

server: avoid health API panic before etcd starts#10982

Merged
ti-chi-bot[bot] merged 1 commit into
tikv:masterfrom
rleungx:fix-health-api-nil-etcd-client
Jul 6, 2026
Merged

server: avoid health API panic before etcd starts#10982
ti-chi-bot[bot] merged 1 commit into
tikv:masterfrom
rleungx:fix-health-api-nil-etcd-client

Conversation

@rleungx

@rleungx rleungx commented Jul 6, 2026

Copy link
Copy Markdown
Member

What problem does this PR solve?

Issue Number: Close #10978

What is changed and how does it work?

The PD health API can be called while PD is serving HTTP requests but the etcd client is not ready yet. In that state, member lookup could dereference a nil etcd client and panic.

This change makes member lookup return ErrEtcdNotStarted when the etcd client is nil, and makes the health API return HTTP 503 for unavailable startup state instead of HTTP 500 or a process-level panic. It also documents the 503 response in the API annotation.

Check List

Tests

  • Unit test
  • Integration test

Code changes

Release note

Fix a PD health API panic that could happen during cluster startup or restart.

@ti-chi-bot ti-chi-bot Bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. do-not-merge/needs-triage-completed dco-signoff: yes Indicates the PR's author has signed the dco. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jul 6, 2026
@coderabbitai

coderabbitai Bot commented Jul 6, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@rleungx, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 31 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 969dc081-9691-40ac-9cfd-b53636e3f6fd

📥 Commits

Reviewing files that changed from the base of the PR and between 14c2449 and d9e9f03.

📒 Files selected for processing (3)
  • server/api/health.go
  • server/api/health_test.go
  • server/cluster/cluster.go
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Signed-off-by: Ryan Leung <rleungx@gmail.com>
@rleungx

rleungx commented Jul 6, 2026

Copy link
Copy Markdown
Member Author

/test pull-error-log-review

@rleungx

rleungx commented Jul 6, 2026

Copy link
Copy Markdown
Member Author

/test pull-unit-test-next-gen-2
/test pull-unit-test-next-gen-3

1 similar comment
@rleungx

rleungx commented Jul 6, 2026

Copy link
Copy Markdown
Member Author

/test pull-unit-test-next-gen-2
/test pull-unit-test-next-gen-3

@rleungx

rleungx commented Jul 6, 2026

Copy link
Copy Markdown
Member Author

/test pull-unit-test-next-gen-3

@codecov

codecov Bot commented Jul 6, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.27%. Comparing base (8abdf30) to head (d9e9f03).
⚠️ Report is 4 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master   #10982   +/-   ##
=======================================
  Coverage   79.26%   79.27%           
=======================================
  Files         541      541           
  Lines       75674    75688   +14     
=======================================
+ Hits        59985    59999   +14     
+ Misses      11467    11466    -1     
- Partials     4222     4223    +1     
Flag Coverage Δ
unittests 79.27% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@rleungx

rleungx commented Jul 6, 2026

Copy link
Copy Markdown
Member Author

/test pull-unit-test-next-gen-2

@rleungx

rleungx commented Jul 6, 2026

Copy link
Copy Markdown
Member Author

/test pull-unit-test-next-gen-3

@rleungx rleungx requested review from bufferflies and lhy1024 July 6, 2026 10:08
@ti-chi-bot ti-chi-bot Bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Jul 6, 2026
@ti-chi-bot

ti-chi-bot Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bufferflies, lhy1024

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:
  • OWNERS [bufferflies,lhy1024]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jul 6, 2026
@ti-chi-bot

ti-chi-bot Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

[LGTM Timeline notifier]

Timeline:

  • 2026-07-06 11:34:14.567262265 +0000 UTC m=+22240.603357311: ☑️ agreed by bufferflies.
  • 2026-07-06 14:02:54.189280338 +0000 UTC m=+31160.225375394: ☑️ agreed by lhy1024.

@ti-chi-bot ti-chi-bot Bot merged commit c3fc28a into tikv:master Jul 6, 2026
40 of 43 checks passed
@rleungx rleungx deleted the fix-health-api-nil-etcd-client branch July 7, 2026 02:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved dco-signoff: yes Indicates the PR's author has signed the dco. lgtm release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[PD] Health API can panic with nil pointer dereference during cluster startup/restart

3 participants