CA-384228: Xapi fail to start on slave when "Synchronising bonds on slave with master" #6817
Conversation
Add sysctl configuration to enable TCP PMTUD on all XenServer hosts. This prevents TCP connection hangs when path MTU is smaller than configured interface MTU (e.g., jumbo frames configured but network infrastructure doesn't support them). Configuration: - net.ipv4.tcp_mtu_probing=1: Enable automatic MTU detection when ICMP blackhole is detected (recommended setting) - net.ipv4.tcp_base_mss=1024: Base MSS for MTU probing The TCP stack will automatically detect packet loss patterns indicating MTU issues and probe down to find the working MTU size. This works even when ICMP Fragmentation Needed messages are blocked by firewalls. Files: - scripts/92-xapi-tcp-mtu.conf: New sysctl configuration file - scripts/Makefile: Install sysctl config to /etc/sysctl.d/ The "92" prefix ensures this loads after basic network configuration (91-net-ipv6.conf) but before local administrator overrides (99-*). Reference: https://blog.cloudflare.com/path-mtu-discovery-in-practice/ Signed-off-by: Gang Ji <[email protected]>
ocaml/xapi/xapi_pool.ml
Outdated
| (* Test MTU connectivity using ping - ICMP-based, informational only *) | ||
| let test_ping size desc = | ||
| try | ||
| let timeout = 3.0 *. 1e9 |> Int64.of_float |> Mtime.Span.of_uint64_ns in |
There was a problem hiding this comment.
Use Mtime.Span.(3 * s) or similar to create a 3s span.
ocaml/xapi/xapi_pool.ml
Outdated
| match (standard_ok, jumbo_ok) with | ||
| | true, false -> | ||
| (* CA-384228 scenario: standard works but jumbo fails *) | ||
| warn |
There was a problem hiding this comment.
Would it make sense to create an alert such that the customer would see it?
There was a problem hiding this comment.
Thanks for the advice, I think it is a great idea, added now, please help to review again, thank you.
There was a problem hiding this comment.
The alert shown in XC after pool join:
ocaml/xapi/xapi_pool.ml
Outdated
| - 1472 = 1500 (standard MTU) - 20 (IP header) - 8 (ICMP header) | ||
| - 8972 = 9000 (jumbo MTU) - 20 (IP header) - 8 (ICMP header) *) | ||
| let standard_mtu_icmp_payload = 1472 in | ||
| let jumbo_mtu_icmp_payload = 8972 in |
There was a problem hiding this comment.
Can we calculate this dynamically based on the actual MTU like mtu - 28, and then try a probe for MTU=1500, and another for the actually configured MTU.
That way even if the user configures an MTU slightly smaller than 9000 it'd work.
Also do we need to take the size of the VLAN tag into account when we're on a VLAN?
gangj
force-pushed
the
private/gangj/CA-384228
branch
from
9d7d3a9 to
1d0e0d4
Compare
Add diagnostic tests during pool join to detect and warn about MTU mismatches, particularly when higher MTU values are configured but the network path doesn't support them. The diagnostics: 1. Query master's management network MTU via RPC 2. Detect VLAN configuration and account for 4-byte overhead 3. Calculate ICMP payload dynamically: MTU - IP header (20) - ICMP header (8) - VLAN (4 if present) 4. Test standard MTU (1500) with ICMP ping 5. Test configured MTU if > 1500 6. Create pool-level alert when CA-384228 scenario detected: - Standard MTU (1500) works - Configured higher MTU fails - This indicates path MTU < configured MTU Key design decisions: - Does NOT block pool join (ICMP may be blocked by firewalls) - Queries master's DB via verified RPC (slave's DB not yet synced) - Called after certificate exchange with verified connection - Creates pool-level alert for customer visibility in XenCenter/CLI - Relies on TCP PMTUD (enabled by sysctl) to handle issues automatically - Diagnostics are informational only, providing visibility The implementation dynamically calculates test packet sizes based on actual configured MTU rather than assuming fixed values, making it work correctly with any MTU configuration (not just jumbo frames). Warning format highlights the issue clearly and references the TCP PMTUD fix that handles it automatically, with guidance for persistent problems. Signed-off-by: Gang Ji <[email protected]>
gangj
force-pushed
the
private/gangj/CA-384228
branch
from
1d0e0d4 to
0d7e423
Compare
| # This is the starting point for MTU probing when enabled | ||
|
|
||
| net.ipv4.tcp_mtu_probing = 1 | ||
| net.ipv4.tcp_base_mss = 1024 |
There was a problem hiding this comment.
configuration files has been a salient point of issues regarding user configuration in xcp-ng. I'm asking the platform teams whether this change follows their recommendations
There was a problem hiding this comment.
Thanks @psafont , I understand your point.
Would you please share more about the good practice or recommendations? Thank you.
There was a problem hiding this comment.
I'm expecting somebody from xcp-ng's platform team to share them here
There was a problem hiding this comment.
I'm not from platform team, but indeed, system configuration files should likely be part of packages like, on xcp-ng side, xcp-ng-release.
- Do we actually need a config file to enforce that?
- Could it be a setting that xapi applies istead? Or would that not be possible to be achieved soon enough for slaves to be able to reach master?
There was a problem hiding this comment.
What happens on xapi updates when this file has been changed manually? Would the next xapi update overwrite any change? We might want to add a line instructing not to change this file.
There was a problem hiding this comment.
If you manually edit a file in /etc/sysctl.d/ that is owned by a package, whether an update overwrites it is controlled by how the package marked that file:
%config(noreplace) → your edited file is kept, and the new packaged version is written as .rpmnew.
%config (without noreplace) → your edited file may be replaced, and your previous version is saved as .rpmsave.Runtime precedence: When sysctl --system is run, settings from /etc/sysctl.d override vendor defaults from /usr/lib/sysctl.d. The last assignment wins if the same key is set multiple times.
There was a problem hiding this comment.
This means, we should write this to /usr/lib/sysctl.d such that a user would put an overwrite into /etc/sysctl.d/?
|
The PR description has " - <title>" and the description " - <another_title>" and the link is not clear for someone outside XenServer. I assume the PR title reference the original issue this fixes and the description one is the solution found to fix it? |
5 participants
CA-384228: Enable TCP Path MTU Discovery by default
Add sysctl configuration to enable TCP PMTUD on all XenServer hosts.
This prevents TCP connection hangs when path MTU is smaller than configured
interface MTU (e.g., jumbo frames configured but network infrastructure
doesn't support them).
Configuration:
blackhole is detected (recommended setting)
The TCP stack will automatically detect packet loss patterns indicating
MTU issues and probe down to find the working MTU size. This works even
when ICMP Fragmentation Needed messages are blocked by firewalls.
Files:
The "92" prefix ensures this loads after basic network configuration
(91-net-ipv6.conf) but before local administrator overrides (99-*).
Reference: https://blog.cloudflare.com/path-mtu-discovery-in-practice/
===
CA-384228: Add MTU diagnostics during pool join
Add diagnostic tests during pool join to detect and warn about MTU
mismatches, particularly when jumbo frames are configured but the
network path doesn't support them.
The diagnostics:
Key design decisions:
Warning format highlights the issue clearly and references the
TCP PMTUD fix that handles it automatically, with guidance for
persistent problems.