"long" cluster name breaks replication

[Dlahacz]

Hi,
We have discovered that after converting from IP to hostname and using the new replication service, it is very easy to break replication by using a "long" cluster name. It appears that the cluster_*_cluster_incoming_addresses field is limited to 255 characters. If I calculated it correctly, the actual limit for the cluster name is currently 11 characters (assuming there are no more than 10 workers).

6.2.122

| cluster_manticorecustomer62122_cluster_incoming_addresses         | 10.0.128.44:9312,10.0.128.44:9315:replication

6.3.8

| cluster_manticorecustomer6381_cluster_incoming_addresses         | manticorecustomer6381-manticoresearch-worker-0.manticorecustomer6381-manticoresearch-worker-replication-svc.manticorecustomer6381.svc.cluster.local:9312,manticorecustomer6381-manticoresearch-worker-0.manticorecustomer6381-manticoresearch-worker-replicatio |

The ALTER CLUSTER ADD command freezes because it is unable to resolve the truncated hostname (without producing an error, as far as we know).

[Dlahacz]

I would say, since all workers are in same namepsace, we can just use <hosname>.<service_name> as a host (https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#namespaces-of-services). There is no need for <namespace>.svc.cluster.local part and it will increase maximum cluster name lenght to 26.

[djklim87]

Unfortunately, we can’t roll back to using IP addresses. In Kubernetes, IPs are unstable—they may be assigned to one pod now and a different one an hour later.

I see only one solution: we can limit the length of the chart name and namespace name to avoid this issue

[djklim87]

As far as I can see, we’ve already limited .Chart.Name to 63 characters. However, it’s still unclear whether this length is insufficient for your case.

Could you provide a Minimal Reproducible Example (MRE) to help clarify the issue?

[djklim87]

I released improved version without using .svc.cluster.local could you check did it solve the problem

[djklim87]

I also created a cluster with 10 nodes in Kubernetes, and there were no replication issues.

The only limitation I’m aware of is the service name length restriction in Kubernetes, which cannot exceed 253 characters. However, this check is already implemented in our Helm chart

% kubectl -n manticore-helm exec helm-manticoresearch-worker-9 -- mysql -e "insert into manticore_cluster:abc values (0, 'my text')"

% kubectl -n manticore-helm exec helm-manticoresearch-worker-0 -- mysql -e "select * from abc"
+---------------------+---------+
| id                  | t       |
+---------------------+---------+
| 3317718931992477697 | my text |
+---------------------+---------+

kubectl -n manticore-helm exec helm-manticoresearch-worker-8 -- searchd --status | grep incoming_addresses
cluster_manticore_cluster_incoming_addresses: helm-manticoresearch-worker-2.helm-manticoresearch-worker-replication-svc.manticore-helm:9312,helm-manticoresearch-worker-2.helm-manticoresearch-worker-replication-svc.manticore-helm:9315:replication,helm-manticoresearch-worker-3.helm-manticoresearch-worker-replication-svc.manticore-helm:9312,helm-manticoresearch-worker-3.helm-manticoresearch-worker-replication-svc.manticore-helm:9315:replication,helm-manticoresearch-worker-7.helm-manticoresearch-worker-replication-svc.manticore-helm:9312,helm-manticoresearch-worker-7.helm-manticoresearch-worker-replication-svc.manticore-helm:9315:replication,helm-manticoresearch-worker-4.helm-manticoresearch-worker-replication-svc.manticore-helm:9312,helm-manticoresearch-worker-4.helm-manticoresearch-worker-replication-svc.manticore-helm:9315:replication,helm-manticoresearch-worker-8.helm-manticoresearch-worker-replication-svc.manticore-helm:9312,helm-manticoresearch-worker-8.helm-manticoresearch-worker-replication-svc.manticore-helm:9315:replication,helm-manticoresearch-worker-5.helm-manticoresearch-worker-replication-svc.manticore-helm:9312,helm-manticoresearch-worker-5.helm-manticoresearch-worker-replication-svc.manticore-helm:9315:replication,helm-manticoresearch-worker-9.helm-manticoresearch-worker-replication-svc.manticore-helm:9312,helm-manticoresearch-worker-9.helm-manticoresearch-worker-replication-svc.manticore-helm:9315:replication,helm-manticoresearch-worker-0.helm-manticoresearch-worker-replication-svc.manticore-helm:9312,helm-manticoresearch-worker-0.helm-manticoresearch-worker-replication-svc.manticore-helm:9315:replication,helm-manticoresearch-worker-1.helm-manticoresearch-worker-replication-svc.manticore-helm:9312,helm-manticoresearch-worker-1.helm-manticoresearch-worker-replication-svc.manticore-helm:9315:replication,helm-manticoresearch-worker-6.helm-manticoresearch-worker-replication-svc.manticore-helm:9312,helm-manticoresearch-worker-6.helm-manticoresearch-worker-replication-svc.manticore-helm:9315:replication 

So your idea of the incoming_addresses field limited to 255 characters seems incorrect

[Dlahacz]

Oh, we will try to make MRE. In production deployment, we have reduced name lenght and it started working again.