The replicas are dropped when adding new disks into a existing JBOD volume

[boqu]

Description

When adding a new disk to an existing JBOD storage policy, the operator incorrectly treats the missing PVC for the new volume as data loss and executes SYSTEM DROP REPLICA, which removes the replica's ZooKeeper
state (including log_ptr).

Since we set None for both replica and shard in schemaPolicy, so the operator won't try to do any recovery.

Root Cause

It seems the root cause is that the PVC reconciliation flow misclassifies a new volume as data loss:

  // stsReconcileOpts, migrateTableOpts = w.hostPVCsDataVolumeMissedDetectedOptions(host)
  stsReconcileOpts, migrateTableOpts = w.hostPVCsDataLossDetectedOptions(host)

See

clickhouse-operator/pkg/controller/chi/worker-reconciler-chi.go

Line 822 in d342fca

// stsReconcileOpts, migrateTableOpts = w.hostPVCsDataVolumeMissedDetectedOptions(host)

Any idea why we don't use hostPVCsDataVolumeMissedDetectedOptions? Any edge case it won't handle?

Steps to Reproduce

1. Deploy a ClickHouseInstallation with a JBOD storage policy containing one or more disks
2. Add a new disk to the JBOD volume in the CHI spec
3. Observe operator logs showing SYSTEM DROP REPLICA being executed
4. Verify ZooKeeper state (log_ptr, etc.) is removed for the affected replicas

[alex-zaitsev]

@boqu , what is the operator version you are using? Could you provide an example manifest before and after adding a volume?

[boqu]

Hi @alex-zaitsev, please find the manifest before and after. The operator version is 0.26.0.

Before

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "shared-cluster"
  namespace: clickhouse-distributed
  labels:
    app: clickhouse-server
spec:
  defaults:
    distributedDDL:
      profile: default
    storageManagement:
      # Specify PVC provisioner.
      # 1. StatefulSet. PVC would be provisioned by the StatefulSet
      # 2. Operator. PVC would be provisioned by the operator
      provisioner: StatefulSet

      # Specify PVC reclaim policy.
      # 1. Retain. Keep PVC from being deleted
      #    Retaining PVC will also keep backing PV from deletion. This is useful in case we need to keep data intact.
      # 2. Delete
      reclaimPolicy: Retain

    templates:
      hostTemplate: host-template-secure-only
      podTemplate: pod-template
      serviceTemplate: service-template

  restart: ""

  reconciling:
    policy: "nowait"
    configMapPropagationTimeout: 90
    cleanup:
      unknownObjects:
        statefulSet: Delete
        pvc: Delete
        configMap: Delete
        service: Delete
      reconcileFailedObjects:
        statefulSet: Retain
        pvc: Retain
        configMap: Retain
        service: Retain

  configuration:
    files:
      config.d/log_rotation.xml: |-
        <clickhouse>
          <logger>
            <!-- Possible levels: https://github.com/pocoproject/poco/blob/devel/Foundation/include/Poco/Logger.h#L439 -->
            <level>warning</level>
            <log>/var/log/clickhouse-server/clickhouse-server.log</log>
            <errorlog>/var/log/clickhouse-server/clickhouse-server.err.log</errorlog>
            <size>1000M</size>
            <count>10</count>
            <!-- Default behavior is autodetection (log to console if not daemon mode and is tty) -->
            <console>1</console>
            <formatting>
              <type>json</type>
            </formatting>
          </logger>
        </clickhouse>

      config.d/z_log_disable.xml: |
        <clickhouse>
          <opentelemetry_span_log remove="1"/>
          <processors_profile_log remove="1"/>
        </clickhouse>

      config.d/storage_policy_jbod.xml: |
        <clickhouse>        
            <merge_tree>
                <min_bytes_to_rebalance_partition_over_jbod>1073741824</min_bytes_to_rebalance_partition_over_jbod>
            </merge_tree>
            <storage_configuration>
                <disks>
                    <default>
                        <keep_free_space_bytes>536870912</keep_free_space_bytes>
                    </default>
                    <disk_data_1>
                      <type>local</type>
                      <keep_free_space_bytes>536870912</keep_free_space_bytes>
                      <path>/mnt/data/disk_local_1/</path>
                    </disk_data_1>
                    <disk_data_2>
                      <type>local</type>
                      <keep_free_space_bytes>536870912</keep_free_space_bytes>
                      <path>/mnt/data/disk_local_2/</path>
                    </disk_data_2>
                    <disk_s3>
                        <data_cache_enabled>true</data_cache_enabled>
                        <data_cache_max_size>1073741824</data_cache_max_size>
                        <endpoint>****</endpoint>
                        <type>s3</type>
                        <use_environment_credentials>true</use_environment_credentials>
                    </disk_s3>
                    <disk_s3_cache>
                        <disk>disk_s3</disk>
                        <max_size>700Gi</max_size>
                        <path>/mnt/fast-disks/</path>
                        <type>cache</type>
                    </disk_s3_cache>
                    <disk_s3_cache_encrypted>
                        <algorithm>AES_256_CTR</algorithm>
                        <disk>disk_s3_cache</disk>
                        <key>****</key>
                        <type>encrypted</type>
                    </disk_s3_cache_encrypted>
                </disks>
                <policies>
                    <s3tiered>
                        <move_factor>0.100000</move_factor>
                        <perform_ttl_move_on_insert>false</perform_ttl_move_on_insert>
                        <volumes>
                            <n_1_hot>
                                <disk>disk_data_1</disk>
                                <disk>disk_data_2</disk>
                                <max_data_part_size_bytes>214748364800</max_data_part_size_bytes>
                            </n_1_hot>
                            <n_2_cold>
                                <disk>disk_s3_cache</disk>
                            </n_2_cold>
                        </volumes>
                    </s3tiered>
                </policies>
            </storage_configuration>
        </clickhouse>

    clusters:
      - name: "example"

        # Insecure communication.
        # Opens/Closes insecure ports
        insecure: "True"

        # Secure communication.
        # Opens/Closes secure ports
        # Translates into <secure>1</secure> ClickHouse setting for remote replicas
        secure: "False"

        # Shared secret value to secure cluster communications
        secret:
          # Auto-generate shared secret value to secure cluster communications
          auto: "True"

        schemaPolicy:
          replica: None
          shard: None

        layout:
          shardsCount: 2
          replicasCount: 2
    profiles:
      ##############################################################################################################

      # Allow not to specify table engine.
      default/default_table_engine: ReplicatedMergeTree

    zookeeper:
      nodes:
        - host: clickhouse-keeper
          port: 2181
    settings:
      # Max server connections
      max_connections: 16384

  templates:
    hostTemplates:
      - name: host-template-secure-only
        spec:
          # Insecure communication.
          # Opens/Closes insecure ports
          insecure: "yes"

          # Secure communication.
          # Opens/Closes secure ports
          # Translates into <secure>1</secure> ClickHouse setting for remote replicas
          # secure: "yes"

    podTemplates:
      - name: pod-template
        spec:
          serviceAccountName: clickhouse-sa
          containers:
            - image: clickhouse/clickhouse-server:26.2.3.2
              name: clickhouse-pod
              # m8g.4xlarge, 16 CPU, 64 RAM
              resources:
                requests:
                  cpu: "14"
                  memory: "56Gi"
                limits:
                  memory: "60Gi"
              volumeMounts:
                - name: bootstrap-configmap-volume
                  mountPath: /docker-entrypoint-initdb.d
                - name: data-volume-default
                  mountPath: /var/lib/clickhouse
                - name: data-volume-disk-local-1
                  mountPath: /mnt/data/disk_local_1/
                - name: data-volume-disk-local-2
                  mountPath: /mnt/data/disk_local_2/
                - name: data-volume-template-fast-disks
                  mountPath: /mnt/fast-disks/
              command:
                - /bin/bash
                - '-c'
                - chown clickhouse /mnt/data/disk_local_1/ && chown clickhouse /mnt/data/disk_local_2/ && ls -la /mnt/data/ && /entrypoint.sh
          nodeSelector:
            dash0_group: "clickhouse-server-m8g"
          volumes:
            - name: bootstrap-configmap-volume
              configMap:
                name: clickhouse-bootstrap-configmap

    serviceTemplates:
      - name: service-template
        # type ServiceSpec struct from k8s.io/core/v1
        spec:
          ports:
            - name: http
              port: 8123
            - name: https
              port: 8443
            - name: interserver
              port: 9009
          type: ClusterIP
          clusterIP: None

    volumeClaimTemplates:
      ############################################################################################
      # These volumes are used for ClickHouse data storage
      - name: data-volume-default
        # Comment out below to modify gp3 volume specs. E.g. throughput
        # metadata:
        #   annotations:
        #     ebs.csi.aws.com/throughput: "250"
        #     ebs.csi.aws.com/iops: "4000"
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 300Gi
          storageClassName: gp3-encrypted-retained
          volumeMode: Filesystem

      ############################################################################################
      # This volume is used for local SSDs (JBOD disks)
      - name: data-volume-disk-local-1
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 1500Gi
          storageClassName: gp3-encrypted-retained
          volumeMode: Filesystem
      - name: data-volume-disk-local-2
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 1500Gi
          storageClassName: gp3-encrypted-retained
          volumeMode: Filesystem

      ############################################################################################
      # This volume is used for S3 object storage cache
      - name: data-volume-template-fast-disks
        # Comment out below to modify gp3 volume specs. E.g. throughput
        # metadata:
        #   annotations:
        #     ebs.csi.aws.com/throughput: "250"
        #     ebs.csi.aws.com/iops: "4000"
        spec:
          storageClassName: gp3-encrypted-retained
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 800Gi # The local-volume-provisioner will mount the max capacity of local disk. The value we set here doesn't matter.

After

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "shared-cluster"
  namespace: clickhouse-distributed
  labels:
    app: clickhouse-server
spec:
  defaults:
    distributedDDL:
      profile: default
    storageManagement:
      # Specify PVC provisioner.
      # 1. StatefulSet. PVC would be provisioned by the StatefulSet
      # 2. Operator. PVC would be provisioned by the operator
      provisioner: StatefulSet

      # Specify PVC reclaim policy.
      # 1. Retain. Keep PVC from being deleted
      #    Retaining PVC will also keep backing PV from deletion. This is useful in case we need to keep data intact.
      # 2. Delete
      reclaimPolicy: Retain

    templates:
      hostTemplate: host-template-secure-only
      podTemplate: pod-template
      serviceTemplate: service-template

  restart: ""

  reconciling:
    policy: "nowait"
    configMapPropagationTimeout: 90
    cleanup:
      unknownObjects:
        statefulSet: Delete
        pvc: Delete
        configMap: Delete
        service: Delete
      reconcileFailedObjects:
        statefulSet: Retain
        pvc: Retain
        configMap: Retain
        service: Retain

  configuration:
    files:
      config.d/log_rotation.xml: |-
        <clickhouse>
          <logger>
            <!-- Possible levels: https://github.com/pocoproject/poco/blob/devel/Foundation/include/Poco/Logger.h#L439 -->
            <level>warning</level>
            <log>/var/log/clickhouse-server/clickhouse-server.log</log>
            <errorlog>/var/log/clickhouse-server/clickhouse-server.err.log</errorlog>
            <size>1000M</size>
            <count>10</count>
            <!-- Default behavior is autodetection (log to console if not daemon mode and is tty) -->
            <console>1</console>
            <formatting>
              <type>json</type>
            </formatting>
          </logger>
        </clickhouse>

      config.d/z_log_disable.xml: |
        <clickhouse>
          <opentelemetry_span_log remove="1"/>
          <processors_profile_log remove="1"/>
        </clickhouse>

      config.d/storage_policy_jbod.xml: |
        <clickhouse>        
            <merge_tree>
                <min_bytes_to_rebalance_partition_over_jbod>1073741824</min_bytes_to_rebalance_partition_over_jbod>
            </merge_tree>
            <storage_configuration>
                <disks>
                    <default>
                        <keep_free_space_bytes>536870912</keep_free_space_bytes>
                    </default>
                    <disk_data_1>
                      <type>local</type>
                      <keep_free_space_bytes>536870912</keep_free_space_bytes>
                      <path>/mnt/data/disk_local_1/</path>
                    </disk_data_1>
                    <disk_data_2>
                      <type>local</type>
                      <keep_free_space_bytes>536870912</keep_free_space_bytes>
                      <path>/mnt/data/disk_local_2/</path>
                    </disk_data_2>
                    <disk_data_3>
                      <type>local</type>
                      <keep_free_space_bytes>536870912</keep_free_space_bytes>
                      <path>/mnt/data/disk_local_3/</path>
                    </disk_data_3>
                    <disk_s3>
                        <data_cache_enabled>true</data_cache_enabled>
                        <data_cache_max_size>1073741824</data_cache_max_size>
                        <endpoint>****</endpoint>
                        <type>s3</type>
                        <use_environment_credentials>true</use_environment_credentials>
                    </disk_s3>
                    <disk_s3_cache>
                        <disk>disk_s3</disk>
                        <max_size>700Gi</max_size>
                        <path>/mnt/fast-disks/</path>
                        <type>cache</type>
                    </disk_s3_cache>
                    <disk_s3_cache_encrypted>
                        <algorithm>AES_256_CTR</algorithm>
                        <disk>disk_s3_cache</disk>
                        <key>********</key>
                        <type>encrypted</type>
                    </disk_s3_cache_encrypted>
                </disks>
                <policies>
                    <s3tiered>
                        <move_factor>0.100000</move_factor>
                        <perform_ttl_move_on_insert>false</perform_ttl_move_on_insert>
                        <volumes>
                            <n_1_hot>
                                <disk>disk_data_1</disk>
                                <disk>disk_data_2</disk>
                                <disk>disk_data_3</disk>
                                <max_data_part_size_bytes>214748364800</max_data_part_size_bytes>
                            </n_1_hot>
                            <n_2_cold>
                                <disk>disk_s3_cache</disk>
                            </n_2_cold>
                        </volumes>
                    </s3tiered>
                </policies>
            </storage_configuration>
        </clickhouse>

    clusters:
      - name: "example"

        # Insecure communication.
        # Opens/Closes insecure ports
        insecure: "True"

        # Secure communication.
        # Opens/Closes secure ports
        # Translates into <secure>1</secure> ClickHouse setting for remote replicas
        secure: "False"

        # Shared secret value to secure cluster communications
        secret:
          # Auto-generate shared secret value to secure cluster communications
          auto: "True"

        schemaPolicy:
          replica: None
          shard: None

        layout:
          shardsCount: 2
          replicasCount: 2
    profiles:
      ##############################################################################################################

      # Allow not to specify table engine.
      default/default_table_engine: ReplicatedMergeTree

    zookeeper:
      nodes:
        - host: clickhouse-keeper
          port: 2181
    settings:
      # Max server connections
      max_connections: 16384

  templates:
    hostTemplates:
      - name: host-template-secure-only
        spec:
          # Insecure communication.
          # Opens/Closes insecure ports
          insecure: "yes"

          # Secure communication.
          # Opens/Closes secure ports
          # Translates into <secure>1</secure> ClickHouse setting for remote replicas
          # secure: "yes"

    podTemplates:
      - name: pod-template
        spec:
          serviceAccountName: clickhouse-sa
          containers:
            - image: clickhouse/clickhouse-server:26.2.3.2
              name: clickhouse-pod
              # m8g.4xlarge, 16 CPU, 64 RAM
              resources:
                requests:
                  cpu: "14"
                  memory: "56Gi"
                limits:
                  memory: "60Gi"
              volumeMounts:
                - name: bootstrap-configmap-volume
                  mountPath: /docker-entrypoint-initdb.d
                - name: data-volume-default
                  mountPath: /var/lib/clickhouse
                - name: data-volume-disk-local-1
                  mountPath: /mnt/data/disk_local_1/
                - name: data-volume-disk-local-2
                  mountPath: /mnt/data/disk_local_2/
                - name: data-volume-disk-local-3
                  mountPath: /mnt/data/disk_local_3/                  
                - name: data-volume-template-fast-disks
                  mountPath: /mnt/fast-disks/
              command:
                - /bin/bash
                - '-c'
                - chown clickhouse /mnt/data/disk_local_1/ && chown clickhouse /mnt/data/disk_local_2/ && ls -la /mnt/data/ && /entrypoint.sh
          nodeSelector:
            dash0_group: "clickhouse-server-m8g"
          volumes:
            - name: bootstrap-configmap-volume
              configMap:
                name: clickhouse-bootstrap-configmap

    serviceTemplates:
      - name: service-template
        # type ServiceSpec struct from k8s.io/core/v1
        spec:
          ports:
            - name: http
              port: 8123
            - name: https
              port: 8443
            - name: interserver
              port: 9009
          type: ClusterIP
          clusterIP: None

    volumeClaimTemplates:
      ############################################################################################
      # These volumes are used for ClickHouse data storage
      - name: data-volume-default
        # Comment out below to modify gp3 volume specs. E.g. throughput
        # metadata:
        #   annotations:
        #     ebs.csi.aws.com/throughput: "250"
        #     ebs.csi.aws.com/iops: "4000"
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 300Gi
          storageClassName: gp3-encrypted-retained
          volumeMode: Filesystem

      ############################################################################################
      # This volume is used for local SSDs (JBOD disks)
      - name: data-volume-disk-local-1
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 1500Gi
          storageClassName: gp3-encrypted-retained
          volumeMode: Filesystem
      - name: data-volume-disk-local-2
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 1500Gi
          storageClassName: gp3-encrypted-retained
          volumeMode: Filesystem

      - name: data-volume-disk-local-3
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 250Gi
          storageClassName: gp3-encrypted-retained
          volumeMode: Filesystem
      ############################################################################################
      # This volume is used for S3 object storage cache
      - name: data-volume-template-fast-disks
        # Comment out below to modify gp3 volume specs. E.g. throughput
        # metadata:
        #   annotations:
        #     ebs.csi.aws.com/throughput: "250"
        #     ebs.csi.aws.com/iops: "4000"
        spec:
          storageClassName: gp3-encrypted-retained
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 800Gi # The local-volume-provisioner will mount the max capacity of local disk. The value we set here doesn't matter.

[alex-zaitsev]

@boqu, thank you for manifests, those are clear and clean. This is weird, though. The problem that you describe has been fixed a few releases back in https://github.com/Altinity/clickhouse-operator/releases/tag/release-0.25.5

Do you have operator logs by any chance?

[boqu]

@alex-zaitsev I don't have the full logs during that time. But I attached a snapshot, which shows that the operator was dropping the replica as it thought it's a data loss.

Do you remember which PR/commit the fix was introduced?

[alex-zaitsev]

@boqu , do you have a few earlier log records by any chance? That would help to understand why operator decided to drop replica. It logs decisions pretty accurately.

See below related code paths. In fact, we faced this problem only once in years of operation. In that case volume took long time to provision (maybe due to account limits), and PVC was in bad state. That's why we added a guard to check for replica is active. Adding a new volume is a standard operation our customers do quite often. We need to get to the bottom of what has happened in your setup.

• Here where decision is made

  clickhouse-operator/pkg/controller/chi/worker-deleter.go

  Line 362 in a5bdabc

  if w.ensureClusterSchemer(hostToRunOn).IsHostActiveReplica(ctx, hostToRunOn, hostToDrop) {

• Host level check

  clickhouse-operator/pkg/model/chi/schemer/schemer.go

  Line 56 in a5bdabc

  func (s *ClusterSchemer) IsHostActiveReplica(ctx context.Context, hostToRunOn, hostToCheck *api.Host) bool {

• SQL Check for replica is active

  clickhouse-operator/pkg/model/chi/schemer/sql.go

  Line 251 in a5bdabc

  func (s *ClusterSchemer) sqlIsReplicaActive(replica string) string {

  
  9f01ecf

[boqu]

@alex-zaitsev I managed to get more logs.

I0317 03:08:20.680358       1 worker.go:536] createTemplatedCR():CHI:clickhouse-distributed/shared-cluster:CR has an ancestor, use it as a base for reconcile. CR: clickhouse-distributed/shared-cluster
I0317 03:08:20.681572       1 chi.go:57] prepareListOfTemplates():CHI:clickhouse-distributed/shared-cluster:Found applicable templates num: 0
I0317 03:08:20.681589       1 chi.go:36] ApplyTemplates():CHI:clickhouse-distributed/shared-cluster:Applied templates num: 0
I0317 03:08:20.701436       1 statefulset-reconciler.go:103] unknown:Have StatefulSet available, try to perform label-based comparison for sts: clickhouse-distributed/chi-shared-cluster-repl-otel-0-1
I0317 03:08:20.709276       1 statefulset-reconciler.go:103] unknown:Have StatefulSet available, try to perform label-based comparison for sts: clickhouse-distributed/chi-shared-cluster-repl-otel-1-0
I0317 03:08:20.720845       1 statefulset-reconciler.go:103] unknown:Have StatefulSet available, try to perform label-based comparison for sts: clickhouse-distributed/chi-shared-cluster-repl-otel-1-1
I0317 03:08:20.756770       1 chi.go:57] prepareListOfTemplates():CHI:clickhouse-distributed/shared-cluster:Found applicable templates num: 0
I0317 03:08:20.766786       1 statefulset-reconciler.go:103] unknown:Have StatefulSet available, try to perform label-based comparison for sts: clickhouse-distributed/chi-shared-cluster-repl-otel-0-0
I0317 03:08:20.766806       1 object-status.go:54] GetObjectStatusFromMetas():unknown:cur and new objects ARE DIFFERENT based on object version label: Update of the object is required. Object: clickhouse-distributed/chi-shared-cluster-repl-otel-0-0. Cur: b550972041fc286d62cd88925693f43ee21c19c5 New: 37e5a037a40e358208984767e5771ce90dac66cb
I0317 03:08:20.783321       1 statefulset-reconciler.go:103] unknown:Have StatefulSet available, try to perform label-based comparison for sts: clickhouse-distributed/chi-shared-cluster-repl-otel-1-0
I0317 03:08:20.790134       1 statefulset-reconciler.go:103] unknown:Have StatefulSet available, try to perform label-based comparison for sts: clickhouse-distributed/chi-shared-cluster-repl-otel-1-1
I0317 03:08:21.093501       1 worker-reconciler-chi.go:163] CHI:clickhouse-distributed/shared-cluster:
I0317 03:08:21.167053       1 worker.go:355] markReconcileStart():CHI:clickhouse-distributed/shared-cluster:reconcile started, task id: auto-e5f49b88-768f-492f-9561-bf79acd361dd
I0317 03:08:21.299662       1 controller-discoverer.go:91] discoveryConfigMaps():CHI:clickhouse-distributed/shared-cluster:Register ConfigMap in Discovery clickhouse-distributed/chi-shared-cluster-common-configd
I0317 03:08:21.299684       1 controller-discoverer.go:91] discoveryConfigMaps():CHI:clickhouse-distributed/shared-cluster:Register ConfigMap in Discovery clickhouse-distributed/chi-shared-cluster-common-usersd
I0317 03:08:22.090753       1 worker.go:527] Host:1-0[1/0]:clickhouse-distributed/shared-cluster:Host status: modified. Host: ns:clickhouse-distributed|chi:shared-cluster|clu:repl-otel|sha:1|rep:0|host:1-0
I0317 03:08:22.090759       1 worker.go:527] Host:1-1[1/1]:clickhouse-distributed/shared-cluster:Host status: modified. Host: ns:clickhouse-distributed|chi:shared-cluster|clu:repl-otel|sha:1|rep:1|host:1-1
I0317 03:08:22.090780       1 worker-reconciler-chi.go:229] CHI:clickhouse-distributed/shared-cluster:Unable to use full fan-out mode. Counters: modified: 4 . CR: clickhouse-distributed/shared-cluster
I0317 03:08:23.278043       1 worker-reconciler-chi.go:626] worker-reconciler-chi.go:626:reconcileClusterShardsAndHosts():start:reconcileClusterShardsAndHosts start
I0317 03:08:23.278550       1 export.go:83] versionConstraint host: 0-0 version: 26.2.3[26.2.3.2/parsed from the tag: '26.2.3.2'] constraint: 21.* matches: false
I0317 03:08:23.278574       1 worker.go:182] shouldForceRestartHost():Host:0-0[0/0]:clickhouse-distributed/shared-cluster:Config change(s) require host restart. Host: 0-0
I0317 03:08:24.078151       1 worker.go:536] createTemplatedCR():CHI:clickhouse-distributed/shared-cluster:CR has an ancestor, use it as a base for reconcile. CR: clickhouse-distributed/shared-cluster
I0317 03:08:24.078186       1 chi.go:36] ApplyTemplates():CHI:clickhouse-distributed/shared-cluster:Applied templates num: 0
I0317 03:08:24.079443       1 chi.go:36] ApplyTemplates():CHI:clickhouse-distributed/shared-cluster:Applied templates num: 0
I0317 03:08:24.100310       1 statefulset-reconciler.go:103] unknown:Have StatefulSet available, try to perform label-based comparison for sts: clickhouse-distributed/chi-shared-cluster-repl-otel-1-0
I0317 03:08:24.107164       1 object-status.go:54] GetObjectStatusFromMetas():unknown:cur and new objects ARE DIFFERENT based on object version label: Update of the object is required. Object: clickhouse-distributed/chi-shared-cluster-repl-otel-1-1. Cur: 1f77efa150da6f6c185e1ccb6e82b59a6db46ecd New: 9cc4d7031b8045466078ba339953d9d8f1cb7b8d
I0317 03:08:24.278886       1 worker-wait-exclude-include-restart.go:286] descendHostInClickHouseCluster():Host:0-0[0/0]:clickhouse-distributed/shared-cluster:going to descent host. Host/shard/cluster: 0/0/repl-otel
I0317 03:08:25.678580       1 worker.go:536] createTemplatedCR():CHI:clickhouse-distributed/shared-cluster:CR has an ancestor, use it as a base for reconcile. CR: clickhouse-distributed/shared-cluster
I0317 03:08:25.695511       1 object-status.go:54] GetObjectStatusFromMetas():unknown:cur and new objects ARE DIFFERENT based on object version label: Update of the object is required. Object: clickhouse-distributed/chi-shared-cluster-repl-otel-0-1. Cur: 2331afee6875a0539193586e297657a1158e11db New: a67450d21abaa3ff088c77a7ff06d48a98fc45fa
I0317 03:08:25.709885       1 object-status.go:54] GetObjectStatusFromMetas():unknown:cur and new objects ARE DIFFERENT based on object version label: Update of the object is required. Object: clickhouse-distributed/chi-shared-cluster-repl-otel-1-1. Cur: 1f77efa150da6f6c185e1ccb6e82b59a6db46ecd New: 9cc4d7031b8045466078ba339953d9d8f1cb7b8d
I0317 03:08:25.739108       1 worker-reconciler-chi.go:207] logSWVersion():Host:0-0[0/0]:clickhouse-distributed/shared-cluster:Host software version: 0-0 26.2.3[26.2.3.2/parsed from the tag: '26.2.3.2']
I0317 03:08:25.739127       1 worker-reconciler-chi.go:207] logSWVersion():Host:1-0[1/0]:clickhouse-distributed/shared-cluster:Host software version: 1-0 26.2.3[26.2.3.2/parsed from the tag: '26.2.3.2']
I0317 03:08:25.739131       1 worker-reconciler-chi.go:207] logSWVersion():Host:1-1[1/1]:clickhouse-distributed/shared-cluster:Host software version: 1-1 26.2.3[26.2.3.2/parsed from the tag: '26.2.3.2']
I0317 03:08:25.739137       1 worker-reconciler-chi.go:210] logSWVersion():CHI:clickhouse-distributed/shared-cluster:CR software versions min=26.2.3[26.2.3.2/parsed from the tag: '26.2.3.2'] max=26.2.3[26.2.3.2/parsed from the tag: '26.2.3.2']
modified spec items num: 7
diff item [0]:'.Templates.PodTemplates[0].Spec.Containers[0].VolumeMounts[4].Name' = '"data-volume-disk-local-3"'
prev spec items num: 12
diff item [6]:'.Configuration.Clusters[0].Layout.Shards[0].Hosts[0].Runtime.Version' = 'nil'
diff item [6]:'.Templates.VolumeClaimTemplates[3].Spec.VolumeMode' = 'nil'
exclude: app=clickhouse-server,clickhouse.altinity.com/ConfigMap=ChiStorage,clickhouse.altinity.com/app=chop,clickhouse.altinity.com/chi=shared-cluster,clickhouse.altinity.com/namespace=clickhouse-distributed
diff item [1]:'.TaskID' = '"auto-5bcf401c-a4b0-4036-a631-723a9a167bd0"'
diff item [1]:'.Templates.VolumeClaimTemplates[3].Spec.Resources.Requests["storage"].i.value' = '268435456000'
diff item [10]:'.Templates.VolumeClaimTemplates[3].Name' = '"data-volume-template-fast-disks"'
diff item [6]:'.Configuration.Files.data["config.d/storage_policy_jbod.xml"].scalar' = '"<clickhouse>\n    <!-- = 10 * 1024 * 1024 * 1024 (10 GiB) -->\n    <mark_cache_size>10737418240</mark_cache_size>\n    <!-- Increase thread pool sizes to ensure there's sufficient threads available for processing (default: 10_000) -->\n    <max_thread_poo...(value truncated for brevity)'
diff item [0]:'.Templates.VolumeClaimTemplates[3].Name' = '"data-volume-disk-local-3"'

diff item [8]:'.Configuration.Clusters[0].Layout.Shards[0].Hosts[0].Runtime.Version' = 'nil'
diff item [9]:'.Configuration.Clusters[0].Layout.Shards[0].Hosts[1].Runtime.Version' = 'nil'
I0317 03:11:33.001532       1 poller.go:80] Poll():delete StatefulSet: clickhouse-distributed/chi-shared-cluster-repl-otel-0-0:OK delete StatefulSet: clickhouse-distributed/chi-shared-cluster-repl-otel-0-0
I0317 03:11:33.001553       1 statefulset-reconciler.go:537] Host:0-0[0/0]:clickhouse-distributed/shared-cluster:OK delete StatefulSet clickhouse-distributed/chi-shared-cluster-repl-otel-0-0
I0317 03:11:33.006694       1 storage-reconciler.go:283] storage-reconciler.go:283:reconcilePVC():start:Host:0-0[0/0]:clickhouse-distributed/shared-cluster:reconcile PVC (clickhouse-distributed/data-volume-default-chi-shared-cluster-repl-otel-0-0-0/0-0)
I0317 03:11:33.012265       1 storage-pvc.go:71] UpdateOrCreate():unknown:PVC storage was increased externally to greater value and value cannot be decreased, using greater value
I0317 03:11:33.043277       1 storage-reconciler.go:303] storage-reconciler.go:284:reconcilePVC():end:Host:0-0[0/0]:clickhouse-distributed/shared-cluster:reconcile PVC (clickhouse-distributed/data-volume-default-chi-shared-cluster-repl-otel-0-0-0/0-0)
I0317 03:11:33.048570       1 storage-reconciler.go:283] storage-reconciler.go:283:reconcilePVC():start:Host:0-0[0/0]:clickhouse-distributed/shared-cluster:reconcile PVC (clickhouse-distributed/data-volume-disk-local-1-chi-shared-cluster-repl-otel-0-0-0/0-0)
I0317 03:11:33.064323       1 storage-reconciler.go:303] storage-reconciler.go:284:reconcilePVC():end:Host:0-0[0/0]:clickhouse-distributed/shared-cluster:reconcile PVC (clickhouse-distributed/data-volume-disk-local-1-chi-shared-cluster-repl-otel-0-0-0/0-0)
I0317 03:11:33.069274       1 storage-reconciler.go:283] storage-reconciler.go:283:reconcilePVC():start:Host:0-0[0/0]:clickhouse-distributed/shared-cluster:reconcile PVC (clickhouse-distributed/data-volume-disk-local-2-chi-shared-cluster-repl-otel-0-0-0/0-0)
I0317 03:11:33.083869       1 storage-reconciler.go:303] storage-reconciler.go:284:reconcilePVC():end:Host:0-0[0/0]:clickhouse-distributed/shared-cluster:reconcile PVC (clickhouse-distributed/data-volume-disk-local-2-chi-shared-cluster-repl-otel-0-0-0/0-0)
I0317 03:11:33.089201       1 storage-reconciler.go:212] Host:0-0[0/0]:clickhouse-distributed/shared-cluster:PVC (clickhouse-distributed/0-0/data-volume-disk-local-3/data-volume-disk-local-3-chi-shared-cluster-repl-otel-0-0-0) not found - operator should create the model: false
W0317 03:11:33.089228       1 storage-reconciler.go:145] Host:0-0[0/0]:clickhouse-distributed/shared-cluster:PVC is either newly added to the host or was lost earlier (clickhouse-distributed/0-0/data-volume-disk-local-3/data-volume-disk-local-3-chi-shared-cluster-repl-otel-0-0-0)
E0317 03:11:33.089240       1 storage-reconciler.go:164] reconcilePVCFromVolumeMount():Host:0-0[0/0]:clickhouse-distributed/shared-cluster:Unable to reconcile nil PVC: clickhouse-distributed/data-volume-disk-local-3-chi-shared-cluster-repl-otel-0-0-0
I0317 03:11:33.208156       1 storage-reconciler.go:283] storage-reconciler.go:283:reconcilePVC():start:Host:0-0[0/0]:clickhouse-distributed/shared-cluster:reconcile PVC (clickhouse-distributed/data-volume-template-fast-disks-chi-shared-cluster-repl-otel-0-0-0/0-0)
I0317 03:11:33.609568       1 storage-reconciler.go:303] storage-reconciler.go:284:reconcilePVC():end:Host:0-0[0/0]:clickhouse-distributed/shared-cluster:reconcile PVC (clickhouse-distributed/data-volume-template-fast-disks-chi-shared-cluster-repl-otel-0-0-0/0-0)
I0317 03:11:33.609630       1 statefulset-reconciler.go:341] createStatefulSet():Host:0-0[0/0]:clickhouse-distributed/shared-cluster:Create StatefulSet: clickhouse-distributed/chi-shared-cluster-repl-otel-0-0 - started
I0317 03:11:33.807792       1 statefulset-reconciler.go:407] doCreateStatefulSet():Host:0-0[0/0]:clickhouse-distributed/shared-cluster
I0317 03:11:33.807806       1 statefulset-reconciler.go:410] Create StatefulSet: clickhouse-distributed/chi-shared-cluster-repl-otel-0-0
W0317 03:11:33.818458       1 statefulset-reconciler.go:454] waitHostStatefulSetToLaunch():Host:0-0[0/0]:clickhouse-distributed/shared-cluster:Host is not properly launched - no waiting sts at all. Host: 0-0
I0317 03:11:33.863726       1 worker-boilerplate.go:146] processReconcilePod():unknown:Add Pod. clickhouse-distributed/chi-shared-cluster-repl-otel-0-0-0
I0317 03:11:34.410242       1 statefulset-reconciler.go:368] shouldAbortOrContinueCreateStatefulSet():Host:0-0[0/0]:clickhouse-distributed/shared-cluster:Create StatefulSet: clickhouse-distributed/chi-shared-cluster-repl-otel-0-0 - completed
I0317 03:11:34.847001       1 worker-reconciler-chi.go:422] worker-reconciler-chi.go:389:reconcileHostStatefulSet():end:Host:0-0[0/0]:clickhouse-distributed/shared-cluster:reconcile StatefulSet end
I0317 03:11:34.874070       1 storage-reconciler.go:283] storage-reconciler.go:283:reconcilePVC():start:Host:0-0[0/0]:clickhouse-distributed/shared-cluster:reconcile PVC (clickhouse-distributed/data-volume-default-chi-shared-cluster-repl-otel-0-0-0/0-0)
I0317 03:11:35.088591       1 storage-pvc.go:71] UpdateOrCreate():unknown:PVC storage was increased externally to greater value and value cannot be decreased, using greater value
I0317 03:11:35.260999       1 storage-reconciler.go:303] storage-reconciler.go:284:reconcilePVC():end:Host:0-0[0/0]:clickhouse-distributed/shared-cluster:reconcile PVC (clickhouse-distributed/data-volume-default-chi-shared-cluster-repl-otel-0-0-0/0-0)
I0317 03:11:35.408389       1 storage-reconciler.go:283] storage-reconciler.go:283:reconcilePVC():start:Host:0-0[0/0]:clickhouse-distributed/shared-cluster:reconcile PVC (clickhouse-distributed/data-volume-disk-local-1-chi-shared-cluster-repl-otel-0-0-0/0-0)
I0317 03:11:35.821812       1 storage-reconciler.go:303] storage-reconciler.go:284:reconcilePVC():end:Host:0-0[0/0]:clickhouse-distributed/shared-cluster:reconcile PVC (clickhouse-distributed/data-volume-disk-local-1-chi-shared-cluster-repl-otel-0-0-0/0-0)
I0317 03:11:36.008495       1 storage-reconciler.go:283] storage-reconciler.go:283:reconcilePVC():start:Host:0-0[0/0]:clickhouse-distributed/shared-cluster:reconcile PVC (clickhouse-distributed/data-volume-disk-local-2-chi-shared-cluster-repl-otel-0-0-0/0-0)
I0317 03:11:36.413004       1 storage-reconciler.go:303] storage-reconciler.go:284:reconcilePVC():end:Host:0-0[0/0]:clickhouse-distributed/shared-cluster:reconcile PVC (clickhouse-distributed/data-volume-disk-local-2-chi-shared-cluster-repl-otel-0-0-0/0-0)
I0317 03:11:36.607964       1 storage-reconciler.go:283] storage-reconciler.go:283:reconcilePVC():start:Host:0-0[0/0]:clickhouse-distributed/shared-cluster:reconcile PVC (clickhouse-distributed/data-volume-disk-local-3-chi-shared-cluster-repl-otel-0-0-0/0-0)
I0317 03:11:37.017171       1 storage-reconciler.go:303] storage-reconciler.go:284:reconcilePVC():end:Host:0-0[0/0]:clickhouse-distributed/shared-cluster:reconcile PVC (clickhouse-distributed/data-volume-disk-local-3-chi-shared-cluster-repl-otel-0-0-0/0-0)
I0317 03:11:37.215697       1 storage-reconciler.go:283] storage-reconciler.go:283:reconcilePVC():start:Host:0-0[0/0]:clickhouse-distributed/shared-cluster:reconcile PVC (clickhouse-distributed/data-volume-template-fast-disks-chi-shared-cluster-repl-otel-0-0-0/0-0)
I0317 03:11:37.612852       1 storage-reconciler.go:303] storage-reconciler.go:284:reconcilePVC():end:Host:0-0[0/0]:clickhouse-distributed/shared-cluster:reconcile PVC (clickhouse-distributed/data-volume-template-fast-disks-chi-shared-cluster-repl-otel-0-0-0/0-0)
I0317 03:11:37.612918       1 cluster.go:84] Run query on: chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local. of [chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local.]
E0317 03:11:37.613636       1 connection.go:212] QueryContext():FAILED Query(http://clickhouse_operator:***@chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local.:8123) doRequest: transport failed to send a request to ClickHouse: dial tcp: lookup chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local. on 172.20.0.10:53: no such host for SQL: SELECT version()
W0317 03:11:37.613650       1 cluster.go:91] QueryAny():FAILED to run query on: chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local. of [chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local.] skip to next. err: doRequest: transport failed to send a request to ClickHouse: dial tcp: lookup chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local. on 172.20.0.10:53: no such host
E0317 03:11:37.613662       1 cluster.go:95] QueryAny():FAILED to run query on all hosts [chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local.]
W0317 03:11:37.613678       1 worker-app-version.go:37] getHostClickHouseVersion():Host:0-0[0/0]:clickhouse-distributed/shared-cluster:Failed to get ClickHouse version on host: 0-0 err: FAILED to run query on all hosts [chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local.]
W0317 03:11:37.613689       1 worker-app-version.go:54] func1():Host:0-0[0/0]:clickhouse-distributed/shared-cluster:Host is NOT alive: 0-0
I0317 03:11:42.614752       1 cluster.go:84] Run query on: chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local. of [chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local.]
E0317 03:11:42.615500       1 connection.go:212] QueryContext():FAILED Query(http://clickhouse_operator:***@chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local.:8123) doRequest: transport failed to send a request to ClickHouse: dial tcp: lookup chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local. on 172.20.0.10:53: no such host for SQL: SELECT version()
W0317 03:11:42.615515       1 cluster.go:91] QueryAny():FAILED to run query on: chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local. of [chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local.] skip to next. err: doRequest: transport failed to send a request to ClickHouse: dial tcp: lookup chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local. on 172.20.0.10:53: no such host
E0317 03:11:42.615530       1 cluster.go:95] QueryAny():FAILED to run query on all hosts [chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local.]
W0317 03:11:42.615547       1 worker-app-version.go:37] getHostClickHouseVersion():Host:0-0[0/0]:clickhouse-distributed/shared-cluster:Failed to get ClickHouse version on host: 0-0 err: FAILED to run query on all hosts [chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local.]
W0317 03:11:42.615558       1 worker-app-version.go:54] func1():Host:0-0[0/0]:clickhouse-distributed/shared-cluster:Host is NOT alive: 0-0
I0317 03:11:47.616108       1 cluster.go:84] Run query on: chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local. of [chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local.]
E0317 03:11:47.616777       1 connection.go:212] QueryContext():FAILED Query(http://clickhouse_operator:***@chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local.:8123) doRequest: transport failed to send a request to ClickHouse: dial tcp: lookup chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local. on 172.20.0.10:53: no such host for SQL: SELECT version()
W0317 03:11:47.616793       1 cluster.go:91] QueryAny():FAILED to run query on: chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local. of [chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local.] skip to next. err: doRequest: transport failed to send a request to ClickHouse: dial tcp: lookup chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local. on 172.20.0.10:53: no such host
E0317 03:11:47.616806       1 cluster.go:95] QueryAny():FAILED to run query on all hosts [chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local.]
W0317 03:11:47.616823       1 worker-app-version.go:37] getHostClickHouseVersion():Host:0-0[0/0]:clickhouse-distributed/shared-cluster:Failed to get ClickHouse version on host: 0-0 err: FAILED to run query on all hosts [chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local.]
W0317 03:11:47.616836       1 worker-app-version.go:54] func1():Host:0-0[0/0]:clickhouse-distributed/shared-cluster:Host is NOT alive: 0-0
I0317 03:12:52.638531       1 cluster.go:84] Run query on: chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local. of [chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local.]
E0317 03:12:52.639748       1 connection.go:212] QueryContext():FAILED Query(http://clickhouse_operator:***@chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local.:8123) doRequest: transport failed to send a request to ClickHouse: dial tcp 10.10.57.222:8123: connect: connection refused for SQL: SELECT version()
W0317 03:12:52.639761       1 cluster.go:91] QueryAny():FAILED to run query on: chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local. of [chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local.] skip to next. err: doRequest: transport failed to send a request to ClickHouse: dial tcp 10.10.57.222:8123: connect: connection refused
W0317 03:12:52.639788       1 worker-app-version.go:37] getHostClickHouseVersion():Host:0-0[0/0]:clickhouse-distributed/shared-cluster:Failed to get ClickHouse version on host: 0-0 err: FAILED to run query on all hosts [chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local.]
I0317 03:12:52.639810       1 poller.go:117] Poll():clickhouse-distributed/0-0:WAIT: clickhouse-distributed/0-0
I0317 03:12:55.059165       1 chi.go:57] prepareListOfTemplates():CHI:clickhouse-distributed/shared-cluster:Found applicable templates num: 0
I0317 03:12:55.060417       1 chi.go:57] prepareListOfTemplates():CHI:clickhouse-distributed/shared-cluster:Found applicable templates num: 0
I0317 03:12:55.068564       1 statefulset-reconciler.go:103] unknown:Have StatefulSet available, try to perform label-based comparison for sts: clickhouse-distributed/chi-shared-cluster-repl-otel-0-0
I0317 03:12:55.068585       1 object-status.go:47] GetObjectStatusFromMetas():unknown:cur and new objects are equal based on object version label. Update of the object is not required. Object: clickhouse-distributed/chi-shared-cluster-repl-otel-0-0
I0317 03:12:55.087207       1 object-status.go:54] GetObjectStatusFromMetas():unknown:cur and new objects ARE DIFFERENT based on object version label: Update of the object is required. Object: clickhouse-distributed/chi-shared-cluster-repl-otel-1-0. Cur: e1b7c20c13ec4ab189579e9d2740447960cfffee New: be9de6e39cc3ce8fb4d3cf4b721baa1b279e09e9
I0317 03:12:55.094554       1 statefulset-reconciler.go:103] unknown:Have StatefulSet available, try to perform label-based comparison for sts: clickhouse-distributed/chi-shared-cluster-repl-otel-1-1
I0317 03:12:55.094571       1 object-status.go:54] GetObjectStatusFromMetas():unknown:cur and new objects ARE DIFFERENT based on object version label: Update of the object is required. Object: clickhouse-distributed/chi-shared-cluster-repl-otel-1-1. Cur: 1f77efa150da6f6c185e1ccb6e82b59a6db46ecd New: 9cc4d7031b8045466078ba339953d9d8f1cb7b8d
I0317 03:12:55.126667       1 chi.go:57] prepareListOfTemplates():CHI:clickhouse-distributed/shared-cluster:Found applicable templates num: 0
I0317 03:12:55.127830       1 chi.go:36] ApplyTemplates():CHI:clickhouse-distributed/shared-cluster:Applied templates num: 0
I0317 03:12:55.138689       1 object-status.go:47] GetObjectStatusFromMetas():unknown:cur and new objects are equal based on object version label. Update of the object is not required. Object: clickhouse-distributed/chi-shared-cluster-repl-otel-0-0
I0317 03:12:55.157341       1 statefulset-reconciler.go:103] unknown:Have StatefulSet available, try to perform label-based comparison for sts: clickhouse-distributed/chi-shared-cluster-repl-otel-1-0
I0317 03:12:55.157357       1 object-status.go:54] GetObjectStatusFromMetas():unknown:cur and new objects ARE DIFFERENT based on object version label: Update of the object is required. Object: clickhouse-distributed/chi-shared-cluster-repl-otel-1-0. Cur: e1b7c20c13ec4ab189579e9d2740447960cfffee New: be9de6e39cc3ce8fb4d3cf4b721baa1b279e09e9
I0317 03:12:55.473176       1 worker-reconciler-chi.go:207] logSWVersion():Host:0-1[0/1]:clickhouse-distributed/shared-cluster:Host software version: 0-1 26.2.3[26.2.3.2/parsed from the tag: '26.2.3.2']
I0317 03:12:55.473185       1 worker-reconciler-chi.go:207] logSWVersion():Host:1-0[1/0]:clickhouse-distributed/shared-cluster:Host software version: 1-0 26.2.3[26.2.3.2/parsed from the tag: '26.2.3.2']
I0317 03:12:55.473190       1 worker-reconciler-chi.go:207] logSWVersion():Host:1-1[1/1]:clickhouse-distributed/shared-cluster:Host software version: 1-1 26.2.3[26.2.3.2/parsed from the tag: '26.2.3.2']
I0317 03:12:55.476012       1 worker-reconciler-chi.go:163] CHI:clickhouse-distributed/shared-cluster:
I0317 03:12:55.492307       1 worker-config-map.go:70] updateConfigMap():CHI:clickhouse-distributed/shared-cluster:Update ConfigMap clickhouse-distributed/chi-shared-cluster-common-usersd
I0317 03:12:57.640880       1 cluster.go:84] Run query on: chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local. of [chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local.]
I0317 03:12:57.643659       1 worker-app-version.go:41] getHostClickHouseVersion():Host:0-0[0/0]:clickhouse-distributed/shared-cluster:Get ClickHouse version on host: 0-0 version: 26.2.3.2
I0317 03:12:57.643756       1 schemer.go:58] IsHostActiveReplica():Host:0-0[0/0]:clickhouse-distributed/shared-cluster:Check active replica: chi-shared-cluster-repl-otel-0-0 at 0-0
E0317 03:13:01.644035       1 connection.go:212] QueryContext():FAILED Query(http://clickhouse_operator:***@chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local.:8123) doRequest: transport failed to send a request to ClickHouse: context deadline exceeded for SQL: SELECT sumMap(replica_is_active)['chi-shared-cluster-repl-otel-0-0'] FROM system.replicas
W0317 03:13:01.644057       1 cluster.go:91] QueryAny():FAILED to run query on: chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local. of [chi-shared-cluster-repl-otel-0-0.clickhouse-distributed.svc.cluster.local.] skip to next. err: doRequest: transport failed to send a request to ClickHouse: context deadline exceeded
I0317 03:13:01.644088       1 schemer.go:65] IsHostActiveReplica():Host:0-0[0/0]:clickhouse-distributed/shared-cluster:The host 0-0 is active
I0317 03:13:01.644100       1 worker-deleter.go:360] canDropReplica():Replica is an active one. Need ti check whether it can be dropped. hostToDrop: 0-0
I0317 03:13:01.644114       1 worker-deleter.go:367] canDropReplica():Accepted force drop replica upon storage loss. hostToDrop: 0-0

[boqu]

@alex-zaitsev Have you got a chance to take a look at this issue after I provided more logs?