[BUG] ISM policy doens’t apply

[tantra35]

In out production cluster based on opendistro [1.13.2] we have multiple Index Management Index Policies with ism_template defined field, where we declare index_patterns. For example

        "ism_template": {
            "index_patterns": [
                "*"
            ],
            "priority": 10,
            "last_updated_time": 1626953706345
        }

this policy must be applied to any newly created index, that doesn’t match index patterns in other index policies, but nothing happens, and to that indexes doens’t have any policy applied

On test stand where configured much less index policies - all working, but in production does not, but in production cluster we have only about 20 policies

How this can be, and how we can solve this problem?

[tantra35]

Little update, we fully recreate all policies in production cluster, but result was discouraging: policies that were previously applied without problems have ceased to be applied, and vice versa. Result is fully random:

For example policy

{
    "policy_id": "ci",
    "description": "Политика для индексов ci-*",
    "last_updated_time": 1627034284633,
    "schema_version": 1,
    "error_notification": null,
    "default_state": "search",
    "states": [
        {
            "name": "search",
            "actions": [],
            "transitions": [
                {
                    "state_name": "delete",
                    "conditions": {
                        "min_index_age": "30d"
                    }
                }
            ]
        },
        {
            "name": "delete",
            "actions": [
                {
                    "delete": {}
                }
            ],
            "transitions": []
        }
    ],
    "ism_template": {
        "index_patterns": [
            "ci-*"
        ],
        "priority": 100,
        "last_updated_time": 1627034284633
    }
}

before recreation works as expected and was applied to indexes like ci-, after recreation all new created indexes doesn't hold any policy, this strange behavior happens in different combinations

[tantra35]

Seems we found the problem:

index management request templates from .opendistro-ism-config index and templates returned by elasticsearch request similar to this:

curl -s --insecure --cert ../usercert-admin.crt --key ../usercert-admin-key.pem 'https://securedelklogsv2-input.service.consul:9200/.opendistro-ism-config/_search?pretty' -H'Content-Type: application/json' -d' {
  "query": {
    "exists": {
      "field": "policy.ism_template"
    }
  }
}'; echo

but this request returns only 10 results by default https://www.elastic.co/guide/en/elasticsearch/reference/7.x/paginate-search-results.html

and in version 1.13.2 opendistro have buggy request in
https://github.com/opendistro-for-elasticsearch/index-management/blob/v1.13.2.0/src/main/kotlin/com/amazon/opendistroforelasticsearch/indexmanagement/indexstatemanagement/ManagedIndexCoordinator.kt#L306

in mainstream of deveplopment of this plugin problem was fixed

[skratyan]

How you fixed this problem?

[tantra35]

we just apply our patch аnd rebuild securityplugin

patch
From 22650245885aa9842e6d2a776420039c4a65af5f Mon Sep 17 00:00:00 2001
From: tantra35 <ruslan.usifov@gmail.com>
Date: Wed, 4 Aug 2021 22:38:39 +0300
Subject: [PATCH] fix wrong ism managment

---
 .../indexstatemanagement/ManagedIndexCoordinator.kt          | 2 +-
 .../action/indexpolicy/TransportIndexPolicyAction.kt         | 5 ++++-
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/src/main/kotlin/com/amazon/opendistroforelasticsearch/indexmanagement/indexstatemanagement/ManagedIndexCoordinator.kt b/src/main/kotlin/com/amazon/opendistroforelasticsearch/indexmanagement/indexstatemanagement/ManagedIndexCoordinator.kt
index a94b82a..64a586d 100644
--- a/src/main/kotlin/com/amazon/opendistroforelasticsearch/indexmanagement/indexstatemanagement/ManagedIndexCoordinator.kt
+++ b/src/main/kotlin/com/amazon/opendistroforelasticsearch/indexmanagement/indexstatemanagement/ManagedIndexCoordinator.kt
@@ -308,7 +308,7 @@ class ManagedIndexCoordinator(
             .source(
                 SearchSourceBuilder().query(
                     QueryBuilders.existsQuery(ISM_TEMPLATE_FIELD)
-                )
+                ).size(MAX_HITS)
             )
             .indices(INDEX_MANAGEMENT_INDEX)
 
diff --git a/src/main/kotlin/com/amazon/opendistroforelasticsearch/indexmanagement/indexstatemanagement/transport/action/indexpolicy/TransportIndexPolicyAction.kt b/src/main/kotlin/com/amazon/opendistroforelasticsearch/indexmanagement/indexstatemanagement/transport/action/indexpolicy/TransportIndexPolicyAction.kt
index 6e48cb6..4bb1280 100644
--- patch
a/src/main/kotlin/com/amazon/opendistroforelasticsearch/indexmanagement/indexstatemanagement/transport/action/indexpolicy/TransportIndexPolicyAction.kt
+++ b/src/main/kotlin/com/amazon/opendistroforelasticsearch/indexmanagement/indexstatemanagement/transport/action/indexpolicy/TransportIndexPolicyAction.kt
@@ -17,6 +17,7 @@ package com.amazon.opendistroforelasticsearch.indexmanagement.indexstatemanageme
 
 import com.amazon.opendistroforelasticsearch.indexmanagement.IndexManagementIndices
 import com.amazon.opendistroforelasticsearch.indexmanagement.IndexManagementPlugin
+import com.amazon.opendistroforelasticsearch.indexmanagement.indexstatemanagement.ManagedIndexCoordinator.Companion.MAX_HITS 
 import com.amazon.opendistroforelasticsearch.indexmanagement.indexstatemanagement.elasticapi.filterNotNullValues
 import com.amazon.opendistroforelasticsearch.indexmanagement.indexstatemanagement.elasticapi.getPolicyToTemplateMap
 import com.amazon.opendistroforelasticsearch.indexmanagement.indexstatemanagement.findConflictingPolicyTemplates
@@ -107,7 +108,9 @@ class TransportIndexPolicyAction @Inject constructor(
             val searchRequest = SearchRequest()
                 .source(
                     SearchSourceBuilder().query(
-                    QueryBuilders.existsQuery(ISM_TEMPLATE_FIELD)))
+                        QueryBuilders.existsQuery(ISM_TEMPLATE_FIELD)
+                    ).size(MAX_HITS)
+                )
                 .indices(IndexManagementPlugin.INDEX_MANAGEMENT_INDEX)
 
             client.search(searchRequest, object : ActionListener<SearchResponse> {
-- 
2.28.0.windows.1
 

but later aws fix this, but not rebuild plugin due they migrate to opensearch