How to use ElasticSearch ILM and save money

This in blog I’m gonna explain how to configure the automatic rotation of indexes in ElasticSearch and how to automatically move the indices to instances that don’t need high IO so it’s cheaper. I’m gonna show you how to do that using the ElasticSearch Hot-Warm architecture.

The process works by writing the data in an alias that points to the current index, when the index mets the conditions defined in the ILM policy then a new index is created and the alias is updated to point to the new index, then the previous index is eventually moved to a warm which should be cheaper.

Creating a new policy for index rotation

The next code shows how to create a new ILM policy to rotate indices when they reach 60 days or they have a size of 50 GB.

The policy name in this case is index-rollover

PUT /_ilm/policy/index-rollover
{
    "policy": {
        "phases": {
            "hot": {
                "actions": {
                    "rollover": {
                        "max_age": "60d",
                        "max_size": "50gb"
                    },
                    "set_priority": {
                        "priority": 50
                    }
                }
            },
            "warm": {
                "min_age": "60d",
                "actions": {
                    "forcemerge": {
                        "max_num_segments": 1
                    },
                    "shrink": {
                        "number_of_shards": 1
                    },
                    "allocate": {
                        "require": {
                            "data": "warm"
                        }
                    },
                    "set_priority": {
                        "priority": 25
                    }
                }
            }
        }
    }
}

Create a template for the indices

This template is applied to all indices that start by truora-ilm- and it defines what ILM policy should be applied and what’s the alias.

PUT /_template/truora-ilm-template
{
    "index_patterns": [
        "truora-ilm-*"
    ],
    "settings": {
        "refresh_interval": "30s",
        "number_of_replicas": 1,
        "index.lifecycle.name": "index-rollover",      
    	"index.lifecycle.rollover_alias": "truora-ilm"
    }
}

Create your first index

It’s important that the index have a suffix like -<number>, because the ILM policy is going to increase that number automatically when rotating the index.

Also, that index should match the index defined in the template.

PUT /truora-ilm-000000001
{
  "aliases": {
    "truora-ilm": {
      "is_write_index": true
    }
  }
}

To know that the server is well configured see:

GET /truora-ilm-*/_ilm/explain
{
  "indices": {
    "truora-ilm-000000001": {
      "index": "truora-ilm-000000001",
      "managed": true,
      "policy": "index-rollover",
      "lifecycle_date_millis": 1581439470759,
      "age": "5.54m",
      "phase": "hot",
      "phase_time_millis": 1581439476093,
      "action": "unfollow",
      "action_time_millis": 1581439479632,
      "step": "wait-for-follow-shard-tasks",
      "step_time_millis": 1581439482298,
      "phase_execution": {
        "policy": "index-rollover",
        "phase_definition": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_size": "50gb",
              "max_age": "60d"
            },
            "set_priority": {
              "priority": 50
            }
          }
        },
        "version": 6,
        "modified_date_in_millis": 1577206181905
      }
    }
  }
}

That’s all!, you can now start writing to truora-ilm (or however you’ve called it), once the conditions are met for the policy it’s gonna be rotated automatically.

elasticsearch  ilm 

See also