elasticsearch-datastream总结

一，什么是datastream

datastream是为了更方便的管理时序数据的生命周期而基于ilm扩展的特殊功能。

二，datastream的特点

• 1，文档必须有@timestamp字段
• 2，定义ilm时无需定义iml别名
• 3，可以对datastream进行插入和查询文档，但不能删除和更新文档
• 4，datastream像别名一样管理多个真实索引（隐藏在dataStream背后），这种索引的名称规则：

.ds-datastream名称-yyyy.MM.dd-六位序列号

如下（假设datastream名称是test-data-stream）：

.ds-test-data-stream-2022.05.09-000001

三，使用步骤

1，定义ilm

PUT _ilm/policy/my-lifecycle-policy
{
  "policy": {
    "phases": {
      
      "hot": {
        "min_age": "0ms", 
        "actions": {
          "rollover": {
            "max_primary_shard_size": "50gb",
            "max_docs": 3,
            "max_age": "7d",
            "max_size": "5gb"
          }
        }
      },
      "warm": {
        "min_age": "30s",
        "actions": {
          "shrink": {
            "number_of_shards": 1
          },
          "forcemerge": {
            "max_num_segments": 1
          },
          "allocate": {
            "number_of_replicas": 0
          }
        }
      },
      "cold": {
        "min_age": "60s",
        "actions": {
          "freeze" : { }
        }
      },
      "delete": {
        "min_age": "735d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

2，定义mapping和setting

PUT _component_template/my-mappings
{
  "template": {
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date",
          "format": "date_optional_time||epoch_millis"
        },
        "message": {
          "type": "wildcard"
        }
      }
    }
  },
  "_meta": {
    "description": "Mappings for @timestamp and message fields",
    "my-custom-meta-field": "More arbitrary metadata"
  }
}

# Creates a component template for index settings
PUT _component_template/my-settings
{
  "template": {
    "settings": {
      "index.lifecycle.name": "my-lifecycle-policy"
    }
  },
  "_meta": {
    "description": "Settings for ILM",
    "my-custom-meta-field": "More arbitrary metadata"
  }
}

3，设置刷新时间为1s，方便测试

PUT _cluster/settings
{
  "transient": {
    "indices.lifecycle.poll_interval": "1s"
  }
}

4，定义索引模板

PUT _index_template/my-index-template
{
  "index_patterns": ["test-data-stream*"],
  "data_stream": { },
  "composed_of": [ "my-mappings", "my-settings" ],
  "priority": 500,
  "_meta": {
    "description": "Template for my time series data",
    "my-custom-meta-field": "More arbitrary metadata"
  }
}

5，插入数据，生成dataStream索引（隐式生成，系统会根据模板中的data_stream标志自动删除dataStream）

PUT test-data-stream/_bulk
{ "create":{ } }
{ "@timestamp": "2099-03-08T11:04:05.000Z","message": "test" }
PUT test-data-stream/_bulk
{ "create":{ } }
{ "@timestamp": "2099-03-08T11:04:05.000Z","message": "test" }
PUT test-data-stream/_bulk
{ "create":{ } }
{ "@timestamp": "2099-03-08T11:04:05.000Z","message": "test" }
PUT test-data-stream/_bulk
{ "create":{ } }
{ "@timestamp": "2099-03-08T11:04:05.000Z","message": "test" }

6，验证结果

GET test-data-stream/_ilm/explain

GET test-data-stream/_search
GET .ds-test-data-stream-2022.05.09-000001/_search