Configuring Fluent Bit for Elasticsearch ILM Rollover

This guide delves into the benefits of adopting the EFK stack over the conventional ELK stack, highlighting the significant enhancements it brings to KubeRocketCI. It provides a comprehensive walkthrough on configuring the EFK stack, enabling the integration of a sophisticated logging system with your applications.

Overview of the ELK Stack

The ELK (Elasticsearch, Logstash, and Kibana) stack is renowned for its capability to consolidate logs from various systems and applications. It facilitates the analysis of these logs and the creation of visualizations for monitoring applications and infrastructure, expediting troubleshooting, enhancing security analytics, and more.

Below is an overview of the default components within the ELK stack:

• Beats family - A suite of log shippers that transport logs from their sources to the stack, including Filebeat, Metricbeat, Packetbeat, among others. Beats can either replace Logstash or complement it.
• Logstash - A robust log processing framework that collects, processes, stores, and searches logs.
• Elasticsearch - A highly scalable search and analytics engine built on the Lucene Java library.
• Kibana - A powerful visualization tool that queries data from Elasticsearch, enabling insightful data analysis.

EFK Stack Overview

We use FEK (also called EFK) (Fluent Bit, Elasticsearch, Kibana) stack in Kubernetes instead of ELK because this stack provides us with the support for Logsight for Stage Verification and Incident Detection. In addition to it, Fluent Bit has a smaller memory fingerprint than Logstash. Fluent Bit has the Inputs, Parsers, Filters and Outputs plugins similarly to Logstash.

Automate Elasticsearch Index Rollover With ILM

In this guide, index rollover with the Index Lifecycle Management ILM is automated in the FEK stack.

The resources can be created via API using curl, Postman, Kibana Dev Tools console or via GUI. They are going to be created them using Kibana Dev Tools.

1. Go to Management -> Dev Tools in the Kibana dashboard:

   

2. Create index lifecycle policy with the index rollover:

   note

   This policy can also be created in GUI in Management -> Stack Management -> Index Lifecycle Policies

   Index Lifecycle has several phases: Hot, Warm, Cold, Frozen, Delete. Indices also have different priorities in each phase. The warmer the phase, the higher the priority is supposed to be, e.g., 100 for the hot phase, 50 for the warm phase, and 0 for the cold phase.

   In this Use Case, only the Hot and Delete phases are configured. So an index will be created, rolled over to a new index when 1gb in size or 1day in time and deleted in 7 days. The rollover may not happen exactly at 1GB because it depends on how often Kibana checks the index size. Kibana usually checks the index size every 10 minutes but this can be changed by setting the indices.lifecycle.poll_interval monitoring timer.

   The index lifecycle policy example:

   Index Lifecycle Policy

   PUT _ilm/policy/fluent-bit-policy
   {
     "policy": {
       "phases": {
         "hot": {
           "min_age": "0ms",
           "actions": {
             "set_priority": {
               "priority": 100
             },
             "rollover": {
               "max_size": "1gb",
               "max_primary_shard_size": "1gb",
               "max_age": "1d"
             }
           }
         },
         "delete": {
           "min_age": "7d",
           "actions": {
             "delete": {
               "delete_searchable_snapshot": true
             }
           }
         }
       }
     }
   }

   Insert the code above into the Dev Tools and click the arrow to send the PUT request.

3. Create an index template so that a new index is created according to this template after the rollover:

   note

   This policy can also be created in GUI in Management -> Stack Management -> Index Management -> Index Templates.

   Expand the menu below to see the index template example:

   Index Template

   PUT /_index_template/fluent-bit
   {
     "index_patterns": ["fluent-bit-kube-*"],
     "template": {
       "settings": {
         "index": {
           "lifecycle": {
             "name": "fluent-bit-policy",
             "rollover_alias": "fluent-bit-kube"
           },
           "number_of_shards": "1",
           "number_of_replicas": "0"
         }
       }
     }
   }
   note

   • index.lifecycle.rollover_alias is required when using a policy containing the rollover action and specifies which alias to rollover on behalf of this index. The intention here is that the rollover alias is also defined on the index.

   • number_of_shards is the quantity of the primary shards. Elasticsearch index is really just a logical grouping of one or more physical shards, where each shard is actually a self-contained index. By distributing the documents in an index across multiple shards and distributing those shards across multiple nodes, Elasticsearch can ensure redundancy, which both protects against hardware failures and increases query capacity as nodes are added to a cluster. As the cluster grows (or shrinks), Elasticsearch automatically migrates shards to re-balance the cluster. Please refer to the official documentation here.

   • number_of_replicas is the number of replica shards. A replica shard is a copy of a primary shard. Elasticsearch will never assign a replica to the same node as the primary shard, so make sure you have more than one node in your Elasticsearch cluster if you need to use replica shards. The Elasticsearch cluster details and the quantity of nodes can be checked with:

     GET _cluster/health

   Since we use one node, the number_of_shards is 1 and number_of_replicas is 0. If you put more replicas within one node, your index will get yellow status in Kibana, yet still be working.

4. Create an empty index with write permissions:

   note

   This index can also be created in GUI in Management -> Stack Management -> Index Management -> Indices.

   Index example with the date math format:

   Index

   # URI encoded /<fluent-bit-kube-{now/d}-000001>
   PUT /%3Cfluent-bit-kube-%7Bnow%2Fd%7D-000001%3E
   {
     "aliases": {
       "fluent-bit-kube": {
         "is_write_index": true
       }
     }
   }

   The code above will create an index in the{index_name}-{current_date}-{rollover_index_increment} format. For example: fluent-bit-kube-2023.03.17-000001.

   Please refer to the official documentation on the index rollover with Date Math here.

   note

   It is also possible to use index pattern below if the date math format does not seem applicable:

   Index

   PUT fluent-bit-kube-000001
   {
     "aliases": {
       "fluent-bit-kube": {
         "is_write_index": true
       }
     }
   }

   Check the status of the created index:

   GET fluent-bit-kube*-000001/_ilm/explain

5. Configure Fluent Bit. Play attention to the Elasticsearch Output plugin configuration.

   The important fields in the [OUTPUT] section are Index fluent-bit-kube since we should use the index with the same name as Rollover Alias in Kibana and Logstash_Format Off as we use the Rollover index pattern in Kibana that increments by 1.

   ConfigMap example with Configuration Variables for HTTP_User and HTTP_Passwd:

   ConfigMap fluent-bit

   data:
     fluent-bit.conf: |
       [SERVICE]
           Daemon Off
           Flush 10
           Log_Level info
           Parsers_File parsers.conf
           Parsers_File custom_parsers.conf
           HTTP_Server On
           HTTP_Listen 0.0.0.0
           HTTP_Port 2020
           Health_Check On
   
       [INPUT]
           Name tail
           Tag kube.*
           Path /var/log/containers/*.log
           Parser docker
           Mem_Buf_Limit 5MB
           Skip_Long_Lines Off
           Refresh_Interval 10
       [INPUT]
           Name systemd
           Tag host.*
           Systemd_Filter _SYSTEMD_UNIT=kubelet.service
           Read_From_Tail On
           Strip_Underscores On
   
       [FILTER]
           Name                kubernetes
           Match               kube.*
           Kube_Tag_Prefix     kube.var.log.containers.
           Kube_URL            https://kubernetes.default.svc:443
           Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
           Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
           Merge_Log           Off
           Merge_Log_Key       log_processed
           K8S-Logging.Parser  On
           K8S-Logging.Exclude On
       [FILTER]
           Name nest
           Match kube.*
           Operation lift
           Nested_under kubernetes
           Add_prefix kubernetes.
       [FILTER]
           Name modify
           Match kube.*
           Copy kubernetes.container_name tags.container
           Copy log message
           Copy kubernetes.container_image tags.image
           Copy kubernetes.namespace_name tags.namespace
       [FILTER]
           Name nest
           Match kube.*
           Operation nest
           Wildcard tags.*
           Nested_under tags
           Remove_prefix tags.
   
       [OUTPUT]
           Name            es
           Match           kube.*
           Index           fluent-bit-kube
           Host            elasticsearch-master
           Port            9200
           HTTP_User       ${ES_USER}
           HTTP_Passwd     ${ES_PASSWORD}
           Logstash_Format Off
           Time_Key       @timestamp
           Type            flb_type
           Replace_Dots    On
           Retry_Limit     False
           Trace_Error     Off

6. Create index pattern (Data View starting from Kibana v8.0):

   Go to Management -> Stack Management -> Kibana -> Index patterns and create an index with the fluent-bit-kube-* pattern:

   

7. Check logs in Kibana. Navigate to Analytics -> Discover:

   

   note

   In addition, in the top-right corner of the Discover window, there is a button called Inspect. Clicking on it will reveal the query that Kibana is sending to Elasticsearch. These queries can be used in Dev Tools.

8. Monitor the created indices:

   GET _cat/indices/fluent-bit-kube-*
   note

   Physically, the indices are located on the elasticsearch Kubernetes pod in /usr/share/elasticsearch/data/nodes/0/indices. It is recommended to backup indices only via Snapshots.

   We've configured the index rollover process. Now the index will be rolled over to a new one once it reaches the indicated size or time in the policy, and old indices will be removed according to the policy as well.

   When you create an empty index that corresponds to the pattern indicated in the index template, the index template attaches rollover_alias with the fluent-bit-kube name, policy and other configured data. Then the Fluent Bit Elasticsearch output plugin sends logs to the Index fluent-bit-kube rollover alias. The index rollover process is managed by ILM that increments our indices united by the rollover_alias and distributes the log data to the latest index.

ILM Without Rollover Policy

It is also possible to manage index lifecycle without rollover indicated in the policy. If this is the case, this section will explain how to refactor the index to make it look that way: fluent-bit-kube-2023.03.18.

note

The main drawback of this method is that the indices can be managed only by their creation date.

To manage index lifecycle without rollover policy, follow the steps below:

1. Create a Policy without rollover but with indices deletion:

   Index Lifecycle Policy

   PUT _ilm/policy/fluent-bit-policy
   {
     "policy": {
       "phases": {
         "hot": {
           "min_age": "0ms",
           "actions": {
             "set_priority": {
               "priority": 100
             }
           }
         },
         "delete": {
           "min_age": "7d",
           "actions": {
             "delete": {
               "delete_searchable_snapshot": true
             }
           }
         }
       }
     }
   }

2. Create an index template with the rollover_alias parameter:

   Index Template

   PUT /_index_template/fluent-bit
   {
     "index_patterns": ["fluent-bit-kube-*"],
     "template": {
       "settings": {
         "index": {
           "lifecycle": {
             "name": "fluent-bit-policy",
             "rollover_alias": "fluent-bit-kube"
           },
           "number_of_shards": "1",
           "number_of_replicas": "0"
         }
       }
     }
   }

3. Change the Fluent Bit [OUTPUT] config to this one:

   ConfigMap fluent-bit

   [OUTPUT]
       Name            es
       Match           kube.*
       Host            elasticsearch-master
       Port            9200
       HTTP_User       ${ES_USER}
       HTTP_Passwd     ${ES_PASSWORD}
       Logstash_Format On
       Logstash_Prefix fluent-bit-kube
       Logstash_DateFormat %Y.%m.%d
       Time_Key        @timestamp
       Type            flb_type
       Replace_Dots    On
       Retry_Limit     False
       Trace_Error     On

4. Restart Fluent Bit pods.

Fluent Bit will be producing a new index every day with the new date in its name like in the fluent-bit-kube-2023.03.18 name. Index deleting will be performed according to the policy.

Tips on Fluent Bit Debugging

If you experience a lot of difficulties when dealing with Fluent Bit, this section may help you.

Fluent Bit has docker images labelled -debug, e.g., cr.fluentbit.io/fluent/fluent-bit:2.0.9-debug.

Change that image in the Kubernetes Fluent Bit DaemonSet and add the Trace_Error On parameter to the [OUTPUT] section in the Fluent Bit configmap:

[OUTPUT]
  Trace_Error On

After adding the parameter above, you will start seeing more informative logs that will probably help you find out the reason of the problem.

Related Articles

• Index Lifecycle Management
• Elasticsearch Output