How to Configure Fluent Bit to Collect Logs for Your K8s Cluster?

In the dynamic world of Kubernetes (K8s), efficient log collection is crucial for monitoring and troubleshooting. One powerful tool that simplifies this process is Fluent Bit, a lightweight and flexible log processor and forwarder. In this guide, we will walk through the step-by-step process of configuring Fluent Bit to seamlessly collect logs from your Kubernetes cluster.

Prerequisites:

Before we dive into the configuration process, ensure you have the following prerequisites in place:

1. A running Kubernetes cluster
2. kubectl command-line tool installed
3. Fluent Bit installed on your Kubernetes nodes

Step 1: Install Fluent Bit on Kubernetes:

To get started, install Fluent Bit on your Kubernetes cluster. You can use the following command:

kubectl apply -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-service-account.yaml
kubectl apply -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role.yaml
kubectl apply -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role-binding.yaml
kubectl apply -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/elasticsearch/fluent-bit-configmap.yaml
kubectl apply -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/elasticsearch/fluent-bit-ds.yaml

Step 2: Configure Fluent Bit for Log Collection:

Once Fluent Bit is installed, you need to configure it to collect logs from your Kubernetes pods. Create a Fluent Bit ConfigMap using the following example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: kube-system
data:
  fluent-bit.conf: |
    [INPUT]
        Name              tail
        Path              /var/log/containers/*.log
        Parser            docker
        Tag               kube.*

    [OUTPUT]
        Name              es
        Match             kube.*
        Host              <YOUR_ELASTICSEARCH_HOST>
        Port              <YOUR_ELASTICSEARCH_PORT>
        Index             kubernetes

Replace <YOUR_ELASTICSEARCH_HOST> and <YOUR_ELASTICSEARCH_PORT> with your Elasticsearch details.

Step 3: Apply the ConfigMap:

Apply the Fluent Bit ConfigMap to your Kubernetes cluster:

kubectl apply -f fluent-bit-configmap.yaml

Step 4: Restart Fluent Bit:

Restart Fluent Bit to apply the new configuration:

kubectl delete pod -n kube-system -l k8s-app=fluent-bit

Step 5: Verify Logs in Elasticsearch:

Check your Elasticsearch instance for the collected logs under the specified index. You can use tools like Kibana to visualize and analyze the logs.

Congratulations! You have successfully configured Fluent Bit to collect logs from your Kubernetes cluster.

More Examples and Customizations:

Explore Fluent Bit's documentation for additional input and output plugins. You can customize the configuration based on your specific log collection needs.

For example, to collect logs from application-specific directories, modify the 'Path' parameter in the Fluent Bit ConfigMap accordingly.

[INPUT]
    Name              tail
    Path              /var/log/apps/*.log
    Tag               app.*

Feel free to experiment with different configurations to tailor Fluent Bit to your requirements.