How to Observe NGINX Controller with Fluentd?

Observability is crucial for ensuring the smooth operation of any application or system. NGINX Controller, a comprehensive application delivery platform, plays a vital role in managing and optimizing NGINX deployments. To enhance observability, integrating Fluentd, a powerful open-source data collector, can provide valuable insights into NGINX Controller's performance and behavior. In this guide, we will walk you through the steps of observing NGINX Controller using Fluentd, empowering you to gain actionable intelligence and improve the overall health of your NGINX infrastructure.

Setting Up Fluentd for NGINX Controller Observability:

Install Fluentd:

The first step is to install Fluentd on the system where NGINX Controller is running. Use the following commands for a typical installation:

sudo apt-get install -y ruby-full build-essential
sudo gem install fluentd

Configure Fluentd:

Once Fluentd is installed, create a configuration file for NGINX Controller observability. Create a new configuration file, for example, nginx_controller.conf, and include the following basic configuration:

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<match nginx_controller.**>
  @type stdout
</match>

This basic configuration sets up Fluentd to listen on port 24224 for incoming logs from NGINX Controller.

Start Fluentd:

Launch Fluentd with the configured file:

fluentd -c /path/to/nginx_controller.conf

Configuring NGINX Controller for Fluentd Integration:

Access NGINX Controller Configuration:

Navigate to the NGINX Controller dashboard and access the system configuration settings.

Update Logging Settings:

In the NGINX Controller configuration, locate the logging settings section. Configure NGINX Controller to forward logs to Fluentd by specifying the Fluentd host and port.

logging:
  fluentd:
    enabled: true
    host: fluentd-host
    port: 24224

Replace fluentd-host with the actual IP or hostname of the system running Fluentd.

Save and Apply Changes:

Save the configuration changes and apply them to ensure NGINX Controller starts sending logs to Fluentd.

Verifying Observability:

Check Fluentd Logs:

Monitor Fluentd logs to ensure it is receiving and processing logs from NGINX Controller.

tail -f /var/log/td-agent/td-agent.log

Explore NGINX Controller Metrics:

Visit the Fluentd dashboard or use other visualization tools to explore NGINX Controller metrics and logs ingested by Fluentd.

More Examples and Advanced Observability:

Adding Custom Filters:

Enhance observability by adding custom filters in the Fluentd configuration. For example, filter logs based on severity levels or specific NGINX Controller components.

Integrating with Elasticsearch:

Extend observability by integrating Fluentd with Elasticsearch for long-term storage and advanced querying capabilities.

<match nginx_controller.**>
  @type elasticsearch
  host elasticsearch-host
  port 9200
  index_name fluentd-nginx-controller
</match>

Replace elasticsearch-host with the actual IP or hostname of your Elasticsearch instance.