Docker Container Health Monitoring Using Fluentd Prometheus Plugin

Here, I will be briefing about how you can use “Fluentd Prometheus Plugin” to monitor your services health using  docker container or Kubernetes pod logs.

I struggled a lot to find it out while working on a POC project and thought of writing it so that it would be helpful for others .

Installing and starting Prometheus

You first need to install “Prometheus”.

Docker Images (help)

$ docker run --name prometheus -d -p 127.0.0.1:9090:9090 quay.io/prometheus/prometheus

Or can directly download precompiled binary from here.

Select your OS and architecture and download and extract binary.

You will find following files and directories under Prometheus.

vagrant@carrier0-worker:~/monitoring/prometheus-1.6.1.linux-amd64$ ls -l
total 71008
drwxrwxr-x 2 vagrant vagrant 4096 Apr 19 13:25 console_libraries
drwxrwxr-x 2 vagrant vagrant 4096 Apr 19 13:25 consoles
drwx—— 263 vagrant vagrant 4096 Jul 4 11:42 data
-rw-rw-r– 1 vagrant vagrant 11357 Apr 19 13:25 LICENSE
-rw-rw-r– 1 vagrant vagrant 2793 Apr 19 13:25 NOTICE
-rw-rw-r– 1 vagrant vagrant 1182 May 29 09:25 promemetheus.backup
-rwxr-xr-x 1 vagrant vagrant 62676496 Apr 19 13:27 prometheus
-rw-rw-r– 1 vagrant vagrant 502 May 29 09:39 prometheus.yml
-rwxr-xr-x 1 vagrant vagrant 9996715 Apr 19 13:28 promtool

One which is most useful is “prometheus.yml”  where you will be making scrap configuration. 

My Scrap Configuration (to be used with fluentd promethues plugin)

Prometheus.Yml

Goto promethues directory and run

./prometheus -config.file=prometheus.yml

 

Installing and starting Fluent Prometheus Plugin

Next you need to get the Fluentd Promethues installable. It’s easy, this is how you can get it running.

Goto https://github.com/kazegusuri/fluent-plugin-prometheus  and install it by running following

command.

$ gem install fluent-plugin-prometheus

It will be installed within /var/lib/gems/2.3.0/gems  

You can start it by going to prometheus fluentd directory and running following command :-

bundle exec fluentd -c misc/fluentd_sample.conf -v

where, fluentd_sample.conf  is sample configuration file for fluentd. Fluent will start listening on port http://localhost:24231

Using fluentd to serve metrics from structured JSON logs

To get a particular metric instrumented:

  1. Log the metrics value via some structure JSON logger with the following format:

{

 

  “<metric_name>”:”<metric_value>”,

  “label_1″:”value_1”,

  “label_2″:”value_2”

 

}

add the matching metric definition under the appropriate filter in the fluentd config for your service (i.e.misc/fluentd_sample.conf):

<metric>

    name sample_metric

    type <metric_type>

    desc sample metric description

    key <metric_name>

    <labels>

       label_1 ${label_1}

       label_2 ${label_2}

    </labels>

  </metric>

NOTE: Only base metrics are supported (i.e. Counter, Gauge, Histogram, Summary). Vectorized metrics are not supported.

~$ curl http://localhost:24231/metrics | grep sample

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

100 11268  100 11268    0     0  1661k      0 –:–:– –:–:– –:–:– 1833k

# TYPE sample_error_counter_metric counter

# HELP sample_error_counter_metric sample app error count

sample_error_counter_metric{counter_label_1=”counter_value_1″,counter_label_2=”counter_value_2″} 15072.0

# TYPE sample_users_gauge_metric gauge

# HELP sample_users_gauge_metric sample app user count

sample_users_gauge_metric{gauge_label_1=”gauge_value_1″,gauge_label_2=”gauge_value_2″} 12.0

 

Leave a Reply

Your email address will not be published. Required fields are marked *