Here, I will be briefing about how you can use “Fluentd Prometheus Plugin” to monitor your services health using docker container or Kubernetes pod logs.
I struggled a lot to find it out while working on a POC project and thought of writing it so that it would be helpful for others .
Installing and starting Prometheus
You first need to install “Prometheus”.
Docker Images (help)
$ docker run --name prometheus -d -p 127.0.0.1:9090:9090 quay.io/prometheus/prometheus
Or can directly download precompiled binary from here.
Select your OS and architecture and download and extract binary.
You will find following files and directories under Prometheus.
vagrant@carrier0-worker:~/monitoring/prometheus-1.6.1.linux-amd64$ ls -l
drwxrwxr-x 2 vagrant vagrant 4096 Apr 19 13:25 console_libraries
drwxrwxr-x 2 vagrant vagrant 4096 Apr 19 13:25 consoles
drwx—— 263 vagrant vagrant 4096 Jul 4 11:42 data
-rw-rw-r– 1 vagrant vagrant 11357 Apr 19 13:25 LICENSE
-rw-rw-r– 1 vagrant vagrant 2793 Apr 19 13:25 NOTICE
-rw-rw-r– 1 vagrant vagrant 1182 May 29 09:25 promemetheus.backup
-rwxr-xr-x 1 vagrant vagrant 62676496 Apr 19 13:27 prometheus
-rw-rw-r– 1 vagrant vagrant 502 May 29 09:39 prometheus.yml
-rwxr-xr-x 1 vagrant vagrant 9996715 Apr 19 13:28 promtool
One which is most useful is “prometheus.yml” where you will be making scrap configuration.
My Scrap Configuration (to be used with fluentd promethues plugin)
Goto promethues directory and run
Installing and starting Fluent Prometheus Plugin
Next you need to get the Fluentd Promethues installable. It’s easy, this is how you can get it running.
Goto https://github.com/kazegusuri/fluent-plugin-prometheus and install it by running following
$ gem install fluent-plugin-prometheus
It will be installed within /var/lib/gems/2.3.0/gems
You can start it by going to prometheus fluentd directory and running following command :-
bundle exec fluentd -c misc/fluentd_sample.conf -v
where, fluentd_sample.conf is sample configuration file for fluentd. Fluent will start listening on port http://localhost:24231
Using fluentd to serve metrics from structured JSON logs
To get a particular metric instrumented:
- Log the metrics value via some structure JSON logger with the following format:
add the matching metric definition under the appropriate filter in the fluentd config for your service (i.e.misc/fluentd_sample.conf):
desc sample metric description
NOTE: Only base metrics are supported (i.e. Counter, Gauge, Histogram, Summary). Vectorized metrics are not supported.
~$ curl http://localhost:24231/metrics | grep sample
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 11268 100 11268 0 0 1661k 0 –:–:– –:–:– –:–:– 1833k
# TYPE sample_error_counter_metric counter
# HELP sample_error_counter_metric sample app error count
# TYPE sample_users_gauge_metric gauge
# HELP sample_users_gauge_metric sample app user count