7,540
edits
Changes
→Collecting metrics from logs
<br>
=Collecting metrics from the access logs=
==Overview==
The task is to process the access log of HAproxy with a log interpreter parser and generate Prometheus metrics that are available for Prometheus through an HTTP endpontendpoint. We will use the grok-exporter tool, which can do both. It can read logs from a file or stdin and generate metrics based on the logs. The grok-exporter will receive the logs from HAproxy via an rsyslog server. Rsyslog puts will put logs into a file from which grok-exporter will be able to read them. Grok-exporter converts logs into promethues metrics.
Necessary steps:
* You need We have to create a docker image from grok-exporter that has rsyslog in the image. (The container must be able to run the rsyslog server as root, which requires extra openShfit configuration)* The grok-exporter configuration will be placed in into a OpenShfit ConfigMap and the rsyslog workspace must be an OpenShift volume. (writing into the containers file system runtime is really inefficient)
* We have to create a ClasterIP-type service that can perform load-balancing between grok-exporter pods.
* Routers (The HAproxy) routers should be configured to write access logs in debug mode and send them to the remote rsyslog server running next to the grok-exporter. * The rsyslog server running in the grok-exporter pod will both write the received HAproxy access logs into the file '''/var/log/messages''' (emptyDir type volume) and sends it them to '''stdout '''as well.* Logs written to stdout will be picked up by the docker-log-driver as well and forwarded to the centralized log architecture (log retention) * The grok-exporter program reads '''/var/log/messages''', generates prometheus metrics from its the HAproxy access-logs.* The Promethues have to be configured to use '''kubernetes_sd_configs''' to directly collect metric from the grok-exporter pods, not through the Kubernetes service to bypass load-balancing
<br>
==grok-exporter introduction==
Grok-exporter is a tool that can process logs based on regular expressions to produce and covert them into the 4 basic types of prometheus metrics:
* gauge
* counter
* '''file''': we will stick to this
* '''webhook''': This solution could also be used with logstash used as rsyslog server. Logstash can send the logs to the grok-exporter webhook with the logstash plugin "http-output"
* '''stdin''': With rsyslog, stdin can also be used. This requires the use of the '''omprog''' program, that reads can read from a stocket stockets and passes it on pas the content read through stdin: https://www.rsyslog.com/doc/v8-stable/configuration/modules/omprog.html
'''mtail''':<br>
The other alternative solution would be google's '''mtail''', which is smore more efficient in processing logs than the grok engine.<br>
https://github.com/google/mtail
* grok: Location of the grok patterns. Pattern definition will be stored in /grok/patterns folder.
* metrics: This is the most important part. Here you need to define the metrics and the associated regular expression
* server: What Contains the port of the http metrics server should listen to.
<br>
====Metrics====
Metrics must be defined by metric types. The four basic types of prometheus Prometheus metrics are supported: '''Gauge, Counter, Histogram, Summary''' (quantile)
<br>
Each definition contains 4 parts:
* name: This will be the name of the metric
* help: This will be the help text for the metric.
* match: Describe Describes the structure of the log string in a regular expression style. Here you can use pre-defined grok patterns:
** '''BASIC grok patterns''': https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns
** '''HAROXY patterns''': https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/haproxy
* label: You Here we can name add Prometheus labels to the result groups. The name can be referenced in the label section, which will create a label whose value will be the parsed datametrics.
<br>
==== match definition====Grok assumes that each element is separated by a single space in the source log files. In the match section, you have to write a regular expression using grok building blocks. Each building block has the format: '''% {PATTERN NAMEPATTERN_NAME}''' where PATTERN NAME PATTERN_NAME must be an existing pattern predefined in grokpattern. The most common type is '''% {DATA}''', which refers to an arbitrary data structure that contain contains no withe-space. There are several compound patterns that are build up from other basic grok patterns. If you want We can assign the regular expression described by result groups to named variables that can be used as the value of the Prometheus metric or as label values. The variable name must be placed inside the curly bracket of the pattern to be separated by a result group, you must semicolon from the patter name the patterns, for example:.
<pre>
% {DATA:this_is_the_name} this_is_the_name
</Pre>
The result of the regular expression will be assigned to the variable '''this_is_the_name''', which can be referenced when defining the value of the Prometheus metric or the metrics label.
<br>
==== labels definition ====You In the label section we can refer to patterns named in define labels for the generated Prometheus metric. The labels section. This will give are defined with a name:value list, where the value of the field parsed from the given log can be a string to constant or a variable defined for a pattern in the defined labelmatch section. The variable must be referenced in go-template style between double curly brackets starting with a dot. For example, using if we used the '''% {DATA: this_is_the_name}''' patternin the match section, you could write we can define the 'mylabel' Prometheus label with the value of the 'this_is_the_name' variable in the following tagway: <br>
<pre>
mylabel: '{{.this_is_the_name}}'
</Pre>
The following log line is given:
<pre>
7/30/2016 2:37:03 PM adam 1.5
</Pre>
And there is given the following metric rule in the grok config:
<source lang="C++">
metrics:
user: '{{.user}}'
</source>
<pre>
# HELP Example counter metric with labels.
<br>
==== Determine Value of the value of a metric ====For a counter-type metric, you do not we don't need to determine the value of the metric, because it will just simply count the number of matching logs foundmatches of the regular expression. In contrast, for all other types, you we have to specify what is considered a the value. This should be specified defined in the '''value''' section, where of the metric definition. We also have to reference here a named grok pattern from variable defined in the match section like we did it in the label definition. The variable must be referenced in the same way as Go templates as defined in the tagsgo-template style. Here is an example. Eg the The following two log lines are given:
<pre>
7/30/2016 2:37:03 PM adam 1
7/30/2016 2:37:03 PM Adam 5
</Pre>
And for this we define the following histogram, which consists of two buckets, buckets bucket 1 and 2:
<source lang="C++">
metrics:
<br>
==== Functions ====
* add
* subtract
* multiply
* divide
<source lang = "C ++">
value: "{{multiply .val 1000}}"
</source>
<pre>
# HELP Example counter metric with labels.
...
</Pre>
Since the two values will would change to 1000 and 5000 respectively, both will fall into the infinite categorybucket.
<br>
<br>
== Creating a the grok config file ==You need We have to compile a grok pattern that fits in the HAproxy access-log lines and can extract all the attributes that are important to usrequired for creating the response-latency-histogram based on the HAproxy access-logs. The required attributes are the following:
* total response time to respond
* haproxy instance id
* openshfit service namespace
</Pre>
In the config.yml file, we will define a histogram that contains the classic response -time for full requests. This is a classic -latency histogram, that usually containing contains the following buckets (in seconds):
<pre>
[0.1, 0.2, 0.4, 1, 3, 8, 20, 60, 120]
</Pre>
Response time histogram metrics by convention are called : '''<name prefix> _http_request_duration_seconds'''
'''config.yml'''
* '''type:file''' -> read logs from file
* '''path: /var/log/messages''' -> The rsyslog server writes logs to /var/log/messages by default
* '''readall: true''' -> always reads the entire log file. This should only be used for testing, in a live environment, and should this always has to be set to false.* '''patterns_dir: ./patterns''' -> Base directory of the Pattern definitions can be found in the docker image* <pre> value: "{{divide .Tt 1000}}" </pre> The serving response time in the HAproxy log is in milliseconds and must be converted so we convert it to seconds.* '''port: 9144''' -> This The http port will provide of the /metrics endpoint.
<br>
{{warning | do not forget to set the value of '''readall' 'to''' false '' in a live environment as this will greatly it can reduce efficiency}}
<br>
<br>
=== Online grok tester ===
There are several online grok testing tools. These can be used help to compile the required grok pattern expression very effectively: https://grokdebug.herokuapp.com/
:[[File:ClipCapIt-190808-170333.PNG]]
<br>
== making the docker image ==The grok-exporter docker image is available on the docker hub in several versionsvariants. The only problem with them is that they do not include the rsyslog server, what we need is for the HAproxy to send logs directly to the grok-exporter podokankpod. <br>
docker-hub link: https://hub.docker.com/r/palobo/grok_exporter <br>
<br>
The second problem is that they are all based on an ubuntu base image, where that makes it is very difficult to get rsyslog to log on to stdout, which requires required by the Kubernetets centralized log collector to receive for retaining HAproxy logs, so both monitoring and centralized logging can be served. Thousands of We are going to port the original grok Dockerfile will be ported to '''centos 7''' and will be supplemented with the add rsyslog installation of to the rsyslog servernew image.
<br>
All necessary files are available on git-hub: https://github.com/berkiadam/haproxy-metrics/tree/master/grok-exporter-centos <br>
I also created an ubuntu based solution, which is an extension of the original docker-hub solution, which can also be found on git-hub in the '''grok-exporter-ubuntu folder'''. For In the rest of the howotthis chapter, we will always are going to use the cent centOS version.
<br>
<br>
=== Dockerfile ===
We will start with modify the '''palobo / grok_exporter''' Dockerfile, but we will complement extend it with the rsyslog installation and port it to centos: https://github.com/berkiadam/haproxy-metrics/tree/master/grok- CentOS-exporter
<br>
➲[[File:Grok-exporter-docker-build.zip|Download all files required for the build of the Docker image build]]
<br>
CMD sh -c "nohup /usr/sbin/rsyslogd -i ${PID_DIR}/pid -n &" && ./grok_exporter -config /grok/config.yml
</source>
{{note | It is important that we to use at least version 0.2.7 or higher of the grok-exporter, as we use grok-functions in the function handling first appearedmetrics definitions}}
<br>