In this lab we will setup centralized logging.
Follow the official guide: https://portal.influxdata.com/downloads/
Install version 1.8.10. Instructions are hidden behind "Are you interested in InfluxDB 1.x Open Source?"
Find bash script "pinger.sh" in 08-files.
Place this script to /usr/local/bin/pinger.
Create a service "pinger" that runs from user "pinger". Check 08-files for systemd service unit example. Place it into /etc/systemd/system/.
Don't forget to execute on systemd config change:
systemctl daemon-reload
Docs: https://docs.ansible.com/ansible/latest/collections/ansible/builtin/systemd_module.html.
Pinger script requires config file:
/etc/pinger/pinger.conf
Example can be found in 08-files as well.
Add new Grafana datasource: influxdb:latency. It should be provisoned automaticaly via Grafana provisioning.
Use this datasource for new panel in dashboard from previous lab.
No ip addresses allowed.
Install Telegraf on the same VM where InfluxDB is located. Docs: https://portal.influxdata.com/downloads/
Desired telegraf version is 1.24.2.
Installation steps can be done in "influxdb" role.
Configure Telegraph for only syslog input and only influxdb output. Hint:
telegraf config --help
UDP transport is more preferable.
Configure rsyslog on all VMs to send all logs to Telegraf. Docs: https://github.com/influxdata/telegraf/blob/master/plugins/inputs/syslog/README.md
UDP transport is more preferable.
Add one more datasource: influxdb:telegraf
Import Grafana dashboard for Syslog: https://grafana.com/grafana/dashboards/12433
Add new datasource and dashboard to Grafana provisioning.
Install InfluxDB stats exporter: https://github.com/carlpett/influxdb_stats_exporter
Download binary from latest release to /usr/local/bin/.
Create new systemd service. Run it with user prometheus as all other exporter do. Describe the service in /etc/systemd/system/prometheus-influxdb-stats-exporter.service.
Add couple more panels to your Main Grafana dashboard:
- InfluxDB health (influxdb_exporter_stats_query_success)
- InfluxDB write rate (influxdb_write_write_ok)
Don't forget to update json in your Ansible repo!
By default InfluxDB logs every request, which floods the logs.
Add to [http] section of influxdb config:
log-enabled = false
write-tracing = false
Add to [data] section of influxdb config:
query-log-enabled = false
Your repository contains these files and directories:
ansible.cfg
group_vars/all.yaml
hosts
infra.yaml
roles/influxdb/tasks/main.yaml
roles/influxdb_exporter/tasks/main.yaml
roles/pinger/tasks/main.yaml
roles/rsyslog/tasks/main.yaml
Your repository also contains all the required files from the previous labs.
Your repository does not contain Ansible Vault master password.
Everything is installed and configured with this command:
ansible-playbook infra.yaml
Running the same command again does not make any changes to any of the managed hosts.
After playbook execution you should be able to see all logs in one Grafana dashboard.