Store Graph Data in InfluxDB

[optionstechnology]

We have serveral 3rd Party Tools that we are attempting to allow access to the Nagios Graph data - mostly grafana and splunk

I've worked a little with rrdtools but its just a little to clunky for what we are trying to do

From what I can see all three products can use InfluxDB

Ideally I would like to store the data in InfluxDB and access it from Nagios/Grafana/Splunk

I see there is a guide here- https://support.nagios.com/kb/article/n ... u-802.html for getting nagios graph data into grafana

But it does say in that 'break' Nagios XI...

Any thoughts or suggestions on how to do this?

Or even if I can just keep the data in the RRD files for Nagios, but send a copy to InfluxDB? (This might be the best approach)

[ssax]

I'm labbing this up and it's 99% working so far, I should have a response for you tomorrow, we should be able to get it to work with XI with the way I have it setup but one thing to note is that we don't support influxdb/histou/grafana so if you have any issues with those products you will need to reach out to their support for any issues.

If you don't hear from me by the end of day tomorrow, please post on here so that it pops up on our dashboard.

Thank you

[optionstechnology]

Awesome! Thanks for your help

[ssax]

Please send me the output of these commands on your XI server:

Code: Select all

cat /etc/redhat-release
uname -a
df -h
df -i

[optionstechnology]

sent as PM

[ssax]

You forgot this one:

Code: Select all

cat /etc/redhat-release

Additionally, are InfluxDB and Grafana going to be on separate servers?

[optionstechnology]

PM sent

Additionally, are InfluxDB and Grafana going to be on separate servers?

Yeah more than likely

[ssax]

First, if this is a VM, take a VM snapshot AND an XI backup, just in case:

*** DO THIS ON A TEST SYSTEM FIRST SO THAT YOU CAN WORK THROUGH ANY ISSUES BEFORE PUTTING IT IN PRODUCTION ***
- Make sure that you use the same version of the OS and XI so that you can compare apples to apples

https://assets.nagios.com/downloads/nag ... ios-XI.pdf

InfluxDB/Grafana Server:

I spun up a CentOS 7 minimal install server, set the static IP, and put these commands in a file:

Code: Select all

YOURIP=`ip route get 1 | awk '{print $NF;exit}'`
setenforce 0
cat <<EOF >> /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
#     targeted - Targeted processes are protected,
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted
# SETLOCALDEFS= Check local definition changes
SETLOCALDEFS=0
EOF
yum install epel-release -y
cat <<EOF >> /etc/yum.repos.d/influxdb.repo
[influxdb]
name = InfluxDB Repository - RHEL \$releasever
baseurl = https://repos.influxdata.com/rhel/\$releasever/\$basearch/stable
enabled = 1
gpgcheck = 1
gpgkey = https://repos.influxdata.com/influxdb.key
EOF
yum install influxdb -y
systemctl enable influxdb.service
systemctl restart influxdb.service
cat <<EOF >> /etc/yum.repos.d/grafana.repo
[grafana]
name=grafana
baseurl=https://packages.grafana.com/oss/rpm
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packages.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
EOF
yum install grafana -y
systemctl daemon-reload
systemctl enable grafana-server.service
systemctl start grafana-server.service
firewall-cmd --zone=public --add-port=3000/tcp
firewall-cmd --zone=public --add-port=3000/tcp --permanent
firewall-cmd --zone=public --add-port=80/tcp
firewall-cmd --zone=public --add-port=80/tcp --permanent
firewall-cmd --zone=public --add-port=8086/tcp
firewall-cmd --zone=public --add-port=8086/tcp --permanent
yum install httpd php -y
systemctl enable httpd
systemctl restart httpd
cd /tmp
wget -O histou.tar.gz https://github.com/Griesbacher/histou/archive/v0.4.3.tar.gz
mkdir -p /var/www/html/histou
cd /var/www/html/histou
tar xzf /tmp/histou.tar.gz --strip-components 1
cp histou.ini.example histou.ini
cp histou.js /usr/share/grafana/public/dashboards/
sed -i "s/localhost/$YOURIP/g" /usr/share/grafana/public/dashboards/histou.js
curl -k -L -X PUT -H "Content-Type: application/json" -d '{
  "oldPassword": "admin",
  "newPassword": "admin",
  "confirmNew": "admin"
}' http://admin:admin@localhost:3000/api/user/password
cat <<EOF > /etc/grafana/provisioning/datasources/nagflux.yaml
# config file version
apiVersion: 1

# list of datasources to insert/update depending
# on what's available in the datbase
datasources:
   # <string, required> name of the datasource. Required
 - name: nagflux
   # <string, required> datasource type. Required
   type: influxdb
   # <string, required> access mode. direct or proxy. Required
   access: proxy
   # <int> org id. will default to orgId 1 if not specified
   orgId: 1
   # <string> url
   url: http://localhost:8086
   # <string> database name, if used
   database: nagflux
   # <bool> enable/disable basic auth
   basicAuth: false
   isDefault: true
   version: 1
   # <bool> allow users to edit datasources from the UI.
   editable: true
EOF
systemctl restart grafana-server
echo 'Test with these commands:'
echo ''
echo 'curl -k -L -G "http://localhost:8086/query?pretty=true" --data-urlencode "q=show databases"'
echo 'curl -k -L -G "http://localhost:8086/query?db=nagflux&pretty=true" --data-urlencode "q=show series"'
echo 'curl -k -L -G "http://localhost/histou/?host=localhost&service=PING"'
echo ''
echo 'Open a web browser to:'
echo ''
echo "http://$YOURIP:3000/dashboard/script/histou.js?host=localhost&service=PING"

Then run (as root):

Code: Select all

sh yourscriptname

XI Server:

Then on the XI server, put these commands in a file:
- NOTE: CHANGE X.X.X.X in the first line to the IP of your InfluxDB/Grafana server before running

Code: Select all

INFLUXIP='X.X.X.X'
yum install -y golang git
export GOPATH=$HOME/gorepo
mkdir $GOPATH
go get -v -u github.com/griesbacher/nagflux
go build github.com/griesbacher/nagflux 
mkdir -p /opt/nagflux
cp $GOPATH/bin/nagflux /opt/nagflux/
mkdir -p /usr/local/nagios/var/spool/nagfluxperfdata
chown nagios.nagios /usr/local/nagios/var/spool/nagfluxperfdata
cp $GOPATH/src/github.com/griesbacher/nagflux/nagflux.service /usr/lib/systemd/system/
chmod +x /usr/lib/systemd/system/nagflux.service
systemctl daemon-reload
systemctl enable nagflux.service
cat <<EOF >> /opt/nagflux/config.gcfg
[main]
    NagiosSpoolfileFolder = "/usr/local/nagios/var/spool/nagfluxperfdata"
    NagiosSpoolfileWorker = 1
    InfluxWorker = 2
    MaxInfluxWorker = 5
    DumpFile = "nagflux.dump"
    NagfluxSpoolfileFolder = "/usr/local/nagios/var/nagflux"
    FieldSeparator = "&"
    BufferSize = 10000
    FileBufferSize = 65536
    DefaultTarget = "all"

[Log]
    LogFile = ""
    MinSeverity = "INFO"

[InfluxDBGlobal]
    CreateDatabaseIfNotExists = true
    NastyString = ""
    NastyStringToReplace = ""
    HostcheckAlias = "hostcheck"

[InfluxDB "nagflux"]
    Enabled = true
    Version = 1.0
    Address = "http://$INFLUXIP:8086"
    Arguments = "precision=ms&u=root&p=root&db=nagflux"
    StopPullingDataIfDown = true

[InfluxDB "fast"]
    Enabled = false
    Version = 1.0
    Address = "http://$INFLUXIP:8086"
    Arguments = "precision=ms&u=root&p=root&db=fast"
    StopPullingDataIfDown = false
EOF
systemctl restart nagflux.service
cat <<EOF >> /usr/local/nagios/etc/import/commands.cfg
define command {
    command_name    process-host-perfdata-file-bulk
    command_line    /usr/local/nagios/libexec/process-perfdata-file-bulk.sh \$TIMET\$ host
}

define command {
    command_name    process-service-perfdata-file-bulk
    command_line    /usr/local/nagios/libexec/process-perfdata-file-bulk.sh \$TIMET\$ service
}
EOF
cat <<EOF > /usr/local/nagios/libexec/process-perfdata-file-bulk.sh
#!/bin/bash
TIMET="\$1"
TYPE="\$2"

if [[ "\$TYPE" == "host" ]]; then
        VALUE=\$(</usr/local/nagios/var/host-perfdata)
        \$(rm -rf /usr/local/nagios/var/host-perfdata)
        \$(echo "\$VALUE" | tee /usr/local/nagios/var/spool/xidpe/\$TIMET.perfdata.host /usr/local/nagios/var/spool/nagfluxperfdata/\$TIMET.perfdata.host)
        EXITCODE=\$?
else
        VALUE=\$(</usr/local/nagios/var/service-perfdata)
        \$(rm -rf /usr/local/nagios/var/service-perfdata)
        \$(echo "\$VALUE" | tee /usr/local/nagios/var/spool/xidpe/\$TIMET.perfdata.service /usr/local/nagios/var/spool/nagfluxperfdata/\$TIMET.perfdata.service)
        EXITCODE=\$?
fi

exit \$EXITCODE
EOF
chown nagios.nagios /usr/local/nagios/libexec/process-perfdata-file-bulk.sh
chmod +x /usr/local/nagios/libexec/process-perfdata-file-bulk.sh
/usr/local/nagiosxi/scripts/reconfigure_nagios.sh

Then run (as root):

Code: Select all

sh yourscriptname

That should do it, let us know the results.