Issue with Nagios Fusion server

[cdienger]

Let's start with /var/log/mysqld.log, /var/log/httpd/access_log, and /var/log/httpd/error_log.

[anjan]

I am attaching these three files . two things i am highlighting here we are using pgsql and we have installed nagios on core server in /opt/nagios/ path.

Thanks

Anjan

[anjan]

Access Log output

10.15.174.225 - - [24/Aug/2017:18:15:44 -0700] "POST /cgi-bin/cmd.cgi HTTP/1.1" 301 314 "-" "libwww-perl/6.15"
10.15.174.225 - - [24/Aug/2017:19:15:20 -0700] "POST /cgi-bin/cmd.cgi HTTP/1.1" 301 314 "-" "libwww-perl/6.15"
10.15.174.225 - - [24/Aug/2017:19:15:26 -0700] "POST /cgi-bin/cmd.cgi HTTP/1.1" 301 314 "-" "libwww-perl/6.15"
10.15.174.225 - - [24/Aug/2017:19:15:30 -0700] "POST /cgi-bin/cmd.cgi HTTP/1.1" 301 314 "-" "libwww-perl/6.15"
10.15.174.225 - - [24/Aug/2017:19:15:34 -0700] "POST /cgi-bin/cmd.cgi HTTP/1.1" 301 314 "-" "libwww-perl/6.15"
10.15.174.225 - - [24/Aug/2017:19:15:38 -0700] "POST /cgi-bin/cmd.cgi HTTP/1.1" 301 314 "-" "libwww-perl/6.15"
10.15.174.225 - - [24/Aug/2017:19:15:41 -0700] "POST /cgi-bin/cmd.cgi HTTP/1.1" 301 314 "-" "libwww-perl/6.15"
10.15.174.225 - - [24/Aug/2017:19:15:47 -0700] "POST /cgi-bin/cmd.cgi HTTP/1.1" 301 314 "-" "libwww-perl/6.15"
10.15.174.225 - - [24/Aug/2017:20:15:15 -0700] "POST /cgi-bin/cmd.cgi HTTP/1.1" 301 314 "-" "libwww-perl/6.15"
10.15.174.225 - - [24/Aug/2017:20:15:21 -0700] "POST /cgi-bin/cmd.cgi HTTP/1.1" 301 314 "-" "libwww-perl/6.15"
10.15.174.225 - - [24/Aug/2017:20:15:25 -0700] "POST /cgi-bin/cmd.cgi HTTP/1.1" 301 314 "-" "libwww-perl/6.15"
10.15.174.225 - - [24/Aug/2017:20:15:29 -0700] "POST /cgi-bin/cmd.cgi HTTP/1.1" 301 314 "-" "libwww-perl/6.15"
10.15.174.225 - - [24/Aug/2017:20:15:33 -0700] "POST /cgi-bin/cmd.cgi HTTP/1.1" 301 314 "-" "libwww-perl/6.15"
10.15.174.225 - - [24/Aug/2017:20:15:37 -0700] "POST /cgi-bin/cmd.cgi HTTP/1.1" 301 314 "-" "libwww-perl/6.15"
10.15.174.225 - - [24/Aug/2017:20:15:42 -0700] "POST /cgi-bin/cmd.cgi HTTP/1.1" 301 314 "-" "libwww-perl/6.15"
10.15.174.225 - - [24/Aug/2017:21:15:22 -0700] "POST /cgi-bin/cmd.cgi HTTP/1.1" 301 314 "-" "libwww-perl/6.15"
10.15.174.225 - - [24/Aug/2017:21:15:28 -0700] "POST /cgi-bin/cmd.cgi HTTP/1.1" 301 314 "-" "libwww-perl/6.15"
10.15.174.225 - - [24/Aug/2017:21:15:32 -0700] "POST /cgi-bin/cmd.cgi HTTP/1.1" 301 314 "-" "libwww-perl/6.15"
10.15.174.225 - - [24/Aug/2017:21:15:36 -0700] "POST /cgi-bin/cmd.cgi HTTP/1.1" 301 314 "-" "libwww-perl/6.15"
10.15.174.225 - - [24/Aug/2017:21:15:40 -0700] "POST /cgi-bin/cmd.cgi HTTP/1.1" 301 314 "-" "libwww-perl/6.15"
10.15.174.225 - - [24/Aug/2017:21:15:43 -0700] "POST /cgi-bin/cmd.cgi HTTP/1.1" 301 314 "-" "libwww-perl/6.15"
10.15.174.225 - - [24/Aug/2017:21:15:49 -0700] "POST /cgi-bin/cmd.cgi HTTP/1.1" 301 314 "-" "libwww-perl/6.15"

Error Log Output :

[Sun Aug 20 04:02:01 2017] [notice] Digest: generating secret for digest authentication ...
[Sun Aug 20 04:02:01 2017] [notice] Digest: done
[Sun Aug 20 04:02:01 2017] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads.
[Sun Aug 20 04:02:01 2017] [notice] Apache/2.2.3 (Red Hat) configured -- resuming normal operations

[cdienger]

We should see requests from the fusion server when it tries to pull information. Can you PM the full logs to me? Or run:

Code: Select all

tail -f /var/lib/pgsql/pgstartup.log
tail -f /var/log/httpd/access_log
tail -f /var/log/httpd/error_log

Let's get a copy of /var/lib/pgsql/pgstartup.log and /usr/local/nagiosfusion/var/log/* as well.

[anjan]

Followings are the requested log files output. JFYI : These core servers are old versions servers Nagios® Core™ 4.0.2 is there any restriction with nagios core version ?

[root@gms-us01-vm3 libexec]# tail -f /var/lib/pgsql/pgstartup.log
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No space left on device
could not write to log file: No spLOG: logger shutting down

[root@gms-us01-vm3 libexec]# tail -f /var/log/httpd/access_log
10.15.174.225 - - [27/Aug/2017:21:15:01 -0700] "POST /cgi-bin/cmd.cgi HTTP/1.1" 301 314 "-" "libwww-perl/6.15"
10.15.174.225 - - [27/Aug/2017:21:15:06 -0700] "POST /cgi-bin/cmd.cgi HTTP/1.1" 301 314 "-" "libwww-perl/6.15"
10.15.174.225 - - [27/Aug/2017:21:15:11 -0700] "POST /cgi-bin/cmd.cgi HTTP/1.1" 301 314 "-" "libwww-perl/6.15"
10.15.174.225 - - [27/Aug/2017:21:15:15 -0700] "POST /cgi-bin/cmd.cgi HTTP/1.1" 301 314 "-" "libwww-perl/6.15"
10.15.174.225 - - [27/Aug/2017:21:15:19 -0700] "POST /cgi-bin/cmd.cgi HTTP/1.1" 301 314 "-" "libwww-perl/6.15"
10.15.174.225 - - [27/Aug/2017:22:15:02 -0700] "POST /cgi-bin/cmd.cgi HTTP/1.1" 301 314 "-" "libwww-perl/6.15"
10.15.174.225 - - [27/Aug/2017:22:15:07 -0700] "POST /cgi-bin/cmd.cgi HTTP/1.1" 301 314 "-" "libwww-perl/6.15"
10.15.174.225 - - [27/Aug/2017:22:15:11 -0700] "POST /cgi-bin/cmd.cgi HTTP/1.1" 301 314 "-" "libwww-perl/6.15"
10.15.174.225 - - [27/Aug/2017:22:15:15 -0700] "POST /cgi-bin/cmd.cgi HTTP/1.1" 301 314 "-" "libwww-perl/6.15"
10.15.174.225 - - [27/Aug/2017:22:15:19 -0700] "POST /cgi-bin/cmd.cgi HTTP/1.1" 301 314 "-" "libwww-perl/6.15"

[root@gms-us01-vm3 libexec]# tail -f /var/log/httpd/error_log
[Sun Aug 27 04:02:02 2017] [notice] Digest: generating secret for digest authentication ...
[Sun Aug 27 04:02:02 2017] [notice] Digest: done
[Sun Aug 27 04:02:02 2017] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads.
[Sun Aug 27 04:02:02 2017] [notice] Apache/2.2.3 (Red Hat)

[cdienger]

Looks to be a disk space issue. Please check the space with df on the core command line. https://www.tecmint.com/find-top-large- ... -in-linux/ covers how to find large folders and files to help figure out where the space is going and clean it up.