Page 2 of 3
Re: 2014 Update - Operations Screen Color Scheme
Posted: Thu May 29, 2014 4:15 pm
by tmcdonald
mberkley wrote:This method did not work for us, so we simply renamed the folder and we were out of impact. We are testing the new component now.
snapon_admin wrote:I'm having this same error (saw the updated component and got excited so I installed it as well). The removal method did not seem to work for me and we're still having issues getting in. I can't even seem to log into the CLI atm...thoughts?
Scott updated his post to fix the removal path. The correct command is:
Code: Select all
rm -rf /usr/local/nagiosxi/html/includes/components/opscreen
Re: 2014 Update - Operations Screen Color Scheme
Posted: Thu May 29, 2014 4:18 pm
by snapon_admin
Yep, I noticed the path was off right away actually and the component is removed, still not working though. Can't get to the web GUI, but I can log into the CLI now.
Code: Select all
[root@lisl-ngos-01-pv ~]# ls /usr/local/nagiosxi/html/includes/components/
actions birdseye deploydashboard homepagemod metrics nagvis pnp scheduledbackups
active_directory bulkmodifications deploynotification hypermap minemap nocscreen profile scheduledreporting
alertcloud capacityplanning escalationwizard hypermap_replay nagiosbpi nrdsconfigmanager proxy similetimeline
alertstream ccm globaleventhandler isms nagioscore objectnotes rdp snmptranslate
autodiscovery componenthelper.inc.php googlemap jpgraph nagioscorecfg pchart rename snmptrapsender
backendapiurl customcolumn graphexplorer latestalerts nagiosim perfdata reportmod tracerouteaction
bandwidthreport customlogin helpsystem ldapauth nagiosna performancedatatool response xicore
bbmap customlogo highcharts massacknowledge nagiosql pingaction rssnotifications
[root@lisl-ngos-01-pv ~]#
No opscreen.
Re: 2014 Update - Operations Screen Color Scheme
Posted: Thu May 29, 2014 4:21 pm
by tmcdonald
Still getting high load? Let's see some logs:
Code: Select all
tail /var/log/httpd/error_log
tail /var/log/httpd/access_log
Re: 2014 Update - Operations Screen Color Scheme
Posted: Thu May 29, 2014 4:22 pm
by snapon_admin
Code: Select all
[root@lisl-ngos-01-pv ~]# tail /var/log/httpd/error_log
[Thu May 29 15:35:17 2014] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu May 29 15:35:18 2014] [notice] Digest: generating secret for digest authentication ...
[Thu May 29 15:35:18 2014] [notice] Digest: done
[Thu May 29 15:35:19 2014] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.0-fips mod_wsgi/3.2 Python/2.6.6 configured -- resuming normal operations
[Thu May 29 15:39:11 2014] [notice] caught SIGTERM, shutting down
[Thu May 29 15:40:54 2014] [notice] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Thu May 29 15:40:54 2014] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu May 29 15:40:54 2014] [notice] Digest: generating secret for digest authentication ...
[Thu May 29 15:40:54 2014] [notice] Digest: done
[Thu May 29 15:40:56 2014] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.0-fips mod_wsgi/3.2 Python/2.6.6 configured -- resuming normal operations
[root@lisl-ngos-01-pv ~]#
Code: Select all
[root@lisl-ngos-01-pv ~]# tail /var/log/httpd/access_log
::1 - - [29/May/2014:15:42:29 -0500] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.2.15 (CentOS) (internal dummy connection)"
::1 - - [29/May/2014:15:42:30 -0500] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.2.15 (CentOS) (internal dummy connection)"
::1 - - [29/May/2014:15:42:31 -0500] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.2.15 (CentOS) (internal dummy connection)"
::1 - - [29/May/2014:15:42:32 -0500] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.2.15 (CentOS) (internal dummy connection)"
::1 - - [29/May/2014:15:42:36 -0500] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.2.15 (CentOS) (internal dummy connection)"
::1 - - [29/May/2014:15:42:36 -0500] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.2.15 (CentOS) (internal dummy connection)"
10.0.8.9 - - [29/May/2014:15:42:36 -0500] "GET / HTTP/1.1" 302 302 "-" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36"
::1 - - [29/May/2014:15:42:36 -0500] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.2.15 (CentOS) (internal dummy connection)"
10.0.8.9 - - [29/May/2014:15:43:16 -0500] "GET / HTTP/1.1" 302 302 "-" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36"
127.0.0.1 - - [29/May/2014:15:43:17 -0500] "GET / HTTP/1.0" 302 289 "-" "check_http/v2.0.1 (nagios-plugins 2.0.1)"
[root@lisl-ngos-01-pv ~]#
Re: 2014 Update - Operations Screen Color Scheme
Posted: Thu May 29, 2014 4:27 pm
by tmcdonald
And the load?
Looking at your post time you had a 15-minute load average of ~600 and Scott hadn't even posted the OpScreen that long ago, so it had to have been prior.
Re: 2014 Update - Operations Screen Color Scheme
Posted: Thu May 29, 2014 4:28 pm
by snapon_admin
I also tried restarting httpd and it can't start. Getting this:
Code: Select all
[root@lisl-ngos-01-pv conf]# service httpd start
Starting httpd: (98)Address already in use: make_sock: could not bind to address [::]:80
(98)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
Unable to open logs
^C [FAILED]
[root@lisl-ngos-01-pv conf]#
Re: 2014 Update - Operations Screen Color Scheme
Posted: Thu May 29, 2014 4:29 pm
by snapon_admin
Load is MASSIVE.
Code: Select all
[root@lisl-ngos-01-pv conf]# w
16:27:56 up 47 min, 1 user, load average: 717.49, 707.94, 661.18
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
root pts/0 kenow7pr6449.sna 16:14 0.00s 445477days 213054days w
Looking at your post time you had a 15-minute load average of ~600 and Scott hadn't even posted the OpScreen that long ago, so it had to have been prior.
Not sure how it got a 15 minute average. I rebooted the server after removing opscreen didn't fix the issue so the server hasn't even been up 15 minutes.
Re: 2014 Update - Operations Screen Color Scheme
Posted: Thu May 29, 2014 4:32 pm
by tmcdonald
Can you post a ps-ef and a top for us?
Re: 2014 Update - Operations Screen Color Scheme
Posted: Thu May 29, 2014 4:33 pm
by snapon_admin
ps -ef is too long for post, attached as txt.
Code: Select all
top - 16:33:11 up 52 min, 1 user, load average: 733.85, 723.44, 681.32
Tasks: 1027 total, 56 running, 961 sleeping, 0 stopped, 10 zombie
Cpu(s): 88.2%us, 1.1%sy, 0.0%ni, 10.4%id, 0.1%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 8061012k total, 1664496k used, 6396516k free, 60520k buffers
Swap: 262136k total, 0k used, 262136k free, 484812k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17621 nagios 20 0 226m 23m 9120 S 100.0 0.3 70045,20 php
17620 nagios 20 0 226m 28m 7436 R 100.0 0.4 900583:01 php
1892 root 20 0 114m 1396 768 S 100.0 0.0 465301,13 crond
17792 nagios 20 0 9232 1064 920 S 100.0 0.0 5118313h sh
2199 nagios 20 0 70920 39m 1500 S 100.0 0.5 2421618h nagios
2208 nagios 20 0 10408 1400 660 R 100.0 0.0 130131,04 nagios
2206 nagios 20 0 10404 1408 660 R 95.0 0.0 105068,51 nagios
2209 nagios 20 0 10260 1312 660 R 95.0 0.0 110072,05 nagios
2213 nagios 20 0 10404 1384 660 R 93.3 0.0 25016,52 nagios
2211 nagios 20 0 10400 1400 660 R 84.3 0.0 525340,27 nagios
17791 root 20 0 15700 1900 832 R 5.4 0.0 5118313h top
1 root 20 0 19356 1592 1264 S 0.0 0.0 8911547h init
2 root 20 0 0 0 0 S 0.0 0.0 470304,27 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 10197379h ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
6 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
7 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/1
8 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/1
9 root 20 0 0 0 0 S 0.0 0.0 60038,52 ksoftirqd/1
10 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
11 root RT 0 0 0 0 R 0.0 0.0 300194:20 migration/2
12 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/2
13 root 20 0 0 0 0 S 0.0 0.0 20012,57 ksoftirqd/2
14 root RT 0 0 0 0 R 0.0 0.0 0:00.00 watchdog/2
15 root RT 0 0 0 0 R 0.0 0.0 300194:20 migration/3
16 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/3
17 root 20 0 0 0 0 R 0.0 0.0 0:00.05 ksoftirqd/3
18 root RT 0 0 0 0 R 0.0 0.0 0:00.00 watchdog/3
19 root RT 0 0 0 0 S 0.0 0.0 0:00.01 migration/4
20 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/4
21 root 20 0 0 0 0 S 0.0 0.0 50032,23 ksoftirqd/4
22 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/4
23 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/5
24 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/5
25 root 20 0 0 0 0 S 0.0 0.0 20012,57 ksoftirqd/5
26 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/5
27 root RT 0 0 0 0 S 0.0 0.0 0:00.02 migration/6
28 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/6
29 root 20 0 0 0 0 S 0.0 0.0 900583:01 ksoftirqd/6
30 root RT 0 0 0 0 S 0.0 0.0 0:00.01 watchdog/6
31 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/7
32 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/7
33 root 20 0 0 0 0 S 0.0 0.0 0:00.05 ksoftirqd/7
34 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/7
35 root 20 0 0 0 0 S 0.0 0.0 10087308h events/0
36 root 20 0 0 0 0 S 0.0 0.0 300194:20 events/1
37 root 20 0 0 0 0 S 0.0 0.0 300194:20 events/2
38 root 20 0 0 0 0 S 0.0 0.0 300194:20 events/3
39 root 20 0 0 0 0 S 0.0 0.0 300194:20 events/4
40 root 20 0 0 0 0 S 0.0 0.0 300194:20 events/5
41 root 20 0 0 0 0 S 0.0 0.0 300194:20 events/6
42 root 20 0 0 0 0 S 0.0 0.0 300194:20 events/7
43 root 20 0 0 0 0 S 0.0 0.0 300194:20 cgroup
44 root 20 0 0 0 0 R 0.0 0.0 600388:40 khelper
Still dead in the water. Ideas?
Re: 2014 Update - Operations Screen Color Scheme
Posted: Thu May 29, 2014 5:00 pm
by tmcdonald
snapon_admin, I sent you an email with a remote session link. Our offices will be closing soon, so if you can get in quick we can spend a few minutes checking things quickly. Otherwise we will be in at 9AM CDT.