Nagios Logserver RAM Problems

[mcapra]

Hmm, nothing indicative of underlying issues within Elasticsearch or Logstash.

Can I get some fresh information about this system? Please share the outputs of the following commands executed from the CLI of your Nagios Log Server machine:

Code: Select all

ps -aef
free -m
df -h
php -r "phpinfo();"
tail -n 30 /var/log/httpd/error_log

[Sven.Z]

Results are send out via pm.

Thanks for fast help!

[mcapra]

Can we also get a fresh output for:

Code: Select all

curl 'localhost:9200/_cat/indices?v'

My suspicion is that swap may be getting used here which can be a pretty bad thing for the JVM in general. I would suggest disabling swap on the machine if possible:

Code: Select all

sudo swapoff -a

If this machine is virtual and runs in a VMWare environment, I would also recommend reserving the memory to prevent VMWare from unintentionally trashing the JVM heap:

2017_01_12_15_30_24_vcenter.nagios.local_vSphere_Client.png

[Sven.Z]

Result for curl 'localhost:9200/_cat/indices?v':

Code: Select all

[root@nagioslogserver ~]# curl 'localhost:9200/_cat/indices?v'
health status index               pri rep docs.count docs.deleted store.size pri.store.size
       close  logstash-2017.01.04
yellow open   nagioslogserver_log   5   1    1286677            0      129mb          129mb
       close  logstash-2017.01.10
yellow open   logstash-2017.01.13   5   1     204075            0     83.7mb         83.7mb
       close  logstash-2017.01.02
yellow open   nagioslogserver       1   1         61            6    102.4kb        102.4kb
       close  logstash-2017.01.03
       close  logstash-2017.01.05
       close  logstash-2017.01.08
       close  logstash-2017.01.09
       close  logstash-2017.01.01
yellow open   kibana-int            5   1         11            0      106kb          106kb
       close  logstash-2017.01.06
       close  logstash-2017.01.07
yellow open   logstash-2017.01.12   5   1     478277            0     98.8mb         98.8mb
yellow open   logstash-2017.01.11   5   1     475737            0     98.5mb         98.5mb

have do your tips. no swap and reserve all memory for this host.

[mcapra]

Did adjusting the VM settings and disabling swap offer any performance increases, or are you still having troubles with the GUI loading slowly?

[Sven.Z]

The Nagioslogserver was after a certain time window no longer accessible via the web interface. SSH access was always possible. The GUI was never slow. The suspicion was that the system ran in the SWAP and thus did fail. I hope the problem by the changes (SWAP off, lists 0.0.0.0:80, VM-Ware Settings) no longer occurs. I will watch the system the next few days.

[Sven.Z]

One Problem is still there.

If I call the Nagioslogserver frontend and authenticate me, this error occurs after authenticate in httpd error_log.

Code: Select all

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to 2600:3c00::f03c:91ff:fe18:849c: Network is unreachable

What can I do about it?

A little internetsearching give me an answer that is a ip Address from http://www.linode.com Cloud Service i think.
http://sixy.ch/whois?addr=2600%3A3c00%3 ... y=a64fb533
Why do Nagioslogserver Contract a Cloudservice or why the apache Service do that?

Code: Select all

[Querying whois.arin.net]
[whois.arin.net]

#
# ARIN WHOIS data and services are subject to the Terms of Use
# available at: https://www.arin.net/whois_tou.html
#
# If you see inaccuracies in the results, please report at
# https://www.arin.net/public/whoisinaccuracy/index.xhtml
#


#
# The following results may also be obtained via:
# https://whois.arin.net/rest/nets;q=2600%3A3c00%3A%3Af03c%3A91ff%3Afe18%3A849c?showDetails=true&showARIN=false&showNonArinTopLevelNet=false&ext=netref2
#

NetRange:       2600:3C00:: - 2600:3C03:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF
CIDR:           2600:3C00::/30
NetName:        LINODE-US
NetHandle:      NET6-2600-3C00-1
Parent:         NET6-2600 (NET6-2600-1)
NetType:        Direct Allocation
OriginAS:       AS3595, AS36351, AS6939, AS15830, AS8001, AS2516
Organization:   Linode (LINOD)
RegDate:        2011-02-07
Updated:        2014-09-11
Comment:        This block is used for static customer allocations.
Ref:            https://whois.arin.net/rest/net/NET6-2600-3C00-1



OrgName:        Linode
OrgId:          LINOD
Address:        329 E. Jimmie Leeds Road
Address:        Suite A
City:           Galloway
StateProv:      NJ
PostalCode:     08205
Country:        US
RegDate:        2008-04-24
Updated:        2014-08-01
Comment:        http://www.linode.com
Ref:            https://whois.arin.net/rest/org/LINOD


OrgNOCHandle: LNO21-ARIN
OrgNOCName:   Linode Network Operations
OrgNOCPhone:  +1-609-380-7304 
OrgNOCEmail:  [email protected]
OrgNOCRef:    https://whois.arin.net/rest/poc/LNO21-ARIN

OrgTechHandle: LNO21-ARIN
OrgTechName:   Linode Network Operations
OrgTechPhone:  +1-609-380-7304 
OrgTechEmail:  [email protected]
OrgTechRef:    https://whois.arin.net/rest/poc/LNO21-ARIN

OrgAbuseHandle: LAS12-ARIN
OrgAbuseName:   Linode Abuse Support
OrgAbusePhone:  +1-609-380-7100 
OrgAbuseEmail:  [email protected]
OrgAbuseRef:    https://whois.arin.net/rest/poc/LAS12-ARIN

RTechHandle: LNO21-ARIN
RTechName:   Linode Network Operations
RTechPhone:  +1-609-380-7304 
RTechEmail:  [email protected]
RTechRef:    https://whois.arin.net/rest/poc/LNO21-ARIN

RAbuseHandle: LAS12-ARIN
RAbuseName:   Linode Abuse Support
RAbusePhone:  +1-609-380-7100 
RAbuseEmail:  [email protected]
RAbuseRef:    https://whois.arin.net/rest/poc/LAS12-ARIN

RNOCHandle: LNO21-ARIN
RNOCName:   Linode Network Operations
RNOCPhone:  +1-609-380-7304 
RNOCEmail:  [email protected]
RNOCRef:    https://whois.arin.net/rest/poc/LNO21-ARIN


#
# ARIN WHOIS data and services are subject to the Terms of Use
# available at: https://www.arin.net/whois_tou.html
#
# If you see inaccuracies in the results, please report at
# https://www.arin.net/public/whoisinaccuracy/index.xhtml
#

next research about this ip here a hit:

https://www.qualidator.com/WQM/de/Tools ... nagios.com

Code: Select all

com1.nagios.com	AAAA	60	2600:3c00:0:0:f03c:91ff:fe18:849c

it is your com1.nagios.com entry for dns.

my Nagioslogserver didn´t have a ipv6 address for connection over ipv6.

[mcapra]

Quite a thorough investigation you have done

We use Linode for hosting of some internal services. In this case, I suspect that this is is part of the run_update_check Command Subsystem job. This will (among other things) attempt to reach out to Nagios servers to ask what the latest version of Nagios Log Server is. I would say the error is safe to ignore for now.