Page 3 of 3
Re: Nagios Logserver RAM Problems
Posted: Wed Jan 11, 2017 12:57 pm
by mcapra
Hmm, nothing indicative of underlying issues within Elasticsearch or Logstash.
Can I get some fresh information about this system? Please share the outputs of the following commands executed from the CLI of your Nagios Log Server machine:
Code: Select all
ps -aef
free -m
df -h
php -r "phpinfo();"
tail -n 30 /var/log/httpd/error_log
Re: Nagios Logserver RAM Problems
Posted: Thu Jan 12, 2017 8:39 am
by Sven.Z
Results are send out via pm.
Thanks for fast help!
Re: Nagios Logserver RAM Problems
Posted: Thu Jan 12, 2017 4:31 pm
by mcapra
Can we also get a fresh output for:
Code: Select all
curl 'localhost:9200/_cat/indices?v'
My suspicion is that swap may be getting used here which can be a pretty bad thing for the JVM in general. I would suggest disabling swap on the machine if possible:
If this machine is virtual and runs in a VMWare environment, I would also recommend reserving the memory to prevent VMWare from unintentionally trashing the JVM heap:
2017_01_12_15_30_24_vcenter.nagios.local_vSphere_Client.png
Re: Nagios Logserver RAM Problems
Posted: Fri Jan 13, 2017 6:00 am
by Sven.Z
Result for curl 'localhost:9200/_cat/indices?v':
Code: Select all
[root@nagioslogserver ~]# curl 'localhost:9200/_cat/indices?v'
health status index pri rep docs.count docs.deleted store.size pri.store.size
close logstash-2017.01.04
yellow open nagioslogserver_log 5 1 1286677 0 129mb 129mb
close logstash-2017.01.10
yellow open logstash-2017.01.13 5 1 204075 0 83.7mb 83.7mb
close logstash-2017.01.02
yellow open nagioslogserver 1 1 61 6 102.4kb 102.4kb
close logstash-2017.01.03
close logstash-2017.01.05
close logstash-2017.01.08
close logstash-2017.01.09
close logstash-2017.01.01
yellow open kibana-int 5 1 11 0 106kb 106kb
close logstash-2017.01.06
close logstash-2017.01.07
yellow open logstash-2017.01.12 5 1 478277 0 98.8mb 98.8mb
yellow open logstash-2017.01.11 5 1 475737 0 98.5mb 98.5mb
have do your tips. no swap and reserve all memory for this host.
Re: Nagios Logserver RAM Problems
Posted: Fri Jan 13, 2017 11:41 am
by mcapra
Did adjusting the VM settings and disabling swap offer any performance increases, or are you still having troubles with the GUI loading slowly?
Re: Nagios Logserver RAM Problems
Posted: Mon Jan 16, 2017 2:14 am
by Sven.Z
The Nagioslogserver was after a certain time window no longer accessible via the web interface. SSH access was always possible. The GUI was never slow. The suspicion was that the system ran in the SWAP and thus did fail. I hope the problem by the changes (SWAP off, lists 0.0.0.0:80, VM-Ware Settings) no longer occurs. I will watch the system the next few days.
Re: Nagios Logserver RAM Problems
Posted: Mon Jan 16, 2017 3:37 am
by Sven.Z
One Problem is still there.
If I call the Nagioslogserver frontend and authenticate me, this error occurs after authenticate in httpd error_log.
Code: Select all
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to 2600:3c00::f03c:91ff:fe18:849c: Network is unreachable
What can I do about it?
A little internetsearching give me an answer that is a ip Address from
http://www.linode.com Cloud Service i think.
http://sixy.ch/whois?addr=2600%3A3c00%3 ... y=a64fb533
Why do Nagioslogserver Contract a Cloudservice or why the apache Service do that?
Code: Select all
[Querying whois.arin.net]
[whois.arin.net]
#
# ARIN WHOIS data and services are subject to the Terms of Use
# available at: https://www.arin.net/whois_tou.html
#
# If you see inaccuracies in the results, please report at
# https://www.arin.net/public/whoisinaccuracy/index.xhtml
#
#
# The following results may also be obtained via:
# https://whois.arin.net/rest/nets;q=2600%3A3c00%3A%3Af03c%3A91ff%3Afe18%3A849c?showDetails=true&showARIN=false&showNonArinTopLevelNet=false&ext=netref2
#
NetRange: 2600:3C00:: - 2600:3C03:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF
CIDR: 2600:3C00::/30
NetName: LINODE-US
NetHandle: NET6-2600-3C00-1
Parent: NET6-2600 (NET6-2600-1)
NetType: Direct Allocation
OriginAS: AS3595, AS36351, AS6939, AS15830, AS8001, AS2516
Organization: Linode (LINOD)
RegDate: 2011-02-07
Updated: 2014-09-11
Comment: This block is used for static customer allocations.
Ref: https://whois.arin.net/rest/net/NET6-2600-3C00-1
OrgName: Linode
OrgId: LINOD
Address: 329 E. Jimmie Leeds Road
Address: Suite A
City: Galloway
StateProv: NJ
PostalCode: 08205
Country: US
RegDate: 2008-04-24
Updated: 2014-08-01
Comment: http://www.linode.com
Ref: https://whois.arin.net/rest/org/LINOD
OrgNOCHandle: LNO21-ARIN
OrgNOCName: Linode Network Operations
OrgNOCPhone: +1-609-380-7304
OrgNOCEmail: [email protected]
OrgNOCRef: https://whois.arin.net/rest/poc/LNO21-ARIN
OrgTechHandle: LNO21-ARIN
OrgTechName: Linode Network Operations
OrgTechPhone: +1-609-380-7304
OrgTechEmail: [email protected]
OrgTechRef: https://whois.arin.net/rest/poc/LNO21-ARIN
OrgAbuseHandle: LAS12-ARIN
OrgAbuseName: Linode Abuse Support
OrgAbusePhone: +1-609-380-7100
OrgAbuseEmail: [email protected]
OrgAbuseRef: https://whois.arin.net/rest/poc/LAS12-ARIN
RTechHandle: LNO21-ARIN
RTechName: Linode Network Operations
RTechPhone: +1-609-380-7304
RTechEmail: [email protected]
RTechRef: https://whois.arin.net/rest/poc/LNO21-ARIN
RAbuseHandle: LAS12-ARIN
RAbuseName: Linode Abuse Support
RAbusePhone: +1-609-380-7100
RAbuseEmail: [email protected]
RAbuseRef: https://whois.arin.net/rest/poc/LAS12-ARIN
RNOCHandle: LNO21-ARIN
RNOCName: Linode Network Operations
RNOCPhone: +1-609-380-7304
RNOCEmail: [email protected]
RNOCRef: https://whois.arin.net/rest/poc/LNO21-ARIN
#
# ARIN WHOIS data and services are subject to the Terms of Use
# available at: https://www.arin.net/whois_tou.html
#
# If you see inaccuracies in the results, please report at
# https://www.arin.net/public/whoisinaccuracy/index.xhtml
#
next research about this ip here a hit:
https://www.qualidator.com/WQM/de/Tools ... nagios.com
Code: Select all
com1.nagios.com AAAA 60 2600:3c00:0:0:f03c:91ff:fe18:849c
it is your com1.nagios.com entry for dns.
my Nagioslogserver didn´t have a ipv6 address for connection over ipv6.
Re: Nagios Logserver RAM Problems
Posted: Mon Jan 16, 2017 11:18 am
by mcapra
Quite a thorough investigation you have done
We use Linode for hosting of some internal services. In this case, I suspect that this is is part of the
run_update_check Command Subsystem job. This will (among other things) attempt to reach out to Nagios servers to ask what the latest version of Nagios Log Server is. I would say the error is safe to ignore for now.