Re: Nagios Logserver RAM Problems
Posted: Wed Dec 21, 2016 5:57 am
I will check about a long time zone 3-4 weeks before i can say okay the problem is solved. first thanks for the fast help have a nice happy christmas.
Support for Nagios products and services
https://support.nagios.com/forum/
Merry Christmas to you too!Sven.Z wrote:have a nice happy christmas.
Code: Select all
top - 08:14:26 up 15 days, 23:35, 1 user, load average: 0,16, 0,16, 0,11
Tasks: 133 total, 1 running, 121 sleeping, 0 stopped, 11 zombie
%Cpu(s): 0,3 us, 0,4 sy, 0,2 ni, 99,0 id, 0,0 wa, 0,0 hi, 0,1 si, 0,0 st
KiB Mem : 12139968 total, 1511052 free, 4051020 used, 6577896 buff/cache
KiB Swap: 3145724 total, 3145724 free, 0 used. 7427684 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7876 root 20 0 0 0 0 S 9,0 0,0 1:00.91 kworker/u8:1
1306 nagios 20 0 6973264 3,353g 31656 S 5,0 29,0 1913:57 java
8405 root 39 19 3255140 347800 16204 S 1,0 2,9 221:08.13 java
23056 root 20 0 146128 2012 1412 R 0,3 0,0 0:00.46 top
1 root 20 0 213212 28232 2400 S 0,0 0,2 29:37.28 systemd
2 root 20 0 0 0 0 S 0,0 0,0 0:00.81 kthreadd
3 root 20 0 0 0 0 S 0,0 0,0 0:34.54 ksoftirqd/0
5 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 kworker/0:0H
7 root rt 0 0 0 0 S 0,0 0,0 0:10.93 migration/0
8 root 20 0 0 0 0 S 0,0 0,0 0:00.00 rcu_bh
9 root 20 0 0 0 0 S 0,0 0,0 0:00.00 rcuob/0
10 root 20 0 0 0 0 S 0,0 0,0 0:00.00 rcuob/1
11 root 20 0 0 0 0 S 0,0 0,0 0:00.00 rcuob/2
12 root 20 0 0 0 0 S 0,0 0,0 0:00.00 rcuob/3
13 root 20 0 0 0 0 S 0,0 0,0 8:48.89 rcu_sched
14 root 20 0 0 0 0 S 0,0 0,0 2:18.39 rcuos/0
15 root 20 0 0 0 0 S 0,0 0,0 2:31.35 rcuos/1
16 root 20 0 0 0 0 S 0,0 0,0 2:43.71 rcuos/2
17 root 20 0 0 0 0 S 0,0 0,0 2:59.63 rcuos/3
18 root rt 0 0 0 0 S 0,0 0,0 0:31.13 watchdog/0
19 root rt 0 0 0 0 S 0,0 0,0 0:19.56 watchdog/1
20 root rt 0 0 0 0 S 0,0 0,0 4:03.15 migration/1
21 root 20 0 0 0 0 S 0,0 0,0 7:49.69 ksoftirqd/1
23 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 kworker/1:0H
24 root rt 0 0 0 0 S 0,0 0,0 0:14.41 watchdog/2
25 root rt 0 0 0 0 S 0,0 0,0 1:15.78 migration/2
26 root 20 0 0 0 0 S 0,0 0,0 0:49.68 ksoftirqd/2
29 root rt 0 0 0 0 S 0,0 0,0 0:13.89 watchdog/3
30 root rt 0 0 0 0 S 0,0 0,0 1:25.62 migration/3
31 root 20 0 0 0 0 S 0,0 0,0 0:40.36 ksoftirqd/3
33 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 kworker/3:0H
34 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 khelper
35 root 20 0 0 0 0 S 0,0 0,0 0:00.00 kdevtmpfs
36 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 netns
37 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 perf
38 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 writeback
39 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 kintegrityd
40 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 bioset
41 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 kblockd
42 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 md
50 root 20 0 0 0 0 S 0,0 0,0 0:00.80 khungtaskd
51 root 20 0 0 0 0 S 0,0 0,0 0:00.00 kswapd0
52 root 25 5 0 0 0 S 0,0 0,0 0:00.00 ksmd
53 root 39 19 0 0 0 S 0,0 0,0 0:04.17 khugepaged
54 root 20 0 0 0 0 S 0,0 0,0 0:00.00 fsnotify_mark
55 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 crypto
63 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 kthrotld
65 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 kmpath_rdacd
66 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 kpsmoused
68 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 ipv6_addrconf
87 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 deferwq
118 root 20 0 0 0 0 S 0,0 0,0 0:37.04 kauditd
295 root 0 -20 0 0 0 S 0,0 0,0 0:00.00 ata_sff
[root@nagioslogserver ~]# curl 'localhost:9200/_cat/indices?v'
health status index pri rep docs.count docs.deleted store.size pri.store.size
yellow open logstash-2017.01.04 5 1 426089 0 89.5mb 89.5mb
yellow open nagioslogserver_log 5 1 1220983 0 110.8mb 110.8mb
close logstash-2016.12.28
close logstash-2017.01.02
yellow open nagioslogserver 1 1 59 5 102.5kb 102.5kb
yellow open logstash-2017.01.03 5 1 346846 0 73.1mb 73.1mb
yellow open logstash-2017.01.05 5 1 103364 0 52.6mb 52.6mb
close logstash-2016.12.30
close logstash-2016.12.24
close logstash-2016.12.26
yellow open kibana-int 5 1 11 0 106kb 106kb
close logstash-2016.12.27
close logstash-2016.12.29
close logstash-2016.12.25
close logstash-2016.12.31
close logstash-2017.01.01Code: Select all
[root@nagioslogserver ~]# systemctl httpd.service status
Unknown operation 'httpd.service'.
[root@nagioslogserver ~]# systemctl status httpd.service
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
Active: active (running) since Do 2016-12-22 08:51:28 CET; 1 weeks 6 days ago
Docs: man:httpd(8)
man:apachectl(8)
Process: 12652 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=1/FAILURE)
Process: 1135 ExecReload=/usr/sbin/httpd $OPTIONS -k graceful (code=exited, status=0/SUCCESS)
Main PID: 13111 (httpd)
Status: "Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec"
CGroup: /system.slice/httpd.service
└─13111 /usr/sbin/httpd -DFOREGROUND
Jan 04 08:13:41 sudo[16167]: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/et... status
Jan 04 08:13:41 sudo[16168]: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/et... status
Jan 04 08:14:07 sudo[16255]: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/et... status
Jan 04 08:14:07 sudo[16256]: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/et... status
Jan 04 08:14:18 sudo[16308]: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/et... status
Jan 04 08:14:18 sudo[16309]: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/et... status
Jan 04 08:14:22 sudo[16333]: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/et... status
Jan 04 08:14:22 sudo[16334]: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/et... status
Jan 04 08:14:33 sudo[16383]: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/et... status
Jan 04 08:14:33 sudo[16384]: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/et... status
Hint: Some lines were ellipsized, use -l to show in full.Code: Select all
Jan 04 08:14:33 sudo[16384]: apache : TTY=unknown ; PWD=/var/www/html/nagioslogserver/www ; USER=root ; COMMAND=/etc/init.d/elasticsearch statusCode: Select all
Alias /nagioslogserver "/var/www/html/nagioslogserver/www/"
<Directory "/var/www/html/nagioslogserver/www/">
SSLRequireSSL
Options FollowSymLinks
AllowOverride None
Order allow,deny
Allow from all
</Directory>
RewriteEngine on
RewriteCond $1 !^(index\.php|scripts|media|app|js|css|img|font|vendor|config.js)
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule nagioslogserver/(.*)$ /var/www/html/nagioslogserver/www/index.php/$1 [L,QSA]
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}/nagioslogserver [L,QSA]Code: Select all
# When we also provide SSL we have to listen to the
# the HTTPS port in addition.
#
Listen 443 https
##
## SSL Global Context
##
## All SSL configuration in this context applies both to
## the main server and all SSL-enabled virtual hosts.
##
# Pass Phrase Dialog:
# Configure the pass phrase gathering process.
# The filtering dialog program (`builtin' is a internal
# terminal dialog) has to provide the pass phrase on stdout.
SSLPassPhraseDialog exec:/usr/libexec/httpd-ssl-pass-dialog
# Inter-Process Session Cache:
# Configure the SSL Session Cache: First the mechanism
# to use and second the expiring timeout (in seconds).
SSLSessionCache shmcb:/run/httpd/sslcache(512000)
SSLSessionCacheTimeout 300
# Pseudo Random Number Generator (PRNG):
# Configure one or more sources to seed the PRNG of the
# SSL library. The seed data should be of good random quality.
# WARNING! On some platforms /dev/random blocks if not enough entropy
# is available. This means you then cannot use the /dev/random device
# because it would lead to very long connection times (as long as
# it requires to make more entropy available). But usually those
# platforms additionally provide a /dev/urandom device which doesn't
# block. So, if available, use this one instead. Read the mod_ssl User
# Manual for more details.
SSLRandomSeed startup file:/dev/urandom 256
SSLRandomSeed connect builtin
#SSLRandomSeed startup file:/dev/random 512
#SSLRandomSeed connect file:/dev/random 512
#SSLRandomSeed connect file:/dev/urandom 512
#
# Use "SSLCryptoDevice" to enable any supported hardware
# accelerators. Use "openssl engine -v" to list supported
# engine names. NOTE: If you enable an accelerator and the
# server does not start, consult the error logs and ensure
# your accelerator is functioning properly.
#
SSLCryptoDevice builtin
#SSLCryptoDevice ubsec
##
## SSL Virtual Host Context
##
<VirtualHost _default_:443>
# General setup for the virtual host, inherited from global configuration
#DocumentRoot "/var/www/html"
#ServerName www.example.com:443
# Use separate log files for the SSL virtual host; note that LogLevel
# is not inherited from httpd.conf.
ErrorLog logs/ssl_error_log
TransferLog logs/ssl_access_log
LogLevel warn
# SSL Engine Switch:
# Enable/Disable SSL for this virtual host.
SSLEngine on
# SSL Protocol support:
# List the enable protocol levels with which clients will be able to
# connect. Disable SSLv2 access by default:
SSLProtocol all -SSLv2
# SSL Cipher Suite:
# List the ciphers that the client is permitted to negotiate.
# See the mod_ssl documentation for a complete list.
SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5:!SEED:!IDEA
# Speed-optimized SSL Cipher configuration:
# If speed is your main concern (on busy HTTPS servers e.g.),
# you might want to force clients to specific, performance
# optimized ciphers. In this case, prepend those ciphers
# to the SSLCipherSuite list, and enable SSLHonorCipherOrder.
# Caveat: by giving precedence to RC4-SHA and AES128-SHA
# (as in the example below), most connections will no longer
# have perfect forward secrecy - if the server's key is
# compromised, captures of past or future traffic must be
# considered compromised, too.
#SSLCipherSuite RC4-SHA:AES128-SHA:HIGH:MEDIUM:!aNULL:!MD5
#SSLHonorCipherOrder on
# Server Certificate:
# Point SSLCertificateFile at a PEM encoded certificate. If
# the certificate is encrypted, then you will be prompted for a
# pass phrase. Note that a kill -HUP will prompt again. A new
# certificate can be generated using the genkey(1) command.
SSLCertificateFile /etc/pki/tls/certs/ca.crt
# Server Private Key:
# If the key is not combined with the certificate, use this
# directive to point at the key file. Keep in mind that if
# you've both a RSA and a DSA private key you can configure
# both in parallel (to also allow the use of DSA ciphers, etc.)
SSLCertificateKeyFile /etc/pki/tls/private/ca.key
# Server Certificate Chain:
# Point SSLCertificateChainFile at a file containing the
# concatenation of PEM encoded CA certificates which form the
# certificate chain for the server certificate. Alternatively
# the referenced file can be the same as SSLCertificateFile
# when the CA certificates are directly appended to the server
# certificate for convinience.
#SSLCertificateChainFile /etc/pki/tls/certs/server-chain.crt
# Certificate Authority (CA):
# Set the CA certificate verification path where to find CA
# certificates for client authentication or alternatively one
# huge file containing all of them (file must be PEM encoded)
#SSLCACertificateFile /etc/pki/tls/certs/ca-bundle.crt
# Client Authentication (Type):
# Client certificate verification type and depth. Types are
# none, optional, require and optional_no_ca. Depth is a
# number which specifies how deeply to verify the certificate
# issuer chain before deciding the certificate is not valid.
#SSLVerifyClient require
#SSLVerifyDepth 10
# Access Control:
# With SSLRequire you can do per-directory access control based
# on arbitrary complex boolean expressions containing server
# variable checks and other lookup directives. The syntax is a
# mixture between C and Perl. See the mod_ssl documentation
# for more details.
#<Location />
#SSLRequire ( %{SSL_CIPHER} !~ m/^(EXP|NULL)/ \
# and %{SSL_CLIENT_S_DN_O} eq "Snake Oil, Ltd." \
# and %{SSL_CLIENT_S_DN_OU} in {"Staff", "CA", "Dev"} \
# and %{TIME_WDAY} >= 1 and %{TIME_WDAY} <= 5 \
# and %{TIME_HOUR} >= 8 and %{TIME_HOUR} <= 20 ) \
# or %{REMOTE_ADDR} =~ m/^192\.76\.162\.[0-9]+$/
#</Location>
# SSL Engine Options:
# Set various options for the SSL engine.
# o FakeBasicAuth:
# Translate the client X.509 into a Basic Authorisation. This means that
# the standard Auth/DBMAuth methods can be used for access control. The
# user name is the `one line' version of the client's X.509 certificate.
# Note that no password is obtained from the user. Every entry in the user
# file needs this password: `*******'. (edited)
# o ExportCertData:
# This exports two additional environment variables: SSL_CLIENT_CERT and
# SSL_SERVER_CERT. These contain the PEM-encoded certificates of the
# server (always existing) and the client (only existing when client
# authentication is used). This can be used to import the certificates
# into CGI scripts.
# o StdEnvVars:
# This exports the standard SSL/TLS related `SSL_*' environment variables.
# Per default this exportation is switched off for performance reasons,
# because the extraction step is an expensive operation and is usually
# useless for serving static content. So one usually enables the
# exportation for CGI and SSI requests only.
# o StrictRequire:
# This denies access when "SSLRequireSSL" or "SSLRequire" applied even
# under a "Satisfy any" situation, i.e. when it applies access is denied
# and no other module can change it.
# o OptRenegotiate:
# This enables optimized SSL connection renegotiation handling when SSL
# directives are used in per-directory context.
#SSLOptions +FakeBasicAuth +ExportCertData +StrictRequire
<Files ~ "\.(cgi|shtml|phtml|php3?)$">
SSLOptions +StdEnvVars
</Files>
<Directory "/var/www/cgi-bin">
SSLOptions +StdEnvVars
</Directory>
# SSL Protocol Adjustments:
# The safe and default but still SSL/TLS standard compliant shutdown
# approach is that mod_ssl sends the close notify alert but doesn't wait for
# the close notify alert from client. When you need a different shutdown
# approach you can use one of the following variables:
# o ssl-unclean-shutdown:
# This forces an unclean shutdown when the connection is closed, i.e. no
# SSL close notify alert is send or allowed to received. This violates
# the SSL/TLS standard but is needed for some brain-dead browsers. Use
# this when you receive I/O errors because of the standard approach where
# mod_ssl sends the close notify alert.
# o ssl-accurate-shutdown:
# This forces an accurate shutdown when the connection is closed, i.e. a
# SSL close notify alert is send and mod_ssl waits for the close notify
# alert of the client. This is 100% SSL/TLS standard compliant, but in
# practice often causes hanging connections with brain-dead browsers. Use
# this only for browsers where you know that their SSL implementation
# works correctly.
# Notice: Most problems of broken clients are also related to the HTTP
# keep-alive facility, so you usually additionally want to disable
# keep-alive for those clients, too. Use variable "nokeepalive" for this.
# Similarly, one has to force some clients to use HTTP/1.0 to workaround
# their broken HTTP/1.1 implementation. Use variables "downgrade-1.0" and
# "force-response-1.0" for this.
BrowserMatch "MSIE [2-5]" \
nokeepalive ssl-unclean-shutdown \
downgrade-1.0 force-response-1.0
# Per-Server Logging:
# The home of a custom SSL log file. Use this when you want a
# compact non-error SSL logfile on a virtual host basis.
CustomLog logs/ssl_request_log \
"%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
RewriteEngine on
RewriteCond $1 !^(index\.php|scripts|media|app|js|css|img|font|vendor|config.js)
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule nagioslogserver/(.*)$ /var/www/html/nagioslogserver/www/index.php/$1 [L,QSA]
</VirtualHost>Code: Select all
* About to connect() to 172.17.17.77 port 443 (#0)
* Trying 172.17.17.77...
* Connected to 172.17.17.77 (172.17.17.77) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* Server certificate:
* subject: not necessary(i have edited)
* start date: Dez 22 07:50:39 2016 GMT
* expire date: Dez 22 07:50:39 2017 GMT
* common name: nagioslogserver
* issuer: not necessary(i have edited)
* NSS error -8172 (SEC_ERROR_UNTRUSTED_ISSUER) (because it is self signed)
* Peer's certificate issuer has been marked as not trusted by the user.
* Closing connection 0
curl: (60) Peer's certificate issuer has been marked as not trusted by the user.
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
Code: Select all
[root@localhost ~]# curl -vk https://172.17.17.77/nagioslogserver/
* About to connect() to 172.17.17.77 port 443 (#0)
* Trying 172.17.17.77...
* Connected to 172.17.17.77 (172.17.17.77) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* subject: not necessary(i have edited)
* start date: Dez 22 07:50:39 2016 GMT
* expire date: Dez 22 07:50:39 2017 GMT
* common name: nagioslogserver
* issuer: not necessary(i have edited)
> GET /nagioslogserver/ HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 172.17.17.77
> Accept: */*
>
< HTTP/1.1 302 Found
< Date: Tue, 10 Jan 2017 07:53:27 GMT
< Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips PHP/5.4.16
< X-Powered-By: PHP/5.4.16
< Set-Cookie: ls_session=a%3A5%3A%7Bs%3A10%3A%22session_id%22%3Bs%3A32%3A%22cb721dd2ae85d7dadea662b7a26c8241%22%3Bs%3A10%3A%22ip_address%22%3Bs%3A12%3A%22172.17.17.59%22%3Bs%3A10%3A%22user_agent%22%3Bs%3A11%3A%22curl%2F7.29.0%22%3Bs%3A13%3A%22last_activity%22%3Bi%3A1484034807%3Bs%3A9%3A%22user_data%22%3Bs%3A0%3A%22%22%3B%7D45a2e3ef6a1aec199b4ad0607a5b797ecdd65c03; expires=Tue, 10-Jan-2017 09:53:27 GMT; path=/
< Location: /nagioslogserver/login?redirect=
< Content-Length: 0
< Content-Type: text/html; charset=UTF-8
<
* Connection #0 to host 172.17.17.77 left intact
Code: Select all
/var/log/logstash/
/var/log/easticsearch/