jolson wrote:Mitch,
To be clear - the backup script that is hanging is /usr/local/nagioslogserver/scripts/create_backup.sh. This script controls configuration backup, not the backup of your actual logs. If you take a look at the filesizes of the backups in /store/backups/nagioslogserver/ you will see what I mean:
Code: Select all
[root@localhost ~]# ls -lh /store/backups/nagioslogserver/
-rw-r--r-- 1 nagios users 155K Apr 8 09:55 nagioslogserver.2015-04-07.1428419172.tar.gz
-rw-r--r-- 1 nagios users 300K Apr 8 10:06 nagioslogserver.2015-04-08.1428505577.tar.gz
-rw-r--r-- 1 nagios users 1.3M Apr 15 10:06 nagioslogserver.2015-04-15.1429110402.tar.gz
-rw-r--r-- 1 nagios users 1.5M Apr 16 10:06 nagioslogserver.2015-04-16.1429196807.tar.gz
-rw-r--r-- 1 nagios users 1.6M Apr 17 10:07 nagioslogserver.2015-04-17.1429283221.tar.gz
These backups are not controlled from the 'Backup and Maintenance' screen - they are pre-configured and automatic. For more information on this issue, another customer is experiencing similar symptoms:
http://support.nagios.com/forum/viewtop ... 38&t=32218
Thanks for the explanation. Then I have another issue since I'm not getting any backups in the setup repository space, but that is for another time
jolson wrote:I am having trouble reproducing this problem in lab, and due to that I can't open a bug report on it. I need to know how/why this is happening. Any chance you could collect the following information for me?
-The amount of nodes in your cluster
Only 1
jolson wrote:-The amount of logs coming in per 10 minutes (you can see this on your dashboard)
count per 10m | (7442402 hits) Is that what you are asking for?
jolson wrote:-Any custom logstash inputs/filters/outputs? If so, could you list them?
Code: Select all
#
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Wed, 29 Apr 2015 12:39:20 -0400
#
#
# Global Configuration
#
input {
syslog {
type => 'syslog'
port => 5544
}
tcp {
type => 'eventlog'
port => 3515
codec => json {
charset => 'CP1252'
}
}
tcp {
type => 'import_raw'
tags => 'import_raw'
port => 2056
}
tcp {
type => 'import_json'
tags => 'import_json'
port => 2057
codec => json
}
syslog {
type => 'syslog'
port => 514
}
syslog {
type => 'asa'
port => 6544
}
}
filter {
if [program] == 'apache_access' {
grok {
match => [ 'message', '%{COMBINEDAPACHELOG}']
}
date {
match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
}
mutate {
replace => [ 'type', 'apache_access' ]
convert => [ 'bytes', 'integer' ]
convert => [ 'response', 'integer' ]
}
}
if [program] == 'apache_error' {
grok {
match => [ 'message', '\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:class}\] \[%{WORD:originator} %{IP:clientip}\] %{GREEDYDATA:errmsg}']
}
mutate {
replace => [ 'type', 'apache_error' ]
}
}
if [program] == 'TrexSyncPubRep' {
mutate {
replace => [ 'type', 'TrexSyncPubRep' ]
}
}
if [type] == 'asa' {
grok{
match => ['message', '%{SYSLOG5424PRI}%%{WORD:LogType}-%{INT:LogSeverity}-%{INT:LogMessageNumber}: Group = %{IPORHOST:Group}, Username = %{IPORHOST:username}, IP = %{IP:IPAddress}, Session disconnected. Session Type: %{WORD:SessionType}, Duration: %{CUSTOM1:DurationDays=[0-9]?}%{CUSTOM2=d? ?}%{INT:DurationHours:int}h:%{INT:DurationMinutes:int}m:%{INT:DurationSeconds:int}s, Bytes xmt: %{INT:BytesTransmitted:int}, Bytes rcv: %{INT:BytesReceived:int}, Reason: %{GREEDYDATA:Reason}']
}
geoip {
source => "IPAddress"
}
}
if [program] == 'apache_access' {
geoip {
source => 'clientip'
}
}
if [program] == 'TrexSyncRep' {
mutate {
replace => [ 'type', 'TrexSyncRep' ]
}
}
if [program] == 'Jupiter_log' {
mutate {
replace => [ 'type', 'Jupiter' ]
}
}
if [program] == 'diablo_in1_video_management' {
mutate {
replace => [ 'type', 'diablo' ]
}
}
if [program] == 'PUB_API_ACCESS' {
mutate {
replace => [ 'type', 'APIaccess' ]
}
}
if [program] == 'sudo' {
mutate {
replace => [ 'type', 'sudo' ]
}
}
if [program] == 'opt_lrms_logs_cmgopher' {
mutate {
replace => [ 'type', 'CMGopher_LRMS' ]
}
}
if [program] == 'var_opt_lrms_log_uam' {
mutate {
replace => [ 'type', 'UAMGopher_LRMS' ]
}
}
if [program] == 'opt_lrms_logs_uam' {
mutate {
replace => [ 'type', 'UAMGopher_LRMS' ]
}
}
if [program] == 'opt_lrms_logs_cm' {
mutate {
replace => [ 'type', 'CM_LRMS' ]
}
}
if [program] == 'Epsy_log' {
mutate {
replace => [ 'type', 'Epsy_log' ]
}
}
if [program] == 'Wowzastream_access' {
mutate {
replace => [ 'type', 'wowzastream' ]
}
}
if [program] == 'Wowzastream_error' {
mutate {
replace => [ 'type', 'wowzastream' ]
}
}
}
#
# Local Configuration
#
jolson wrote:-Any customizations on the box you can think of? E.g. NRPE installed, non-default syslogger, etc.
I do have the nagiosXI client install on it and except for OS security patches nothing else I can think of. I did move the base
install to a larger disk but that was a while back so things have been working since then.
jolson wrote:Could you please post the results of these commands:
Code: Select all
[root@IGAnagioslog local]# cat /var/log/elasticsearch/*.log
[2015-04-29 03:57:35,898][INFO ][cluster.metadata ] [bb8f313e-98b6-4e1d-8ac4-19e6421ac511] [logstash-2015.04.29] update_mapping [apache_access] (dynamic)
[2015-04-29 05:14:40,084][INFO ][cluster.metadata ] [bb8f313e-98b6-4e1d-8ac4-19e6421ac511] [logstash-2015.04.29] update_mapping [eventlog] (dynamic)
[2015-04-29 05:14:51,559][INFO ][cluster.metadata ] [bb8f313e-98b6-4e1d-8ac4-19e6421ac511] [logstash-2015.04.29] update_mapping [eventlog] (dynamic)
[2015-04-29 05:14:53,163][INFO ][cluster.metadata ] [bb8f313e-98b6-4e1d-8ac4-19e6421ac511] [logstash-2015.04.29] update_mapping [eventlog] (dynamic)
[2015-04-29 05:15:58,783][INFO ][cluster.metadata ] [bb8f313e-98b6-4e1d-8ac4-19e6421ac511] [logstash-2015.04.29] update_mapping [eventlog] (dynamic)
[2015-04-29 05:17:32,378][INFO ][cluster.metadata ] [bb8f313e-98b6-4e1d-8ac4-19e6421ac511] [logstash-2015.04.29] update_mapping [eventlog] (dynamic)
[2015-04-29 08:20:42,535][INFO ][cluster.metadata ] [bb8f313e-98b6-4e1d-8ac4-19e6421ac511] [logstash-2015.04.29] update_mapping [eventlog] (dynamic)
[2015-04-29 08:20:43,490][INFO ][cluster.metadata ] [bb8f313e-98b6-4e1d-8ac4-19e6421ac511] [logstash-2015.04.29] update_mapping [eventlog] (dynamic)
Code: Select all
[root@IGAnagioslog local]# cat /var/log/httpd/error_log
[Sun Apr 26 03:34:03 2015] [notice] Digest: generating secret for digest authentication ...
[Sun Apr 26 03:34:03 2015] [notice] Digest: done
[Sun Apr 26 03:34:03 2015] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 configured -- resuming normal operations
jolson wrote:Code: Select all
ls -lh /store/backups/nagioslogserver/
Code: Select all
[root@IGAnagioslog local]# ls -lh /store/backups/nagioslogserver/
total 451M
drwxrwxrwx 2 nagios users 4.0K Apr 21 13:06 1429636007
drwxrwxrwx 2 nagios users 4.0K Apr 22 13:06 1429722407
drwxrwxrwx 2 nagios users 4.0K Apr 23 13:06 1429808811
drwxrwxrwx 2 nagios users 4.0K Apr 24 13:06 1429895212
drwxrwxrwx 2 nagios users 4.0K Apr 25 13:07 1429981621
drwxrwxrwx 2 nagios users 4.0K Apr 26 13:07 1430068022
drwxrwxrwx 2 nagios users 4.0K Apr 27 13:07 1430154427
drwxrwxrwx 2 nagios users 4.0K Apr 28 13:07 1430240831
-rw-r--r-- 1 nagios users 21M Mar 27 13:06 nagioslogserver.2015-03-27.1427475951.tar.gz
-rw-r--r-- 1 nagios users 5.3M Mar 28 13:05 nagioslogserver.2015-03-28.1427562351.tar.gz
-rw-r--r-- 1 nagios users 5.6M Mar 29 13:05 nagioslogserver.2015-03-29.1427648752.tar.gz
-rw-r--r-- 1 nagios users 21M Mar 30 13:06 nagioslogserver.2015-03-30.1427735162.tar.gz
-rw-r--r-- 1 nagios users 5.6M Mar 31 13:06 nagioslogserver.2015-03-31.1427821562.tar.gz
-rw-r--r-- 1 nagios users 21M Apr 1 13:06 nagioslogserver.2015-04-01.1427907966.tar.gz
-rw-r--r-- 1 nagios users 21M Apr 2 13:06 nagioslogserver.2015-04-02.1427994366.tar.gz
-rw-r--r-- 1 nagios users 21M Apr 3 13:06 nagioslogserver.2015-04-03.1428080766.tar.gz
-rw-r--r-- 1 nagios users 21M Apr 4 13:06 nagioslogserver.2015-04-04.1428167167.tar.gz
-rw-r--r-- 1 nagios users 21M Apr 5 13:06 nagioslogserver.2015-04-05.1428253572.tar.gz
-rw-r--r-- 1 nagios users 21M Apr 6 13:06 nagioslogserver.2015-04-06.1428339977.tar.gz
-rw-r--r-- 1 nagios users 21M Apr 7 13:06 nagioslogserver.2015-04-07.1428426381.tar.gz
-rw-r--r-- 1 nagios users 5.6M Apr 8 13:06 nagioslogserver.2015-04-08.1428512781.tar.gz
-rw-r--r-- 1 nagios users 21M Apr 9 13:06 nagioslogserver.2015-04-09.1428599182.tar.gz
-rw-r--r-- 1 nagios users 21M Apr 10 13:06 nagioslogserver.2015-04-10.1428685587.tar.gz
-rw-r--r-- 1 nagios users 21M Apr 11 13:06 nagioslogserver.2015-04-11.1428771991.tar.gz
-rw-r--r-- 1 nagios users 21M Apr 12 13:06 nagioslogserver.2015-04-12.1428858392.tar.gz
-rw-r--r-- 1 nagios users 21M Apr 13 13:07 nagioslogserver.2015-04-13.1428944796.tar.gz
-rw-r--r-- 1 nagios users 21M Apr 14 13:07 nagioslogserver.2015-04-14.1429031196.tar.gz
-rw-r--r-- 1 nagios users 21M Apr 15 13:07 nagioslogserver.2015-04-15.1429117596.tar.gz
-rw-r--r-- 1 nagios users 21M Apr 16 13:07 nagioslogserver.2015-04-16.1429203997.tar.gz
-rw-r--r-- 1 nagios users 21M Apr 17 13:07 nagioslogserver.2015-04-17.1429290397.tar.gz
-rw-r--r-- 1 nagios users 21M Apr 18 13:07 nagioslogserver.2015-04-18.1429376797.tar.gz
-rw-r--r-- 1 nagios users 21M Apr 19 13:07 nagioslogserver.2015-04-19.1429463201.tar.gz
-rw-r--r-- 1 nagios users 21M Apr 20 13:07 nagioslogserver.2015-04-20.1429549602.tar.gz
[root@IGAnagioslog local]# df -kh
Filesystem Size Used Avail Use% Mounted on
rootfs 99G 5.7G 92G 6% /
devtmpfs 2.0G 208K 2.0G 1% /dev
tmpfs 2.0G 0 2.0G 0% /dev/shm
/dev/sda1 99G 5.7G 92G 6% /
/dev/mapper/vg_nagioslog-lv_nagioslog_data
477G 256G 222G 54% /usr/local/nagioslogserver
/dev/mapper/vg_nagioslog-lv_nagioslog_rep
49G 33M 49G 1% /usr/local/nagioslogserver_bkuprep
jolson wrote:Thanks for your patience here Mitch, I'm hoping we can hunt whatever this is down and have it resolved soon.
No problem.
See-ya
Mitch