Waiting for Database Startup

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Locked
kwhogster
Posts: 644
Joined: Wed Oct 14, 2015 6:51 pm
Location: Wood Ridge NJ USA
Contact:
Contact kwhogster

Waiting for Database Startup

Post by kwhogster »

CentOS 7
Nagios LogServer

Installed a few days ago and all was good but today when I try to browse the site I get this page



Waiting for Database Startup

It looks like your local elasticsearch service is starting.

Why am I getting this error?

Elasticsearch can take a little while to start up because of it's indexing. This may take a few seconds.

The page will refresh automatically after 5 seconds...


I tried stopping and starting the elasticsearch service but no luck

This is what I see

[root@TGCS018 ~]# sudo service elasticsearch status
● elasticsearch.service - LSB: This service manages the elasticsearch daemon
Loaded: loaded (/etc/rc.d/init.d/elasticsearch)
Active: failed (Result: exit-code) since Sat 2016-08-13 10:24:01 EDT; 17min ago
Docs: man:systemd-sysv-generator(8)
Process: 23979 ExecStop=/etc/rc.d/init.d/elasticsearch stop (code=exited, status=1/FAILURE)
Process: 23506 ExecStart=/etc/rc.d/init.d/elasticsearch start (code=exited, status=0/SUCCESS)

Aug 13 10:22:16 TGCS018.our.network.tgcsnet.com runuser[23517]: pam_unix(runu...
Aug 13 10:22:16 TGCS018.our.network.tgcsnet.com runuser[23517]: pam_unix(runu...
Aug 13 10:22:16 TGCS018.our.network.tgcsnet.com elasticsearch[23506]: Startin...
Aug 13 10:22:16 TGCS018.our.network.tgcsnet.com systemd[1]: Started LSB: This...
Aug 13 10:23:40 TGCS018.our.network.tgcsnet.com systemd[1]: Stopping LSB: Thi...
Aug 13 10:24:01 TGCS018.our.network.tgcsnet.com elasticsearch[23979]: Stoppin...
Aug 13 10:24:01 TGCS018.our.network.tgcsnet.com systemd[1]: elasticsearch.ser...
Aug 13 10:24:01 TGCS018.our.network.tgcsnet.com systemd[1]: Stopped LSB: This...
Aug 13 10:24:01 TGCS018.our.network.tgcsnet.com systemd[1]: Unit elasticsearc...
Aug 13 10:24:01 TGCS018.our.network.tgcsnet.com systemd[1]: elasticsearch.ser...
Hint: Some lines were ellipsized, use -l to show in full.

Any ideas?
Top
kwhogster
Posts: 644
Joined: Wed Oct 14, 2015 6:51 pm
Location: Wood Ridge NJ USA
Contact:
Contact kwhogster

Re: Waiting for Database Startup

Post by kwhogster »

Update

I am now pas the error screen and back on the dashboard.

But none of my devices are reporting to the log server

localhost my switch my router and a windows 2012 server with nxlog installed

What am I missing
Top
rkennedy
Posts: 6579
Joined: Mon Oct 05, 2015 11:45 am

Re: Waiting for Database Startup

Post by rkennedy »

What is the output of getenforce? Can you post your logstash.log file for us to look at? It's located at /var/log/logstash/logstash.log.
Former Nagios Employee
Top
kwhogster
Posts: 644
Joined: Wed Oct 14, 2015 6:51 pm
Location: Wood Ridge NJ USA
Contact:
Contact kwhogster

Re: Waiting for Database Startup

Post by kwhogster »

[root@TGCS018 ~]# getenforce
Permissive

for the log file it is empty

[root@TGCS018 ~]# ls -ltr /var/log/logstash/
total 9920
-rw-r--r--. 1 nagios users 278 Aug 9 22:06 logstash.log-20160811.gz
-rw-r--r--. 1 nagios users 4543517 Aug 13 03:07 logstash.log-20160813.gz
-rw-r--r--. 1 nagios users 5607280 Aug 13 11:39 logstash.log-20160814.gz
-rw-r--r--. 1 nagios users 0 Aug 14 03:08 logstash.log

Thoughts?

Thanks Tom
Top
rkennedy
Posts: 6579
Joined: Mon Oct 05, 2015 11:45 am

Re: Waiting for Database Startup

Post by rkennedy »

Hmm, how about your /var/log/elasticsearch/*.log - anything being created by the elasticsearch log file you could post?
Former Nagios Employee
Top
kwhogster
Posts: 644
Joined: Wed Oct 14, 2015 6:51 pm
Location: Wood Ridge NJ USA
Contact:
Contact kwhogster

Re: Waiting for Database Startup

Post by kwhogster »

[root@TGCS018 ~]# ls -ltr /var/log/elasticsearch
total 28
-rw-r--r--. 1 nagios users 0 Aug 9 22:05 b46627e6-fb30-4cf0-b7ac-83e9d8af748 1_index_indexing_slowlog.log
-rw-r--r--. 1 nagios users 0 Aug 9 22:05 b46627e6-fb30-4cf0-b7ac-83e9d8af748 1_index_search_slowlog.log
-rw-r--r--. 1 nagios users 2694 Aug 10 22:07 b46627e6-fb30-4cf0-b7ac-83e9d8af748 1.log-20160811.gz
-rw-r--r--. 1 nagios users 1771 Aug 12 01:56 b46627e6-fb30-4cf0-b7ac-83e9d8af748 1.log-20160812.gz
-rw-r--r--. 1 nagios users 455 Aug 12 20:10 b46627e6-fb30-4cf0-b7ac-83e9d8af748 1.log-20160813.gz
-rw-r--r--. 1 nagios users 3768 Aug 13 11:38 b46627e6-fb30-4cf0-b7ac-83e9d8af748 1.log-20160814.gz
-rw-r--r--. 1 nagios users 951 Aug 14 11:39 b46627e6-fb30-4cf0-b7ac-83e9d8af748 1.log-20160815.gz
-rw-r--r--. 1 nagios users 7557 Aug 15 11:38 b46627e6-fb30-4cf0-b7ac-83e9d8af748 1.log
[root@TGCS018 ~]#


those are empty also

Thoughts????
Top
rkennedy
Posts: 6579
Joined: Mon Oct 05, 2015 11:45 am

Re: Waiting for Database Startup

Post by rkennedy »

This one isn't actually, could you post it? b46627e6-fb30-4cf0-b7ac-83e9d8af748 1.log

Also, please post the full output of ps -ef.
Former Nagios Employee
Top
kwhogster
Posts: 644
Joined: Wed Oct 14, 2015 6:51 pm
Location: Wood Ridge NJ USA
Contact:
Contact kwhogster

Re: Waiting for Database Startup

Post by kwhogster »

Code: Select all

[root@TGCS018 ~]# ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Aug09 ?        00:12:15 /usr/lib/systemd/systemd --syste
root         2     0  0 Aug09 ?        00:00:00 [kthreadd]
root         3     2  0 Aug09 ?        00:00:04 [ksoftirqd/0]
root         5     2  0 Aug09 ?        00:00:00 [kworker/0:0H]
root         7     2  0 Aug09 ?        00:00:23 [migration/0]
root         8     2  0 Aug09 ?        00:00:00 [rcu_bh]
root         9     2  0 Aug09 ?        00:00:00 [rcuob/0]
root        10     2  0 Aug09 ?        00:00:00 [rcuob/1]
root        11     2  0 Aug09 ?        00:00:00 [rcuob/2]
root        12     2  0 Aug09 ?        00:00:00 [rcuob/3]
root        13     2  0 Aug09 ?        00:06:16 [rcu_sched]
root        14     2  0 Aug09 ?        00:02:47 [rcuos/0]
root        15     2  0 Aug09 ?        00:03:09 [rcuos/1]
root        16     2  0 Aug09 ?        00:02:52 [rcuos/2]
root        17     2  0 Aug09 ?        00:02:51 [rcuos/3]
root        18     2  0 Aug09 ?        00:00:16 [watchdog/0]
root        19     2  0 Aug09 ?        00:00:41 [watchdog/1]
root        20     2  0 Aug09 ?        00:00:24 [migration/1]
root        21     2  0 Aug09 ?        00:00:18 [ksoftirqd/1]
root        23     2  0 Aug09 ?        00:00:00 [kworker/1:0H]
root        24     2  0 Aug09 ?        00:00:05 [watchdog/2]
root        25     2  0 Aug09 ?        00:00:16 [migration/2]
root        26     2  0 Aug09 ?        00:00:07 [ksoftirqd/2]
root        29     2  0 Aug09 ?        00:00:02 [watchdog/3]
root        30     2  0 Aug09 ?        00:00:15 [migration/3]
root        31     2  0 Aug09 ?        00:00:02 [ksoftirqd/3]
root        33     2  0 Aug09 ?        00:00:00 [kworker/3:0H]
root        34     2  0 Aug09 ?        00:00:00 [khelper]
root        35     2  0 Aug09 ?        00:00:00 [kdevtmpfs]
root        36     2  0 Aug09 ?        00:00:00 [netns]
root        37     2  0 Aug09 ?        00:00:00 [perf]
root        38     2  0 Aug09 ?        00:00:00 [writeback]
root        39     2  0 Aug09 ?        00:00:00 [kintegrityd]
root        40     2  0 Aug09 ?        00:00:00 [bioset]
root        41     2  0 Aug09 ?        00:00:00 [kblockd]
root        42     2  0 Aug09 ?        00:00:00 [md]
root        50     2  0 Aug09 ?        00:00:00 [khungtaskd]
root        51     2  0 Aug09 ?        00:00:25 [kswapd0]
root        52     2  0 Aug09 ?        00:00:00 [ksmd]
root        53     2  0 Aug09 ?        00:00:35 [khugepaged]
root        54     2  0 Aug09 ?        00:00:00 [fsnotify_mark]
root        55     2  0 Aug09 ?        00:00:00 [crypto]
root        63     2  0 Aug09 ?        00:00:00 [kthrotld]
root        65     2  0 Aug09 ?        00:00:00 [kmpath_rdacd]
root        66     2  0 Aug09 ?        00:00:00 [kpsmoused]
root        68     2  0 Aug09 ?        00:00:00 [ipv6_addrconf]
root        87     2  0 Aug09 ?        00:00:00 [deferwq]
root       117     2  0 Aug09 ?        00:00:30 [kauditd]
root       299     2  0 Aug09 ?        00:00:00 [ata_sff]
root       300     2  0 Aug09 ?        00:00:00 [scsi_eh_0]
root       301     2  0 Aug09 ?        00:00:00 [events_power_ef]
root       302     2  0 Aug09 ?        00:00:00 [scsi_tmf_0]
root       303     2  0 Aug09 ?        00:00:00 [scsi_eh_1]
root       304     2  0 Aug09 ?        00:00:00 [scsi_tmf_1]
root       308     2  0 Aug09 ?        00:00:00 [mpt_poll_0]
root       309     2  0 Aug09 ?        00:00:00 [mpt/0]
root       316     2  0 Aug09 ?        00:00:00 [scsi_eh_2]
root       317     2  0 Aug09 ?        00:00:00 [scsi_tmf_2]
root       318     2  0 Aug09 ?        00:00:00 [ttm_swap]
root       330     2  0 Aug09 ?        00:00:00 [kworker/0:1H]
root       392     2  0 Aug09 ?        00:00:00 [kdmflush]
root       393     2  0 Aug09 ?        00:00:00 [bioset]
root       404     2  0 Aug09 ?        00:00:00 [kdmflush]
root       405     2  0 Aug09 ?        00:00:00 [bioset]
root       418     2  0 Aug09 ?        00:00:00 [xfsalloc]
root       419     2  0 Aug09 ?        00:00:00 [xfs_mru_cache]
root       420     2  0 Aug09 ?        00:00:00 [xfs-buf/dm-0]
root       421     2  0 Aug09 ?        00:00:00 [xfs-data/dm-0]
root       422     2  0 Aug09 ?        00:00:00 [xfs-conv/dm-0]
root       423     2  0 Aug09 ?        00:00:00 [xfs-cil/dm-0]
root       424     2  0 Aug09 ?        00:02:57 [xfsaild/dm-0]
root       494     1  0 Aug09 ?        00:01:33 /usr/lib/systemd/systemd-journal
root       515     1  0 Aug09 ?        00:00:00 /usr/sbin/lvmetad -f
root       524     1  0 Aug09 ?        00:00:01 /usr/lib/systemd/systemd-udevd
root       581     2  0 Aug09 ?        00:00:00 [xfs-buf/sda1]
root       582     2  0 Aug09 ?        00:00:00 [xfs-data/sda1]
root       583     2  0 Aug09 ?        00:00:00 [xfs-conv/sda1]
root       584     2  0 Aug09 ?        00:00:00 [xfs-cil/sda1]
root       585     2  0 Aug09 ?        00:00:00 [xfsaild/sda1]
root       586     2  0 Aug09 ?        00:00:00 [kdmflush]
root       587     2  0 Aug09 ?        00:00:00 [bioset]
root       594     2  0 Aug09 ?        00:00:00 [xfs-buf/dm-2]
root       595     2  0 Aug09 ?        00:00:00 [xfs-data/dm-2]
root       596     2  0 Aug09 ?        00:00:00 [xfs-conv/dm-2]
root       597     2  0 Aug09 ?        00:00:00 [xfs-cil/dm-2]
root       598     2  0 Aug09 ?        00:00:00 [xfsaild/dm-2]
root       617     1  0 Aug09 ?        00:01:08 /sbin/auditd -n
root       648     1  0 Aug09 ?        00:01:57 /usr/sbin/irqbalance --foregroun
root       649     1  0 Aug09 ?        00:00:53 /usr/sbin/NetworkManager --no-da
dbus       650     1  0 Aug09 ?        00:02:14 /bin/dbus-daemon --system --addr
root       665     1  0 Aug09 ?        00:00:00 login -- root
root       691     1  0 Aug09 ?        00:00:00 /usr/sbin/wpa_supplicant -u -f /
polkitd    692     1  0 Aug09 ?        00:00:38 /usr/lib/polkit-1/polkitd --no-d
root       850     1  0 Aug09 ?        00:00:00 /usr/sbin/sshd -D
root       851     1  0 Aug09 ?        00:01:08 /usr/bin/python -Es /usr/sbin/tu
root       853     2  0 17:53 ?        00:00:00 [kworker/u8:2]
root      1886     1  0 Aug09 ?        00:00:04 /usr/libexec/postfix/master -w
postfix   1945  1886  0 Aug09 ?        00:00:00 qmgr -l -t unix -u
nagios    2777     1  1 Aug13 ?        00:49:33 java -Xms4111m -Xmx4111m -Djava.
root      2834     2  0 Aug09 ?        00:00:00 [kworker/3:1H]
postfix   4080  1886  0 18:12 ?        00:00:00 pickup -l -t unix -u
apache    4570 18924  0 08:21 ?        00:00:07 /usr/sbin/httpd -DFOREGROUND
apache    4724 18924  0 08:21 ?        00:00:07 /usr/sbin/httpd -DFOREGROUND
apache    4726 18924  0 08:22 ?        00:00:07 /usr/sbin/httpd -DFOREGROUND
root      5087     2  0 00:17 ?        00:00:04 [kworker/0:2]
apache    6451 18924  0 Aug14 ?        00:00:08 /usr/sbin/httpd -DFOREGROUND
apache    6545 18924  0 Aug14 ?        00:00:08 /usr/sbin/httpd -DFOREGROUND
apache    6704 18924  0 Aug14 ?        00:00:09 /usr/sbin/httpd -DFOREGROUND
apache    8343 18924  0 Aug14 ?        00:00:08 /usr/sbin/httpd -DFOREGROUND
root      8504     1  0 Aug12 ?        00:00:31 /usr/lib/systemd/systemd-logind
root      9929   665  0 Aug09 tty1     00:00:00 -bash
root     12497     2  0 12:27 ?        00:00:00 [kworker/0:0]
root     17062     2  0 19:32 ?        00:00:00 [kworker/2:2]
root     17193     2  0 19:33 ?        00:00:00 [kworker/1:1]
root     17449     2  0 09:40 ?        00:00:20 [kworker/3:1]
root     17704     1  0 Aug09 ?        00:00:47 /usr/sbin/rsyslogd -n
root     17714     2  0 19:36 ?        00:00:00 [kworker/3:2]
root     17774     1  0 Aug09 ?        00:00:09 /usr/sbin/crond -n
ntp      18315     1  0 Aug09 ?        00:00:03 /usr/sbin/ntpd -u ntp:ntp -g
root     18491     2  0 19:41 ?        00:00:00 [kworker/1:2]
root     18709     2  0 Aug09 ?        00:00:11 [kworker/2:1H]
root     18924     1  0 Aug09 ?        00:00:45 /usr/sbin/httpd -DFOREGROUND
root     18979 17774  0 19:44 ?        00:00:00 /usr/sbin/CROND -n
root     18980 17774  0 19:44 ?        00:00:00 /usr/sbin/CROND -n
nagios   18983 18980  0 19:44 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/
nagios   18984 18979  0 19:44 ?        00:00:00 /bin/sh -c /usr/bin/php -q /var/
nagios   18985 18984  0 19:44 ?        00:00:00 /usr/bin/php -q /var/www/html/na
nagios   18986 18983  0 19:44 ?        00:00:00 /usr/bin/php -q /var/www/html/na
root     18987     2  0 19:44 ?        00:00:00 [kworker/2:0]
root     19080   850  0 19:44 ?        00:00:00 sshd: root@pts/1
root     19084 19080  0 19:44 pts/1    00:00:00 -bash
root     19129 19084  0 19:44 pts/1    00:00:00 ps -ef
root     19762     1  0 Aug13 ?        00:00:02 /usr/sbin/xinetd -stayalive -pid
root     20011   649  0 Aug09 ?        00:00:00 /sbin/dhclient -d -q -sf /usr/li
root     20106     2  0 Aug11 ?        00:00:02 [kworker/1:2H]
root     20600     2  0 02:14 ?        00:00:04 [kworker/1:0]
root     22654     2  0 06:34 ?        00:00:02 [kworker/2:1]
apache   23218 18924  0 Aug14 ?        00:00:09 /usr/sbin/httpd -DFOREGROUND
apache   23219 18924  0 Aug14 ?        00:00:09 /usr/sbin/httpd -DFOREGROUND
root     23849   850  0 13:37 ?        00:00:00 sshd: root@pts/0
root     23854 23849  0 13:37 pts/0    00:00:00 -bash
root     27041     2  0 Aug13 ?        00:00:00 [kworker/2:2H]
root     28823     2  0 14:07 ?        00:00:00 [kworker/0:3]
apache   28882 18924  0 07:21 ?        00:00:07 /usr/sbin/httpd -DFOREGROUND
root     29095     2  0 10:51 ?        00:00:00 [kworker/u8:0]

Code: Select all

[root@TGCS018 ~]# cat /var/log/elasticsearch/b46627e6-fb30-4cf0-b7ac-83e9d8af7481.log
[2016-08-15 11:38:02,836][INFO ][KnapsackService          ] add: plugin.knapsack.export.state -> []
[2016-08-15 11:38:02,836][INFO ][KnapsackService          ] update cluster settings: plugin.knapsack.export.state -> [{"mode":"export","started":"2016-08-15T15:38:02.836Z","path":"file:///store/backups/nagioslogserver/1471275482/nagioslogserver.tar.gz","node_name":"ceb4a326-c1f0-4e7a-b8e4-1796edfbb25c"}]
[2016-08-15 11:38:02,836][INFO ][KnapsackExportAction     ] start of export: {"mode":"export","started":"2016-08-15T15:38:02.836Z","path":"file:///store/backups/nagioslogserver/1471275482/nagioslogserver.tar.gz","node_name":"ceb4a326-c1f0-4e7a-b8e4-1796edfbb25c"}
[2016-08-15 11:38:02,836][INFO ][KnapsackExportAction     ] getting settings for indices [nagioslogserver]
[2016-08-15 11:38:02,837][INFO ][KnapsackExportAction     ] found indices: [nagioslogserver]
[2016-08-15 11:38:02,838][INFO ][KnapsackExportAction     ] getting mappings for index nagioslogserver and types []
[2016-08-15 11:38:02,839][INFO ][KnapsackExportAction     ] found mappings: [cf_option, node, reactor_server, query, _default_, snapshot, alert, email_template, commands, snmp_reactor, nrdp_server, user]
[2016-08-15 11:38:02,840][INFO ][KnapsackExportAction     ] adding mapping: cf_option
[2016-08-15 11:38:02,840][INFO ][KnapsackExportAction     ] adding mapping: node
[2016-08-15 11:38:02,840][INFO ][KnapsackExportAction     ] adding mapping: reactor_server
[2016-08-15 11:38:02,840][INFO ][KnapsackExportAction     ] adding mapping: query
[2016-08-15 11:38:02,840][INFO ][KnapsackExportAction     ] adding mapping: _default_
[2016-08-15 11:38:02,840][INFO ][KnapsackExportAction     ] adding mapping: snapshot
[2016-08-15 11:38:02,840][INFO ][KnapsackExportAction     ] adding mapping: alert
[2016-08-15 11:38:02,840][INFO ][KnapsackExportAction     ] adding mapping: email_template
[2016-08-15 11:38:02,840][INFO ][KnapsackExportAction     ] adding mapping: commands
[2016-08-15 11:38:02,841][INFO ][KnapsackExportAction     ] adding mapping: snmp_reactor
[2016-08-15 11:38:02,841][INFO ][KnapsackExportAction     ] adding mapping: nrdp_server
[2016-08-15 11:38:02,841][INFO ][KnapsackExportAction     ] adding mapping: user
[2016-08-15 11:38:02,841][INFO ][KnapsackExportAction     ] getting aliases for index nagioslogserver
[2016-08-15 11:38:02,841][INFO ][KnapsackExportAction     ] found 0 aliases
[2016-08-15 11:38:02,846][INFO ][KnapsackExportAction     ] end of export: {"mode":"export","started":"2016-08-15T15:38:02.836Z","path":"file:///store/backups/nagioslogserver/1471275482/nagioslogserver.tar.gz","node_name":"ceb4a326-c1f0-4e7a-b8e4-1796edfbb25c"}, packets = 35, total bytes transferred = 7235, rate = 1447.000000
[2016-08-15 11:38:02,847][INFO ][KnapsackService          ] remove: plugin.knapsack.export.state -> [{"mode":"export","started":"2016-08-15T15:38:02.836Z","path":"file:///store/backups/nagioslogserver/1471275482/nagioslogserver.tar.gz","node_name":"ceb4a326-c1f0-4e7a-b8e4-1796edfbb25c"}]
[2016-08-15 11:38:02,847][INFO ][KnapsackService          ] update cluster settings: plugin.knapsack.export.state -> []
[2016-08-15 11:38:03,991][INFO ][KnapsackService          ] add: plugin.knapsack.export.state -> []
[2016-08-15 11:38:03,992][INFO ][KnapsackExportAction     ] start of export: {"mode":"export","started":"2016-08-15T15:38:03.991Z","path":"file:///store/backups/nagioslogserver/1471275482/kibana-int.tar.gz","node_name":"ceb4a326-c1f0-4e7a-b8e4-1796edfbb25c"}
[2016-08-15 11:38:03,992][INFO ][KnapsackExportAction     ] getting settings for indices [kibana-int]
[2016-08-15 11:38:03,992][INFO ][KnapsackService          ] update cluster settings: plugin.knapsack.export.state -> [{"mode":"export","started":"2016-08-15T15:38:03.991Z","path":"file:///store/backups/nagioslogserver/1471275482/kibana-int.tar.gz","node_name":"ceb4a326-c1f0-4e7a-b8e4-1796edfbb25c"}]
[2016-08-15 11:38:03,992][INFO ][KnapsackExportAction     ] found indices: [kibana-int]
[2016-08-15 11:38:03,993][INFO ][KnapsackExportAction     ] getting mappings for index kibana-int and types []
[2016-08-15 11:38:03,993][INFO ][KnapsackExportAction     ] found mappings: [dashboard]
[2016-08-15 11:38:03,993][INFO ][KnapsackExportAction     ] adding mapping: dashboard
[2016-08-15 11:38:03,993][INFO ][KnapsackExportAction     ] getting aliases for index kibana-int
[2016-08-15 11:38:03,994][INFO ][KnapsackExportAction     ] found 0 aliases
[2016-08-15 11:38:03,996][INFO ][KnapsackExportAction     ] end of export: {"mode":"export","started":"2016-08-15T15:38:03.991Z","path":"file:///store/backups/nagioslogserver/1471275482/kibana-int.tar.gz","node_name":"ceb4a326-c1f0-4e7a-b8e4-1796edfbb25c"}, packets = 7, total bytes transferred = 2623, rate = 524.600000
[2016-08-15 11:38:03,997][INFO ][KnapsackService          ] remove: plugin.knapsack.export.state -> [{"mode":"export","started":"2016-08-15T15:38:03.991Z","path":"file:///store/backups/nagioslogserver/1471275482/kibana-int.tar.gz","node_name":"ceb4a326-c1f0-4e7a-b8e4-1796edfbb25c"}]
[2016-08-15 11:38:03,997][INFO ][KnapsackService          ] update cluster settings: plugin.knapsack.export.state -> []
[2016-08-15 11:38:05,177][INFO ][KnapsackService          ] add: plugin.knapsack.export.state -> []
[2016-08-15 11:38:05,177][INFO ][KnapsackService          ] update cluster settings: plugin.knapsack.export.state -> [{"mode":"export","started":"2016-08-15T15:38:05.177Z","path":"file:///store/backups/nagioslogserver/1471275482/nagioslogserver_log.tar.gz","node_name":"ceb4a326-c1f0-4e7a-b8e4-1796edfbb25c"}]
[2016-08-15 11:38:05,178][INFO ][KnapsackExportAction     ] start of export: {"mode":"export","started":"2016-08-15T15:38:05.177Z","path":"file:///store/backups/nagioslogserver/1471275482/nagioslogserver_log.tar.gz","node_name":"ceb4a326-c1f0-4e7a-b8e4-1796edfbb25c"}
[2016-08-15 11:38:05,178][INFO ][KnapsackExportAction     ] getting settings for indices [nagioslogserver_log]
[2016-08-15 11:38:05,178][INFO ][KnapsackExportAction     ] found indices: [nagioslogserver_log]
[2016-08-15 11:38:05,182][INFO ][KnapsackExportAction     ] getting mappings for index nagioslogserver_log and types []
[2016-08-15 11:38:05,183][INFO ][KnapsackExportAction     ] found mappings: [JOBS, MAINTENANCE, POLLER, SECURITY]
[2016-08-15 11:38:05,183][INFO ][KnapsackExportAction     ] adding mapping: JOBS
[2016-08-15 11:38:05,183][INFO ][KnapsackExportAction     ] adding mapping: MAINTENANCE
[2016-08-15 11:38:05,183][INFO ][KnapsackExportAction     ] adding mapping: POLLER
[2016-08-15 11:38:05,183][INFO ][KnapsackExportAction     ] adding mapping: SECURITY
[2016-08-15 11:38:05,183][INFO ][KnapsackExportAction     ] getting aliases for index nagioslogserver_log
[2016-08-15 11:38:05,183][INFO ][KnapsackExportAction     ] found 0 aliases
[2016-08-15 11:38:06,060][INFO ][KnapsackExportAction     ] end of export: {"mode":"export","started":"2016-08-15T15:38:05.177Z","path":"file:///store/backups/nagioslogserver/1471275482/nagioslogserver_log.tar.gz","node_name":"ceb4a326-c1f0-4e7a-b8e4-1796edfbb25c"}, packets = 20824, total bytes transferred = 3664879, rate = 732975.800000
[2016-08-15 11:38:06,061][INFO ][KnapsackService          ] remove: plugin.knapsack.export.state -> [{"mode":"export","started":"2016-08-15T15:38:05.177Z","path":"file:///store/backups/nagioslogserver/1471275482/nagioslogserver_log.tar.gz","node_name":"ceb4a326-c1f0-4e7a-b8e4-1796edfbb25c"}]
[2016-08-15 11:38:06,061][INFO ][KnapsackService          ] update cluster settings: plugin.knapsack.export.state -> []

Does that help?

I also ran this

yum install tcpdump

Code: Select all

[root@TGCS018 ~]# tcpdump port 3515
tcpdump: WARNING: SIOCGIFADDR: nflog: No such device
tcpdump: NFLOG link-layer type filtering not implemented
[root@TGCS018 ~]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:must-backplane
ACCEPT     tcp  --  anywhere             anywhere             state NEW tcp dpt:nsca
ACCEPT     tcp  --  anywhere             anywhere             state NEW tcp dpt:nrpe
ACCEPT     tcp  --  nagios.our.network.tgcsnet.com  anywhere             tcp dpt:nrpe state NEW,ESTABLISHED

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
Last edited by tmcdonald on Wed Aug 17, 2016 10:54 am, edited 1 time in total.
Reason: Please use [code][/code] tags around long output
Top
kwhogster
Posts: 644
Joined: Wed Oct 14, 2015 6:51 pm
Location: Wood Ridge NJ USA
Contact:
Contact kwhogster

Re: Waiting for Database Startup

Post by kwhogster »

I just noticed this message that appears on my dashboard

No Results There were no results because no indices were found to match your selected time span

It only appears for a few seconds

Thoughts?
Top
rkennedy
Posts: 6579
Joined: Mon Oct 05, 2015 11:45 am

Re: Waiting for Database Startup

Post by rkennedy »

Just to make sure we're on the same page, at this point we're troubleshooting why the Windows logs are not making it in, correct? It sounds like Elasticsearch is starting properly now.

Does the Windows machine have the proper system date / time set? To test, can you disable your firewall to see if logs come in at that point?
Former Nagios Employee
Top
Locked

Return to “Nagios Log Server”