Waiting for Database Startup

[kwhogster]

All my servers pick up the time source from my ad servers which get the time source from the net.

Stop which firewall on the Cent OS 7 where Log Server is running?

[rkennedy]

But none of my devices are reporting to the log server

localhost my switch my router and a windows 2012 server with nxlog installed

Stop the firewall on your Nagios server. What all machines do you have trying to input logs to it? When you are on the Home page of NLS, how many sources does it tell you are sending logs to NLS?

[kwhogster]

[root@TGCS018 ~]# systemctl stop firewalld
Failed to stop firewalld.service: Unit firewalld.service not loaded.
[root@TGCS018 ~]# systemctl status firewalld
● firewalld.service
Loaded: not-found (Reason: No such file or directory)
Active: inactive (dead)
[root@TGCS018 ~]# service iptables stop
Redirecting to /bin/systemctl stop iptables.service
Failed to stop iptables.service: Unit iptables.service not loaded.
[root@TGCS018 ~]#

It seems that I do not have a firewall or iptables running on my Cent OS 7 which is the Log Server machine


Thoughts

[kwhogster]

What all machines do you have trying to input logs to it?


The localhost Cent OS 7
Meraki MX 60 Router using ip address and port 3515
Cisco Switch using ip address and port 3515
Windows 2012 R2 Server installed NXLOG service configured ip address and port 3515

All this was working until the elasticsearch was not starting

Should we restart nsl and elasticsearch or something ???

[rkennedy]

I noticed your 'Events over time' is blank, this might be related to your browser cache. Could you try opening NLS in a different browser, and see if it's displayed correctly then?

Another thing, it looks like you have your syslog from the Meraki and Cisco being sent in to the same port as 3515 -- are these devices sending their data in using JSON, or syslog data? If you're using the built in filters, you probably want to have them send to 5544.

[kwhogster]

tried edge same results this is not a browser issue I got same results using google chrome

My mistake on my cisco switch and meraki router I have the port as 5544 only on the windows box I using 3515


Thoughts

[rkennedy]

On the NLS machine, please run the following and let it run for a few minutes to collect data.

Code: Select all

tcpdump port 5544

Then, let's take a look at 3515 -

Code: Select all

tcpdump port 3515

Essentially, we're looking to make sure that NLS is actually receiving / seeing the data being sent by the clients. Please post the output from both of them back here.

[kwhogster]

[root@TGCS018 ~]# tcpdump port 5544
tcpdump: WARNING: SIOCGIFADDR: nflog: No such device
tcpdump: NFLOG link-layer type filtering not implemented
[root@TGCS018 ~]# tcpdump port 3515
tcpdump: WARNING: SIOCGIFADDR: nflog: No such device
tcpdump: NFLOG link-layer type filtering not implemented

[rkennedy]

It looks like your defaulting to an invalid interface. Try specifying an interface, and see if it works. What is the output of ifconfig? On my machine for example -

Code: Select all

[root@localhost ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:50:56:84:D4:01
          inet addr:192.168.3.190  Bcast:192.168.255.255  Mask:255.255.0.0
          inet6 addr: fe80::250:56ff:fe84:d401/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:23015952 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12572064 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3639083555 (3.3 GiB)  TX bytes:3066258997 (2.8 GiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:7981474 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7981474 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1421411658 (1.3 GiB)  TX bytes:1421411658 (1.3 GiB)

So I would run -

Code: Select all

tcpdump port 5544 -i eth0

[kwhogster]

Ok needed the name of the adapter got it take a look at the output

The first one was port 5544 this is strange that the computer tgkw005 is the only one this is a VM XP computer I run and I have not installed NXLOG service on this machine

The second one was port 3515 you see TGCS013 that is a Windows 2012 Server that I installed NXLOG service on again why is TGKW005 showing?
Also do not see my Meraki 10.2.8.1 or my switch 10.2.8.6

eno16777984: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.2.8.74 netmask 255.255.252.0 broadcast 10.2.11.255
inet6 fe80::20cfe8a:554e prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:8a:55:4e txqueuelen 1000 (Ethernet)
RX packets 5030262 bytes 6345009212 (5.9 GiB)
RX errors 0 dropped 183 overruns 0 frame 0
TX packets 3126909 bytes 480222302 (457.9 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 0 (Local Loopback)
RX packets 11149539 bytes 2555311829 (2.3 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 11149539 bytes 2555311829 (2.3 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

Code: Select all

[root@TGCS018 ~]# tcpdump port 5544 -i eno16777984
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eno16777984, link-type EN10MB (Ethernet), capture size 65535 bytes
16:57:05.605131 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 243
16:57:07.074202 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 152
16:57:22.002493 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 167
16:57:22.015254 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 165
16:57:45.241904 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 156
16:57:52.125338 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 220
16:57:52.281057 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 224
16:57:52.336398 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 235
16:57:52.401055 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 218
16:57:52.450886 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 203
16:57:52.498124 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 225
16:57:52.553333 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 221
16:57:52.600982 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 215
16:57:52.652796 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 208
16:57:52.700321 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 221
16:57:52.748229 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 219
16:57:52.790939 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 231
16:58:04.531920 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 151
16:58:48.784318 IP 10.2.8.6.personal-agent > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 154
16:58:55.029974 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 150
16:59:07.099059 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 151
16:59:14.914511 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 151
16:59:45.242038 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 158
16:59:46.920977 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 157
16:59:47.371226 IP 10.2.8.6.personal-agent > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 157
16:59:47.523571 IP 10.2.8.6.personal-agent > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 154
16:59:47.532615 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 153
16:59:47.990114 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 168
16:59:48.820703 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 154
16:59:51.466731 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 243
16:59:54.339935 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 169
17:00:13.811160 IP 10.2.8.6.personal-agent > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 138
17:00:13.811467 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 153
17:00:15.417925 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 185
17:00:17.005933 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 154
17:00:17.193594 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 162
17:00:18.017552 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 167
17:00:18.460456 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 165
17:00:27.515591 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 167
17:00:33.213538 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 162
17:00:42.160691 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 154
17:00:42.365449 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 162
17:00:43.201444 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 167
17:00:43.634108 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 165
17:00:44.953937 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 221
17:00:45.132115 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 221
17:00:45.176214 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 219
17:00:45.225217 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 219
17:00:55.364055 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 153
17:01:07.124181 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 151
17:01:07.124257 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 151
17:01:07.125929 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 151
17:01:07.174043 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 151
17:01:07.175380 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 151
17:01:10.335103 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 151
17:01:29.112309 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 149
17:01:30.540238 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 151
17:01:33.665841 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 162
17:01:54.175914 IP 10.2.8.1.49069 > tgkw005.myvm.our.network.tgcsnet.com.5544: UDP, length 165
^C
59 packets captured
59 packets received by filter
0 packets dropped by kernel
[root@TGCS018 ~]# tcpdump port 3515 -i eno16777984
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eno16777984, link-type EN10MB (Ethernet), capture size 65535 bytes
17:02:39.960208 IP tgcs013.our.network.tgcsnet.com.51940 > tgkw005.myvm.our.network.tgcsnet.com.must-backplane: Flags [SEW], seq 2496432202, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
17:02:39.960261 IP tgkw005.myvm.our.network.tgcsnet.com.must-backplane > tgcs013.our.network.tgcsnet.com.51940: Flags [R.], seq 0, ack 2496432203, win 0, length 0
17:02:40.475150 IP tgcs013.our.network.tgcsnet.com.51940 > tgkw005.myvm.our.network.tgcsnet.com.must-backplane: Flags [S], seq 2496432202, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
17:02:40.475216 IP tgkw005.myvm.our.network.tgcsnet.com.must-backplane > tgcs013.our.network.tgcsnet.com.51940: Flags [R.], seq 0, ack 1, win 0, length 0
17:02:40.975177 IP tgcs013.our.network.tgcsnet.com.51940 > tgkw005.myvm.our.network.tgcsnet.com.must-backplane: Flags [S], seq 2496432202, win 8192, options [mss 1460,nop,nop,sackOK], length 0
17:02:40.975252 IP tgkw005.myvm.our.network.tgcsnet.com.must-backplane > tgcs013.our.network.tgcsnet.com.51940: Flags [R.], seq 0, ack 1, win 0, length 0
17:04:00.894897 IP tgcs013.our.network.tgcsnet.com.51980 > tgkw005.myvm.our.network.tgcsnet.com.must-backplane: Flags [SEW], seq 3147142332, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
17:04:00.894947 IP tgkw005.myvm.our.network.tgcsnet.com.must-backplane > tgcs013.our.network.tgcsnet.com.51980: Flags [R.], seq 0, ack 3147142333, win 0, length 0
17:04:01.393993 IP tgcs013.our.network.tgcsnet.com.51980 > tgkw005.myvm.our.network.tgcsnet.com.must-backplane: Flags [S], seq 3147142332, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
17:04:01.394052 IP tgkw005.myvm.our.network.tgcsnet.com.must-backplane > tgcs013.our.network.tgcsnet.com.51980: Flags [R.], seq 0, ack 1, win 0, length 0
17:04:01.894022 IP tgcs013.our.network.tgcsnet.com.51980 > tgkw005.myvm.our.network.tgcsnet.com.must-backplane: Flags [S], seq 3147142332, win 8192, options [mss 1460,nop,nop,sackOK], length 0
17:04:01.894086 IP tgkw005.myvm.our.network.tgcsnet.com.must-backplane > tgcs013.our.network.tgcsnet.com.51980: Flags [R.], seq 0, ack 1, win 0, length 0
^C
12 packets captured
12 packets received by filter
0 packets dropped by kernel
[root@TGCS018 ~]#

Thoughts.

Tom