Empty dashboard but logstash is collecting logs

[uma K]

I have made the required modifications but still logs stopped collecting.

Elasticsearch:
[2018-11-16 10:49:32,909][WARN ][transport.netty ] [abd0aca5-8cbf-4f11-988e-be0d778f5f95] exception caught on transport layer [[id: 0xdebca5fb]], closing connection
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:748)
^C

[scottwilkerson]

Can you post the output of this command

Code: Select all

cat /etc/hosts

It seems like your system may be missing the localhost entry

[uma K]

Hi I have attached etc/hosts from all 4 servers.

Unable to restart node 3 and 4 still. My cluster says only 2 nodes exists in the dashboard.

[scottwilkerson]

Can you look at the following file on all 4 hosts and make sure it contains the IP's of all the other hosts

Code: Select all

/usr/local/nagioslogserver/var/cluster_hosts

If not, add them, then restart elasticsearch

[uma K]

Yea, All the 4 servers have IP's of all the 4 servers in the specified path

/usr/local/nagioslogserver/var/cluster_hosts

[uma K]

Logstash is not collecting any logs. after pipeline is started I dont have any log entry as well.
and server 3 and 4 is unable to start as well.

[root@X1LOGW01 logstash]# tail -f logstash.log
{:timestamp=>"2018-11-16T10:48:18.154000-0800", :message=>"SIGTERM received. Shutting down the agent.", :level=>:warn}
{:timestamp=>"2018-11-16T10:48:18.157000-0800", :message=>"stopping pipeline", :id=>"main"}
{:timestamp=>"2018-11-16T10:48:47.359000-0800", :message=>"Pipeline main started"}
{:timestamp=>"2018-11-16T15:30:14.048000-0800", :message=>"SIGTERM received. Shutting down the agent.", :level=>:warn}
{:timestamp=>"2018-11-16T15:30:14.049000-0800", :message=>"stopping pipeline", :id=>"main"}
{:timestamp=>"2018-11-16T15:30:56.494000-0800", :message=>"Pipeline main started"}

[scottwilkerson]

We may need to setup a remote session to take a look at the cluster, can you open a ticket here and reference this forum thread
https://support.nagios.com/tickets/

Thanks

[uma K]

Can you please assist on this. Currently my production nagios log server is down and logstash is not collecting logs.

[scottwilkerson]

We may need to setup a remote session to take a look at the cluster, can you open a ticket here and reference this forum thread
https://support.nagios.com/tickets/

Thanks

[uma K]

Hello, I have opened a support ticket

268619