Nagios Support Forum

Support for Nagios products and services
https://support.nagios.com/forum/

Problem whit logstash

https://support.nagios.com/forum/viewtopic.php?t=48588

Page 1 of 1

Problem whit logstash

Posted: Tue May 15, 2018 4:37 am
by sbarrera
Hi i´m having a problem whit my nagios log server, i can only login it whit the logstash service inactive, when i started it pop me like waiting for elasticsearch to startup and the elasticsearch.service turn into active(excited) mode.

Ask me for all the information that i can provide.

Re: Problem whit logstash

Posted: Tue May 15, 2018 8:33 am
by scottwilkerson
You should only be able to login if elasticsearch is running. Elasticsearch is the datastore and also holds all your user information.

Logstash actually doesn't interact with user login whatsoever.

Re: Problem whit logstash

Posted: Wed May 16, 2018 2:14 am
by sbarrera
Captura1.PNG
So do you know why when i turn on the logstah.service the elasticsearch.service turn into active(exited) mode and the server pop me Waiting for Elasticsearch.

What could be the problem?

Code: Select all

 service logstash status
Logstash Daemon● logstash.service - LSB: Logstash
   Loaded: loaded (/etc/rc.d/init.d/logstash; bad; vendor preset: disabled)
   Active: active (running) since Wed 2018-05-16 09:10:07 CEST; 9s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 13707 ExecStop=/etc/rc.d/init.d/logstash stop (code=exited, status=0/SUCCESS)
  Process: 6844 ExecStart=/etc/rc.d/init.d/logstash start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/logstash.service
           ├─6854 runuser -s /bin/sh -c exec /usr/local/nagioslogserver/logstash/bin/logstash agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/logstash/logstash.log  -w 4...
           └─6856 java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryE...

May 16 09:10:06 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Starting LSB: Logstash...
May 16 09:10:06 ip-172-31-1-24.eu-west-1.compute.internal runuser[6854]: pam_unix(runuser:session): session opened for user nagios by (uid=0)
May 16 09:10:07 ip-172-31-1-24.eu-west-1.compute.internal logstash[6844]: Starting Logstash Daemon: [  OK  ]
May 16 09:10:07 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Started LSB: Logstash.

Code: Select all

service elasticsearch status
● elasticsearch.service - LSB: This service manages the elasticsearch daemon
   Loaded: loaded (/etc/rc.d/init.d/elasticsearch; bad; vendor preset: disabled)
   Active: active (exited) since Tue 2018-05-15 11:00:48 CEST; 22h ago
     Docs: man:systemd-sysv-generator(8)
  Process: 13975 ExecStop=/etc/rc.d/init.d/elasticsearch stop (code=exited, status=0/SUCCESS)
  Process: 5752 ExecReload=/etc/rc.d/init.d/elasticsearch reload (code=exited, status=7)
  Process: 14048 ExecStart=/etc/rc.d/init.d/elasticsearch start (code=exited, status=0/SUCCESS)

May 15 11:00:48 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Starting LSB: This service manages the elasticsearch daemon...
May 15 11:00:48 ip-172-31-1-24.eu-west-1.compute.internal runuser[14065]: pam_unix(runuser:session): session opened for user nagios by (uid=0)
May 15 11:00:48 ip-172-31-1-24.eu-west-1.compute.internal elasticsearch[14048]: Starting elasticsearch: [  OK  ]
May 15 11:00:48 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Started LSB: This service manages the elasticsearch daemon.

Re: Problem whit logstash

Posted: Wed May 16, 2018 3:21 am
by sbarrera
sbarrera wrote:
Captura1.PNG
So do you know why when i turn on the logstah.service the elasticsearch.service turn into active(exited) mode and the server pop me Waiting for Elasticsearch.

What could be the problem?

Code: Select all

 service logstash status
Logstash Daemon● logstash.service - LSB: Logstash
   Loaded: loaded (/etc/rc.d/init.d/logstash; bad; vendor preset: disabled)
   Active: active (running) since Wed 2018-05-16 09:10:07 CEST; 9s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 13707 ExecStop=/etc/rc.d/init.d/logstash stop (code=exited, status=0/SUCCESS)
  Process: 6844 ExecStart=/etc/rc.d/init.d/logstash start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/logstash.service
           ├─6854 runuser -s /bin/sh -c exec /usr/local/nagioslogserver/logstash/bin/logstash agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/logstash/logstash.log  -w 4...
           └─6856 java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryE...

May 16 09:10:06 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Starting LSB: Logstash...
May 16 09:10:06 ip-172-31-1-24.eu-west-1.compute.internal runuser[6854]: pam_unix(runuser:session): session opened for user nagios by (uid=0)
May 16 09:10:07 ip-172-31-1-24.eu-west-1.compute.internal logstash[6844]: Starting Logstash Daemon: [  OK  ]
May 16 09:10:07 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Started LSB: Logstash.

Code: Select all

service elasticsearch status
● elasticsearch.service - LSB: This service manages the elasticsearch daemon
   Loaded: loaded (/etc/rc.d/init.d/elasticsearch; bad; vendor preset: disabled)
   Active: active (exited) since Tue 2018-05-15 11:00:48 CEST; 22h ago
     Docs: man:systemd-sysv-generator(8)
  Process: 13975 ExecStop=/etc/rc.d/init.d/elasticsearch stop (code=exited, status=0/SUCCESS)
  Process: 5752 ExecReload=/etc/rc.d/init.d/elasticsearch reload (code=exited, status=7)
  Process: 14048 ExecStart=/etc/rc.d/init.d/elasticsearch start (code=exited, status=0/SUCCESS)

May 15 11:00:48 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Starting LSB: This service manages the elasticsearch daemon...
May 15 11:00:48 ip-172-31-1-24.eu-west-1.compute.internal runuser[14065]: pam_unix(runuser:session): session opened for user nagios by (uid=0)
May 15 11:00:48 ip-172-31-1-24.eu-west-1.compute.internal elasticsearch[14048]: Starting elasticsearch: [  OK  ]
May 15 11:00:48 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Started LSB: This service manages the elasticsearch daemon.
i´ve just found this:

Code: Select all

tail -n 5 /var/log/logstash/logstash.log
{:timestamp=>"2018-05-16T09:17:33.745000+0200", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://localhost:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2018-05-16T09:17:34.499000+0200", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://localhost:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2018-05-16T09:17:34.506000+0200", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://localhost:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2018-05-16T09:17:34.692000+0200", :message=>"SIGTERM received. Shutting down the agent.", :level=>:warn}
{:timestamp=>"2018-05-16T09:17:34.693000+0200", :message=>"stopping pipeline", :id=>"main"}
Maybe it will help.

Re: Problem whit logstash

Posted: Wed May 16, 2018 7:51 am
by scottwilkerson
elasticsearch isn't running

Code: Select all

systemctl start elasticsearch
Also, how much memory does this server have?

Re: Problem whit logstash

Posted: Thu May 17, 2018 2:16 am
by sbarrera
Elasticsearch is running:(maybe because it´s an older log i don´t know)

Code: Select all

service elasticsearch status
● elasticsearch.service - LSB: This service manages the elasticsearch daemon
   Loaded: loaded (/etc/rc.d/init.d/elasticsearch; bad; vendor preset: disabled)
   Active: active (running) since Wed 2018-05-16 09:18:11 CEST; 23h ago
     Docs: man:systemd-sysv-generator(8)
  Process: 8368 ExecStop=/etc/rc.d/init.d/elasticsearch stop (code=exited, status=0/SUCCESS)
  Process: 5752 ExecReload=/etc/rc.d/init.d/elasticsearch reload (code=exited, status=7)
  Process: 8378 ExecStart=/etc/rc.d/init.d/elasticsearch start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/elasticsearch.service
           └─8404 java -Xms918m -Xmx918m -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+Heap...

May 16 09:18:11 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Starting LSB: This service manages the elasticsearch daemon...
May 16 09:18:11 ip-172-31-1-24.eu-west-1.compute.internal runuser[8395]: pam_unix(runuser:session): session opened for user nagios by (uid=0)
May 16 09:18:11 ip-172-31-1-24.eu-west-1.compute.internal elasticsearch[8378]: Starting elasticsearch: [  OK  ]
May 16 09:18:11 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Started LSB: This service manages the elasticsearch daemon.
The one that i can´t start up because the server goes down is logstash.service (it turn elasticsearch into active(excited))

Code: Select all

service logstash status
Logstash Daemon● logstash.service - LSB: Logstash
   Loaded: loaded (/etc/rc.d/init.d/logstash; bad; vendor preset: disabled)
   Active: inactive (dead) since Wed 2018-05-16 09:17:35 CEST; 23h ago
     Docs: man:systemd-sysv-generator(8)
  Process: 8178 ExecStop=/etc/rc.d/init.d/logstash stop (code=exited, status=0/SUCCESS)
  Process: 6844 ExecStart=/etc/rc.d/init.d/logstash start (code=exited, status=0/SUCCESS)

May 16 09:10:07 ip-172-31-1-24.eu-west-1.compute.internal logstash[6844]: Starting Logstash Daemon: [  OK  ]
May 16 09:10:07 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Started LSB: Logstash.
May 16 09:17:34 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Stopping LSB: Logstash...
May 16 09:17:34 ip-172-31-1-24.eu-west-1.compute.internal logstash[6844]: IOError: closed stream
May 16 09:17:34 ip-172-31-1-24.eu-west-1.compute.internal logstash[6844]: peeraddr at org/jruby/ext/socket/RubyIPSocket.java:95
May 16 09:17:34 ip-172-31-1-24.eu-west-1.compute.internal logstash[6844]: tcp_receiver at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0...og.rb:169
May 16 09:17:34 ip-172-31-1-24.eu-west-1.compute.internal logstash[6844]: tcp_listener at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0...og.rb:159
May 16 09:17:34 ip-172-31-1-24.eu-west-1.compute.internal runuser[6854]: pam_unix(runuser:session): session closed for user nagios
May 16 09:17:35 ip-172-31-1-24.eu-west-1.compute.internal logstash[8178]: Stopping Logstash Daemon: [  OK  ]
May 16 09:17:35 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Stopped LSB: Logstash.
Hint: Some lines were ellipsized, use -l to show in full
RAM:

Code: Select all

free
              total        used        free      shared  buff/cache   available
Mem:        1881228     1460072       72860       82568      348296       57084
Swap:             0           0           0
Hard Drive:

Code: Select all

df -hT
Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/xvda2     xfs        10G  5.6G  4.5G  56% /
devtmpfs       devtmpfs  897M     0  897M   0% /dev
tmpfs          tmpfs     919M     0  919M   0% /dev/shm
tmpfs          tmpfs     919M   81M  839M   9% /run
tmpfs          tmpfs     919M     0  919M   0% /sys/fs/cgroup
tmpfs          tmpfs     184M     0  184M   0% /run/user/1001
tmpfs          tmpfs     184M     0  184M   0% /run/user/1000

Re: Problem whit logstash

Posted: Thu May 17, 2018 8:21 am
by scottwilkerson
you might want to try rebooting the server

I have no idea why you would be getting the following unless there was a problem reading/writing to a device or drive

Code: Select all

IOError: closed stream
All times are UTC-05:00
Page 1 of 1

Powered by phpBB® Forum Software © phpBB Limited