Problem whit logstash

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Locked
sbarrera
Posts: 11
Joined: Thu Apr 26, 2018 2:40 am

Problem whit logstash

Post by sbarrera »

Hi i´m having a problem whit my nagios log server, i can only login it whit the logstash service inactive, when i started it pop me like waiting for elasticsearch to startup and the elasticsearch.service turn into active(excited) mode.

Ask me for all the information that i can provide.
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: Problem whit logstash

Post by scottwilkerson »

You should only be able to login if elasticsearch is running. Elasticsearch is the datastore and also holds all your user information.

Logstash actually doesn't interact with user login whatsoever.
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
sbarrera
Posts: 11
Joined: Thu Apr 26, 2018 2:40 am

Re: Problem whit logstash

Post by sbarrera »

Captura1.PNG
So do you know why when i turn on the logstah.service the elasticsearch.service turn into active(exited) mode and the server pop me Waiting for Elasticsearch.

What could be the problem?

Code: Select all

 service logstash status
Logstash Daemon● logstash.service - LSB: Logstash
   Loaded: loaded (/etc/rc.d/init.d/logstash; bad; vendor preset: disabled)
   Active: active (running) since Wed 2018-05-16 09:10:07 CEST; 9s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 13707 ExecStop=/etc/rc.d/init.d/logstash stop (code=exited, status=0/SUCCESS)
  Process: 6844 ExecStart=/etc/rc.d/init.d/logstash start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/logstash.service
           ├─6854 runuser -s /bin/sh -c exec /usr/local/nagioslogserver/logstash/bin/logstash agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/logstash/logstash.log  -w 4...
           └─6856 java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryE...

May 16 09:10:06 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Starting LSB: Logstash...
May 16 09:10:06 ip-172-31-1-24.eu-west-1.compute.internal runuser[6854]: pam_unix(runuser:session): session opened for user nagios by (uid=0)
May 16 09:10:07 ip-172-31-1-24.eu-west-1.compute.internal logstash[6844]: Starting Logstash Daemon: [  OK  ]
May 16 09:10:07 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Started LSB: Logstash.

Code: Select all

service elasticsearch status
● elasticsearch.service - LSB: This service manages the elasticsearch daemon
   Loaded: loaded (/etc/rc.d/init.d/elasticsearch; bad; vendor preset: disabled)
   Active: active (exited) since Tue 2018-05-15 11:00:48 CEST; 22h ago
     Docs: man:systemd-sysv-generator(8)
  Process: 13975 ExecStop=/etc/rc.d/init.d/elasticsearch stop (code=exited, status=0/SUCCESS)
  Process: 5752 ExecReload=/etc/rc.d/init.d/elasticsearch reload (code=exited, status=7)
  Process: 14048 ExecStart=/etc/rc.d/init.d/elasticsearch start (code=exited, status=0/SUCCESS)

May 15 11:00:48 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Starting LSB: This service manages the elasticsearch daemon...
May 15 11:00:48 ip-172-31-1-24.eu-west-1.compute.internal runuser[14065]: pam_unix(runuser:session): session opened for user nagios by (uid=0)
May 15 11:00:48 ip-172-31-1-24.eu-west-1.compute.internal elasticsearch[14048]: Starting elasticsearch: [  OK  ]
May 15 11:00:48 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Started LSB: This service manages the elasticsearch daemon.
You do not have the required permissions to view the files attached to this post.
sbarrera
Posts: 11
Joined: Thu Apr 26, 2018 2:40 am

Re: Problem whit logstash

Post by sbarrera »

sbarrera wrote:
Captura1.PNG
So do you know why when i turn on the logstah.service the elasticsearch.service turn into active(exited) mode and the server pop me Waiting for Elasticsearch.

What could be the problem?

Code: Select all

 service logstash status
Logstash Daemon● logstash.service - LSB: Logstash
   Loaded: loaded (/etc/rc.d/init.d/logstash; bad; vendor preset: disabled)
   Active: active (running) since Wed 2018-05-16 09:10:07 CEST; 9s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 13707 ExecStop=/etc/rc.d/init.d/logstash stop (code=exited, status=0/SUCCESS)
  Process: 6844 ExecStart=/etc/rc.d/init.d/logstash start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/logstash.service
           ├─6854 runuser -s /bin/sh -c exec /usr/local/nagioslogserver/logstash/bin/logstash agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/logstash/logstash.log  -w 4...
           └─6856 java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryE...

May 16 09:10:06 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Starting LSB: Logstash...
May 16 09:10:06 ip-172-31-1-24.eu-west-1.compute.internal runuser[6854]: pam_unix(runuser:session): session opened for user nagios by (uid=0)
May 16 09:10:07 ip-172-31-1-24.eu-west-1.compute.internal logstash[6844]: Starting Logstash Daemon: [  OK  ]
May 16 09:10:07 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Started LSB: Logstash.

Code: Select all

service elasticsearch status
● elasticsearch.service - LSB: This service manages the elasticsearch daemon
   Loaded: loaded (/etc/rc.d/init.d/elasticsearch; bad; vendor preset: disabled)
   Active: active (exited) since Tue 2018-05-15 11:00:48 CEST; 22h ago
     Docs: man:systemd-sysv-generator(8)
  Process: 13975 ExecStop=/etc/rc.d/init.d/elasticsearch stop (code=exited, status=0/SUCCESS)
  Process: 5752 ExecReload=/etc/rc.d/init.d/elasticsearch reload (code=exited, status=7)
  Process: 14048 ExecStart=/etc/rc.d/init.d/elasticsearch start (code=exited, status=0/SUCCESS)

May 15 11:00:48 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Starting LSB: This service manages the elasticsearch daemon...
May 15 11:00:48 ip-172-31-1-24.eu-west-1.compute.internal runuser[14065]: pam_unix(runuser:session): session opened for user nagios by (uid=0)
May 15 11:00:48 ip-172-31-1-24.eu-west-1.compute.internal elasticsearch[14048]: Starting elasticsearch: [  OK  ]
May 15 11:00:48 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Started LSB: This service manages the elasticsearch daemon.
i´ve just found this:

Code: Select all

tail -n 5 /var/log/logstash/logstash.log
{:timestamp=>"2018-05-16T09:17:33.745000+0200", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://localhost:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2018-05-16T09:17:34.499000+0200", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://localhost:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2018-05-16T09:17:34.506000+0200", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://localhost:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2018-05-16T09:17:34.692000+0200", :message=>"SIGTERM received. Shutting down the agent.", :level=>:warn}
{:timestamp=>"2018-05-16T09:17:34.693000+0200", :message=>"stopping pipeline", :id=>"main"}
Maybe it will help.
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: Problem whit logstash

Post by scottwilkerson »

elasticsearch isn't running

Code: Select all

systemctl start elasticsearch
Also, how much memory does this server have?
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
sbarrera
Posts: 11
Joined: Thu Apr 26, 2018 2:40 am

Re: Problem whit logstash

Post by sbarrera »

Elasticsearch is running:(maybe because it´s an older log i don´t know)

Code: Select all

service elasticsearch status
● elasticsearch.service - LSB: This service manages the elasticsearch daemon
   Loaded: loaded (/etc/rc.d/init.d/elasticsearch; bad; vendor preset: disabled)
   Active: active (running) since Wed 2018-05-16 09:18:11 CEST; 23h ago
     Docs: man:systemd-sysv-generator(8)
  Process: 8368 ExecStop=/etc/rc.d/init.d/elasticsearch stop (code=exited, status=0/SUCCESS)
  Process: 5752 ExecReload=/etc/rc.d/init.d/elasticsearch reload (code=exited, status=7)
  Process: 8378 ExecStart=/etc/rc.d/init.d/elasticsearch start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/elasticsearch.service
           └─8404 java -Xms918m -Xmx918m -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+Heap...

May 16 09:18:11 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Starting LSB: This service manages the elasticsearch daemon...
May 16 09:18:11 ip-172-31-1-24.eu-west-1.compute.internal runuser[8395]: pam_unix(runuser:session): session opened for user nagios by (uid=0)
May 16 09:18:11 ip-172-31-1-24.eu-west-1.compute.internal elasticsearch[8378]: Starting elasticsearch: [  OK  ]
May 16 09:18:11 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Started LSB: This service manages the elasticsearch daemon.
The one that i can´t start up because the server goes down is logstash.service (it turn elasticsearch into active(excited))

Code: Select all

service logstash status
Logstash Daemon● logstash.service - LSB: Logstash
   Loaded: loaded (/etc/rc.d/init.d/logstash; bad; vendor preset: disabled)
   Active: inactive (dead) since Wed 2018-05-16 09:17:35 CEST; 23h ago
     Docs: man:systemd-sysv-generator(8)
  Process: 8178 ExecStop=/etc/rc.d/init.d/logstash stop (code=exited, status=0/SUCCESS)
  Process: 6844 ExecStart=/etc/rc.d/init.d/logstash start (code=exited, status=0/SUCCESS)

May 16 09:10:07 ip-172-31-1-24.eu-west-1.compute.internal logstash[6844]: Starting Logstash Daemon: [  OK  ]
May 16 09:10:07 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Started LSB: Logstash.
May 16 09:17:34 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Stopping LSB: Logstash...
May 16 09:17:34 ip-172-31-1-24.eu-west-1.compute.internal logstash[6844]: IOError: closed stream
May 16 09:17:34 ip-172-31-1-24.eu-west-1.compute.internal logstash[6844]: peeraddr at org/jruby/ext/socket/RubyIPSocket.java:95
May 16 09:17:34 ip-172-31-1-24.eu-west-1.compute.internal logstash[6844]: tcp_receiver at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0...og.rb:169
May 16 09:17:34 ip-172-31-1-24.eu-west-1.compute.internal logstash[6844]: tcp_listener at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0...og.rb:159
May 16 09:17:34 ip-172-31-1-24.eu-west-1.compute.internal runuser[6854]: pam_unix(runuser:session): session closed for user nagios
May 16 09:17:35 ip-172-31-1-24.eu-west-1.compute.internal logstash[8178]: Stopping Logstash Daemon: [  OK  ]
May 16 09:17:35 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Stopped LSB: Logstash.
Hint: Some lines were ellipsized, use -l to show in full
RAM:

Code: Select all

free
              total        used        free      shared  buff/cache   available
Mem:        1881228     1460072       72860       82568      348296       57084
Swap:             0           0           0
Hard Drive:

Code: Select all

df -hT
Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/xvda2     xfs        10G  5.6G  4.5G  56% /
devtmpfs       devtmpfs  897M     0  897M   0% /dev
tmpfs          tmpfs     919M     0  919M   0% /dev/shm
tmpfs          tmpfs     919M   81M  839M   9% /run
tmpfs          tmpfs     919M     0  919M   0% /sys/fs/cgroup
tmpfs          tmpfs     184M     0  184M   0% /run/user/1001
tmpfs          tmpfs     184M     0  184M   0% /run/user/1000
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: Problem whit logstash

Post by scottwilkerson »

you might want to try rebooting the server

I have no idea why you would be getting the following unless there was a problem reading/writing to a device or drive

Code: Select all

IOError: closed stream
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
Locked