Problem whit logstash
Problem whit logstash
Hi i´m having a problem whit my nagios log server, i can only login it whit the logstash service inactive, when i started it pop me like waiting for elasticsearch to startup and the elasticsearch.service turn into active(excited) mode.
Ask me for all the information that i can provide.
Ask me for all the information that i can provide.
-
- DevOps Engineer
- Posts: 19396
- Joined: Tue Nov 15, 2011 3:11 pm
- Location: Nagios Enterprises
- Contact:
Re: Problem whit logstash
You should only be able to login if elasticsearch is running. Elasticsearch is the datastore and also holds all your user information.
Logstash actually doesn't interact with user login whatsoever.
Logstash actually doesn't interact with user login whatsoever.
Re: Problem whit logstash
What could be the problem?
Code: Select all
service logstash status
Logstash Daemon● logstash.service - LSB: Logstash
Loaded: loaded (/etc/rc.d/init.d/logstash; bad; vendor preset: disabled)
Active: active (running) since Wed 2018-05-16 09:10:07 CEST; 9s ago
Docs: man:systemd-sysv-generator(8)
Process: 13707 ExecStop=/etc/rc.d/init.d/logstash stop (code=exited, status=0/SUCCESS)
Process: 6844 ExecStart=/etc/rc.d/init.d/logstash start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/logstash.service
├─6854 runuser -s /bin/sh -c exec /usr/local/nagioslogserver/logstash/bin/logstash agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/logstash/logstash.log -w 4...
└─6856 java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryE...
May 16 09:10:06 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Starting LSB: Logstash...
May 16 09:10:06 ip-172-31-1-24.eu-west-1.compute.internal runuser[6854]: pam_unix(runuser:session): session opened for user nagios by (uid=0)
May 16 09:10:07 ip-172-31-1-24.eu-west-1.compute.internal logstash[6844]: Starting Logstash Daemon: [ OK ]
May 16 09:10:07 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Started LSB: Logstash.
Code: Select all
service elasticsearch status
● elasticsearch.service - LSB: This service manages the elasticsearch daemon
Loaded: loaded (/etc/rc.d/init.d/elasticsearch; bad; vendor preset: disabled)
Active: active (exited) since Tue 2018-05-15 11:00:48 CEST; 22h ago
Docs: man:systemd-sysv-generator(8)
Process: 13975 ExecStop=/etc/rc.d/init.d/elasticsearch stop (code=exited, status=0/SUCCESS)
Process: 5752 ExecReload=/etc/rc.d/init.d/elasticsearch reload (code=exited, status=7)
Process: 14048 ExecStart=/etc/rc.d/init.d/elasticsearch start (code=exited, status=0/SUCCESS)
May 15 11:00:48 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Starting LSB: This service manages the elasticsearch daemon...
May 15 11:00:48 ip-172-31-1-24.eu-west-1.compute.internal runuser[14065]: pam_unix(runuser:session): session opened for user nagios by (uid=0)
May 15 11:00:48 ip-172-31-1-24.eu-west-1.compute.internal elasticsearch[14048]: Starting elasticsearch: [ OK ]
May 15 11:00:48 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Started LSB: This service manages the elasticsearch daemon.
You do not have the required permissions to view the files attached to this post.
Re: Problem whit logstash
i´ve just found this:sbarrera wrote:So do you know why when i turn on the logstah.service the elasticsearch.service turn into active(exited) mode and the server pop me Waiting for Elasticsearch.
What could be the problem?
Code: Select all
service logstash status Logstash Daemon● logstash.service - LSB: Logstash Loaded: loaded (/etc/rc.d/init.d/logstash; bad; vendor preset: disabled) Active: active (running) since Wed 2018-05-16 09:10:07 CEST; 9s ago Docs: man:systemd-sysv-generator(8) Process: 13707 ExecStop=/etc/rc.d/init.d/logstash stop (code=exited, status=0/SUCCESS) Process: 6844 ExecStart=/etc/rc.d/init.d/logstash start (code=exited, status=0/SUCCESS) CGroup: /system.slice/logstash.service ├─6854 runuser -s /bin/sh -c exec /usr/local/nagioslogserver/logstash/bin/logstash agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/logstash/logstash.log -w 4... └─6856 java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryE... May 16 09:10:06 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Starting LSB: Logstash... May 16 09:10:06 ip-172-31-1-24.eu-west-1.compute.internal runuser[6854]: pam_unix(runuser:session): session opened for user nagios by (uid=0) May 16 09:10:07 ip-172-31-1-24.eu-west-1.compute.internal logstash[6844]: Starting Logstash Daemon: [ OK ] May 16 09:10:07 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Started LSB: Logstash.
Code: Select all
service elasticsearch status ● elasticsearch.service - LSB: This service manages the elasticsearch daemon Loaded: loaded (/etc/rc.d/init.d/elasticsearch; bad; vendor preset: disabled) Active: active (exited) since Tue 2018-05-15 11:00:48 CEST; 22h ago Docs: man:systemd-sysv-generator(8) Process: 13975 ExecStop=/etc/rc.d/init.d/elasticsearch stop (code=exited, status=0/SUCCESS) Process: 5752 ExecReload=/etc/rc.d/init.d/elasticsearch reload (code=exited, status=7) Process: 14048 ExecStart=/etc/rc.d/init.d/elasticsearch start (code=exited, status=0/SUCCESS) May 15 11:00:48 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Starting LSB: This service manages the elasticsearch daemon... May 15 11:00:48 ip-172-31-1-24.eu-west-1.compute.internal runuser[14065]: pam_unix(runuser:session): session opened for user nagios by (uid=0) May 15 11:00:48 ip-172-31-1-24.eu-west-1.compute.internal elasticsearch[14048]: Starting elasticsearch: [ OK ] May 15 11:00:48 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Started LSB: This service manages the elasticsearch daemon.
Code: Select all
tail -n 5 /var/log/logstash/logstash.log
{:timestamp=>"2018-05-16T09:17:33.745000+0200", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://localhost:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2018-05-16T09:17:34.499000+0200", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://localhost:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2018-05-16T09:17:34.506000+0200", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://localhost:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2018-05-16T09:17:34.692000+0200", :message=>"SIGTERM received. Shutting down the agent.", :level=>:warn}
{:timestamp=>"2018-05-16T09:17:34.693000+0200", :message=>"stopping pipeline", :id=>"main"}
-
- DevOps Engineer
- Posts: 19396
- Joined: Tue Nov 15, 2011 3:11 pm
- Location: Nagios Enterprises
- Contact:
Re: Problem whit logstash
elasticsearch isn't running
Also, how much memory does this server have?
Code: Select all
systemctl start elasticsearch
Re: Problem whit logstash
Elasticsearch is running:(maybe because it´s an older log i don´t know)
The one that i can´t start up because the server goes down is logstash.service (it turn elasticsearch into active(excited))
RAM:
Hard Drive:
Code: Select all
service elasticsearch status
● elasticsearch.service - LSB: This service manages the elasticsearch daemon
Loaded: loaded (/etc/rc.d/init.d/elasticsearch; bad; vendor preset: disabled)
Active: active (running) since Wed 2018-05-16 09:18:11 CEST; 23h ago
Docs: man:systemd-sysv-generator(8)
Process: 8368 ExecStop=/etc/rc.d/init.d/elasticsearch stop (code=exited, status=0/SUCCESS)
Process: 5752 ExecReload=/etc/rc.d/init.d/elasticsearch reload (code=exited, status=7)
Process: 8378 ExecStart=/etc/rc.d/init.d/elasticsearch start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/elasticsearch.service
└─8404 java -Xms918m -Xmx918m -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+Heap...
May 16 09:18:11 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Starting LSB: This service manages the elasticsearch daemon...
May 16 09:18:11 ip-172-31-1-24.eu-west-1.compute.internal runuser[8395]: pam_unix(runuser:session): session opened for user nagios by (uid=0)
May 16 09:18:11 ip-172-31-1-24.eu-west-1.compute.internal elasticsearch[8378]: Starting elasticsearch: [ OK ]
May 16 09:18:11 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Started LSB: This service manages the elasticsearch daemon.
Code: Select all
service logstash status
Logstash Daemon● logstash.service - LSB: Logstash
Loaded: loaded (/etc/rc.d/init.d/logstash; bad; vendor preset: disabled)
Active: inactive (dead) since Wed 2018-05-16 09:17:35 CEST; 23h ago
Docs: man:systemd-sysv-generator(8)
Process: 8178 ExecStop=/etc/rc.d/init.d/logstash stop (code=exited, status=0/SUCCESS)
Process: 6844 ExecStart=/etc/rc.d/init.d/logstash start (code=exited, status=0/SUCCESS)
May 16 09:10:07 ip-172-31-1-24.eu-west-1.compute.internal logstash[6844]: Starting Logstash Daemon: [ OK ]
May 16 09:10:07 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Started LSB: Logstash.
May 16 09:17:34 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Stopping LSB: Logstash...
May 16 09:17:34 ip-172-31-1-24.eu-west-1.compute.internal logstash[6844]: IOError: closed stream
May 16 09:17:34 ip-172-31-1-24.eu-west-1.compute.internal logstash[6844]: peeraddr at org/jruby/ext/socket/RubyIPSocket.java:95
May 16 09:17:34 ip-172-31-1-24.eu-west-1.compute.internal logstash[6844]: tcp_receiver at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0...og.rb:169
May 16 09:17:34 ip-172-31-1-24.eu-west-1.compute.internal logstash[6844]: tcp_listener at /usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-2.0...og.rb:159
May 16 09:17:34 ip-172-31-1-24.eu-west-1.compute.internal runuser[6854]: pam_unix(runuser:session): session closed for user nagios
May 16 09:17:35 ip-172-31-1-24.eu-west-1.compute.internal logstash[8178]: Stopping Logstash Daemon: [ OK ]
May 16 09:17:35 ip-172-31-1-24.eu-west-1.compute.internal systemd[1]: Stopped LSB: Logstash.
Hint: Some lines were ellipsized, use -l to show in full
Code: Select all
free
total used free shared buff/cache available
Mem: 1881228 1460072 72860 82568 348296 57084
Swap: 0 0 0
Code: Select all
df -hT
Filesystem Type Size Used Avail Use% Mounted on
/dev/xvda2 xfs 10G 5.6G 4.5G 56% /
devtmpfs devtmpfs 897M 0 897M 0% /dev
tmpfs tmpfs 919M 0 919M 0% /dev/shm
tmpfs tmpfs 919M 81M 839M 9% /run
tmpfs tmpfs 919M 0 919M 0% /sys/fs/cgroup
tmpfs tmpfs 184M 0 184M 0% /run/user/1001
tmpfs tmpfs 184M 0 184M 0% /run/user/1000
-
- DevOps Engineer
- Posts: 19396
- Joined: Tue Nov 15, 2011 3:11 pm
- Location: Nagios Enterprises
- Contact:
Re: Problem whit logstash
you might want to try rebooting the server
I have no idea why you would be getting the following unless there was a problem reading/writing to a device or drive
I have no idea why you would be getting the following unless there was a problem reading/writing to a device or drive
Code: Select all
IOError: closed stream