Log Server install error

[bpennin]

Trying to install the latest versions on RHEL 8: What access is required or how can I tell what connection was reset?
Applying Nagios patches to Logstash...
Installing LS Openstash Plugins
Using bundled JDK: /usr/local/nagioslogserver/logstash/jdk
ERROR: Something went wrong when installing logstash-output-opensearch, logstash-input-syslog, logstash-input-eventlog, logstash-output-exec, logstash-input-file, logstash-filter-json, logstash-filter-ruby, message: Connection reset

===================
INSTALLATION ERROR!
===================
Installation step failed - exiting.

[jsimon]

Hi @bpennin,

"Connection reset" is referring to the TCP connection with the package sources in the related "ERROR" statement. Basically it seems like something went wrong with your network connection while these packages were being downloaded. It's pretty unlikely that this is a software issue with the Nagios Log Server install. I'd recommend double checking that you have a stable connection and trying the install again. If this reoccurs, you may have issues with your local firewall, or some network configuration may be interfering with the request.

[Herlit]

Sorry to hear you've encountered this error! Based on the description, the "connection reset" issue when installing Logstash plugins may be related to network restrictions or access to repositories. Here are a few thoughts:

Check network access: Make sure your RHEL 8 server has access to the Elastic repositories (artifacts.elastic.co). Try curl -v artifacts.elastic.co/downloads/logstash-plugins/ — this will show if the request is getting through or where it's breaking. A proxy or firewall may be blocking the connection.
Permissions: Check which user you're running the installation as. /usr/local/nagioslogserver/logstash/ may require root or write permissions. Try sudo or check the directory owner: ls -ld /usr/local/nagioslogserver/logstash/.
Logs and diagnostics: Check the detailed logs in /usr/local/nagioslogserver/logstash/logs/ or the output of the command with the --debug flag. This may give a hint at what point the connection is being dropped.
Offline installation: If the network is limited, try downloading the plugins manually from artifacts.elastic.co and installing them locally via bin/logstash-plugin install file:///path/to/plugin.zip.
Can you clarify whether there is internet access from the server and what proxy/firewall settings are used? This will help narrow down the causes. Good luck with the installation!

[bpennin]

We have it installed now using the manual zip file. Everything is running, but nothing is listening on port 5544 on the log server. No logs are being seen and connection refused errors in rsyslogd:
cannot connect to xx.xxx.x.xxx:5544: Connection refused [v8.2102.0-15.el8_10.1 try https://www.rsyslog.com/e/2027 ]

Any ideas - frustrating....

[jmichaelson]

Check the status of the logstash service. Execute the following :

Code: Select all

systemctl status logstash.service

If its not running, try and start it:

Code: Select all

systemctl start logstash.service

If that doesn't work, have a look at /usr/local/nagioslogserver/logstash/logs/logstash-plain.log to see if there's any information why it won't start. feel free to post (or attach it) here and we can dive in and have a deeper look at why its not running.

[bpennin]

Found the issue, which seems odd, but here it is - systemctl status logstash had this error:
logstash[417121]: ERROR: Unknown command 'agent'

We did some digging and ended up editing /etc/init.d/logstash

vi /etc/init.d/logstash
remove the word Agent and leave a space in the following line:
DAEMON_OPTS=" -f ${LS_CONF_DIR} -l ${LS_LOG_FILE} ${LS_OPTS}"

Systemctl daemon-reload

Systemctl restart logstash

We see logs coming in and the error is gone. Any ideas why????

[jmichaelson]

I'm unsure. I wasn't expecting RHEL 8 to use init.d instead of systemd when it was there. I've opened an internal issue for this, and even though you're past this hurdle, if you're interested you can atch the changelog for NLS#566 in the future.

[bpennin]

New fun - all of a sudden I cant login from the GUI - I try to us the reset_nagiosadmin_password.sh script and it gets the following:
Enter a new password:
{"error":{"root_cause":[{"type":"unavailable_shards_exception","reason":"[nagioslogserver_user][0] [1] shardIt, [0] active : Timeout waiting for [1m], request: indices:data/write/update"}],"type":"unavailable_shards_exception","reason":"[nagioslogserver_user][0] [1] shardIt, [0] active : Timeout waiting for [1m], request: indices:data/write/update"},"status":503}
{"_shards":{"total":2,"successful":0,"failed":0}}The password has been set for nagiosadmin user.

I am running the latest release and patch or R2

[jmichaelson]

Is OpenSearch dunning? same systemctl command, replace logstash with OpenSearch.

Also do you have multiple nodes in your cluster or is it just a single instance?

If OpenSearch is running, check the disk space on the system to ensure it hasn't filled up; if it has we can start to investigate why.

[bpennin]

Opensearch and logstash are running - i did a reboot:

both are running


df -h on filesystems:

nothing close to full

Tried to reset password again: ./reset_nagiosadmin_password.sh -p goal,1goal
{"error":{"root_cause":[{"type":"unavailable_shards_exception","reason":"[nagioslogserver_user][0] [1] shardIt, [0] active : Timeout waiting for [1m], request: indices:data/write/update"}],"type":"unavailable_shards_exception","reason":"[nagioslogserver_user][0] [1] shardIt, [0] active : Timeout waiting for [1m], request: indices:data/write/update"},"status":503}{"_shards":{"total":2,"successful":0,"failed":0}}The password has been set for nagiosadmin user.

Additional Notes:
After getting logs in the system yesterday we did add a Nagios XI server and tried to add an Apache from the local machine. I was also trying to a fullinstall -a on another node so it could join the cluster but, it failed with some opensearch issues.