Apply and Save and Apply not finishing in WebUI

[jaimie.livingston]

We are evaluating Log Server as an addition to our XI based monitoring, to replace the multitude of home grown ELK stacks created by various groups in our datacenter and hosting operations groups. We have been running a 2.0.8 trial version for a few weeks with no issues.

After the 2.1.0 update, I now have a persistent issue with the Apply Configuration UI operations running without finishing after making changes to inputs, input filters or output filters -- the " Running..." ticker just sits and spins for as long as I'm willing to let it. So far, I've let one iteration run for several hours, just to see if it would complete.

Looking at the actual config files, the configuration seem to get saved and applied, and clicking away from the Apply Configuration screen doesn't generate any errors or warnings that I can find.

In the /usr/local/nagioslogserver, a new tmp5* folder (containing 000_inputs.conf 500_filters.conf 501_live_filters.conf 998_live_outputs.conf 999_outputs.conf) is created each time the Verify, Save and Appy, or Apply buttons are clicked. The previous tmp5* folder is not removed.

Any thoughts on what might be going on?
I'd like to resolve this before I have to start writing my evaluation report in the next few days.

[swolf]

We haven't run into this issue so far during testing. What distribution are you running on?

You mentioned you didn't notice any errors or warnings, so just for due diligence:
Do you see any errors in /var/log/messages or in your apache log (default /var/log/httpd/error_log for RPM-based distros, /var/log/apache2/error.log for DEB-based distros)?
Assuming you don't, if you open your browser console (ctrl+shift+i), do you see any errors in the console or in the network tab?

The tmp folders shouldn't be causing any issues. You can remove them if you'd like, but it won't affect anything.

[jaimie.livingston]

Hi There...

Distro Info...

# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"


I found nothing in /var/log/messages, /var/log/httpd/error.log, and no errors in DevTools console or network output.
I also poked around for log files in the /usr/local/nagioslogserver/* directories and found nothing unusual.

On a lark, I did a system reboot earlier, and the Apply completed normally the first time after the reboot, but goes back to sit-n-spin at "Running..." for subsequent Apply actions.

AFAICT, the changes I've been making to input, input filters, and output filters are applied to the live configs, and I can find no issues in the server function.
That said, I'm still in the familiarisation stage of my evaluation, so there could be something amiss that I just don't know enough to notice.

Jaimie Livingston

[ssax]

Please attach your /etc/sudoers as well as the output of these commands:

Code: Select all

chage -l nagios
chage -l apache

Please run this tail command (as root) and leave it running:

Code: Select all

tail -Fn0 /var/log/messages /var/log/httpd/error_log /var/log/httpd/ssl_error_log /usr/local/nagioslogserver/var/auditlog.log /usr/local/nagioslogserver/var/hosts.log /usr/local/nagioslogserver/var/jobs.log /usr/local/nagioslogserver/var/poller.log /var/log/elasticsearch/*.log >> /tmp/SUPPORTOUTPUT.txt

Then, follow this process:
- Close your browser, reopen it (use Chrome if you're not, please try with another browser as well as it could be proxy related)
- Login
- Note the time (send that time to us)
- Go make the changes but don't apply configuration yet
- Open up Developer Tools in Chrome / Firebug / etc and send me the entire output (the response tab in chrome developer tools)
- Click the Network Tab
- Leave the developer tools open and Apply Configuration
- Once it's been stuck for a bit, find the latest one for this and left-click on it and send me a screenshot of it, then do the same with the Response tab (I want to see status/return codes AND the full output from the Response tab (if anything)):

Code: Select all

http://YOUR_LOG_SERVER/nagioslogserver/api/system/get_cmd_info

- You'll see it making requests to that over and over because it's trying to get the results back from the backend.

Now you can CTRL-C the tail command from the beginning to stop it, and send me the resulting /tmp/SUPPORTOUTPUT.txt file that was generated by the tail command.


swolf wanted me to post this as well:

As far as verifying that the server is working correctly, you could look at the Audit Log. Go to Admin->Audit Log and filter by "CONFIG". You should see messages like "New configuration was applied." roughly corresponding to each time you saved/applied. If you filter by "JOBS," you should be able to find messages like "Finished running command: apply_config" at about the same time.

Thank you!

[jaimie.livingston]

I fubar'ed the server and restored it from a previous vm snapshot, and the problem is no longer a problem. Just a mystery.
No need to keep working on this at this time, I suppose.

Thanks for the help and suggestions.

[scottwilkerson]

I fubar'ed the server and restored it from a previous vm snapshot, and the problem is no longer a problem. Just a mystery.
No need to keep working on this at this time, I suppose.

Thanks for the help and suggestions.

Glad it's not a problem any longer

Locking thread