System Status stays in the "processing" state

[itheodoridis]

Hello.
In our installation of Nagios Log Server I had to move the vm across datacenters and rename it. The ip address remained the same.
After some initial problems, with a lot of CPU usage (it remained over 80%) I deleted old files and increased the memory of the vm to 6GΒ of Ram from the initial of 4GB.

After some time the stability came back. Every page on the admin tab seems fine but the system status page remains in a state as if processing to check Elastic Search and Logstash services and never produces a result.(See picture NLS Bad state, I have erased the IP and hostname for confidentiality issues).

NLSbadstate.JPG

I have also included a pic showing what the page should show, taken from another NLS installation we have with a free license (See NLS good state

NLSgoodstate.JPG

)

I tried searching for a solution but didn't come up with much. I found a similar post that seemed to be about the server being able to resolve its ip to its hostname, so I solved that issue but it didn't change the outcome.

Do you have any suggestions? I am obviously concerned if everything is working correctly but apart from logs being gathered I would feel much better if the server responded correctly about it's state.

Let me also say that the server has gone through consecutive updates since version 1.4.0 up to 2.0.4. We had it licensed when it was at version 2.0.2 (i think).
Let me know if I can provide any input to clarify the situation further.
Thank you.
Best Regards,
Ioannis Theodoridis
Network & Systems Engineer
Bank of Greece

[cdienger]

You can verify the status by running the following on the command line of the machine:

service elasticsearch status
service logstash status

most of the time you'll see that they are actually up and this behavior is usually caused by the client being unable to make a connection to the machine to get the status. You mentioned fixing the dns issue -can you explain a bit more? Is your client machine(the one opening the browser) able to resolve the host name seen in the drop down? It is also important that the IP addresses can resolve back into hostnames. Make sure that "nslookup <machine_ip>" returns the hostname - do this on your client and also the NSL command line.

It could also be related certificates - are the machines configured to use https?

[scottwilkerson]

Do you have multiple instances in this cluster?
If you select a different instance do you get the same result?

Also, from your workstation, can you reach the IP of the instance you are looking at listed in the drop down? Does it list a hostname, and if so it will be trying to use the hostname to make the connection, make sure it resolves correctly from your workstation.

[itheodoridis]

You can verify the status by running the following on the command line of the machine:

service elasticsearch status
service logstash status

most of the time you'll see that they are actually up and this behavior is usually caused by the client being unable to make a connection to the machine to get the status. You mentioned fixing the dns issue -can you explain a bit more? Is your client machine(the one opening the browser) able to resolve the host name seen in the drop down? It is also important that the IP addresses can resolve back into hostnames. Make sure that "nslookup <machine_ip>" returns the hostname - do this on your client and also the NSL command line.

It could also be related certificates - are the machines configured to use https?

Hello, there is only one instance, although I tried adding a hostname to the cluster hostname field to see if that would make things better. It made no difference. Since there is only one instance and I am getting back a UI, I have access to the system. Being able to resolve it on the client machine? I will try to fix that but it's not that easy. I will let you know. I don't expect it to make a difference as I am accessing it by ip address but I will try it nevertheless.
What I meant in my message is that the NLS itself can resolve its own hostname. it's taken care of in the upstream DNS server. I can't point my client machine to that server because I am using DHCP and can't add that server to the main site's servers or even forwarders (it's a different domain).
The cli commands run fine and services are up. I am getting logs and can see them in the dashboard. I just can't get a good answer in system status.

[itheodoridis]

Do you have multiple instances in this cluster?
If you select a different instance do you get the same result?

Also, from your workstation, can you reach the IP of the instance you are looking at listed in the drop down? Does it list a hostname, and if so it will be trying to use the hostname to make the connection, make sure it resolves correctly from your workstation.

I followed both of your advices and I stand amazed! I managed to shortcircuit my dns settings and be able to correctly resolve both the ip to a hostname and vice versa from my workstation and finally saw the status. But what does that mean? Sure I can now stop and restart services from the UI but does the logserver itself function correctly? How can I make sure?
There is just one server, just one instance. I put in a hostname in the cluster hostname in global settings when trying to make things work.

[cdienger]

The best way would be to go to the Home or Dashboards screen and see if anything is coming in.

[itheodoridis]

Logs are coming in normally.

[cdienger]

Glad to hear are we good to lock this thread up?