Nagios user java command using over 200% CPU

[rferebee]

@cdienger I PM'd you with an updated system profile

Elasticsearch died this morning due to a memory commit error. Appreciate your assistance.

[cdienger]

The system is running with the maximum amount of memory of which elasticsearch will automatically take half. This halving can actually still result in an allocation for elasticsearch that causes memory performance problems. Let's try setting this manually to rule out this problem. Edit /etc/sysconfig/elasticsearch and change line 7 from:

ES_HEAP_SIZE=$(expr $(free -m|awk '/^Mem:/{print $2}') / 2 )m

to:

ES_HEAP_SIZE=3100m

and then restart elasticsearch:

service elasticsearch restart

[rferebee]

Ok, I made the requested change. I'll keep you posted.

[cdienger]

Sounds good. I'll keep my fingers crossed.

[rferebee]

I'm unsure why the two would be related, but since making that change yesterday I am unable to login to the console. It spins for a few minutes and then tells me my username does not exist.

I've rebooted the cluster to no avail. It still won't let me login.

[cdienger]

Are you using the nagiosadmin account or another one? Can you test with the nagiosadmin account? You can also try creating a new admin account just to see if that can get in by running:

[user]curl -XPUT 'http://localhost:9200/nagioslogserver/user/99' -d '{"username":"someuser","password":"c678bcf3b5138b9263a95c44d28097f22c2e02877193d2c25313478821d45c19","auth_type":"admin","email":"[email protected]","language":"default","apiaccess":"1","apikey":"1396e08757545557073844695e5b64caa0bd3ad3","created":"2015-01-23 10:00:00","created_by":0,"default_dashboard":"/dashboard/elasticsearch/default"}'[/user]

this creates a user called "someuser" with a password of "nagiosls".

Check the status of the indices with:

curl 'localhost:9200/_cat/indices?pretty'

[rferebee]

I tried my LDAP account as well as the nagiosadmin account. Both responding with username doesn't exist.

Unfortunately, I had to roll back this change since this is a production system. I don't think checking the indices would help with anything at this point.

[cdienger]

Thanks for the update.

And regarding the profile for future reference, in cases where you can't login to the web ui, you can run "/usr/local/nagioslogserver/scripts/profile.sh" from the command line which will create system-profile.tar.gz.

[rferebee]

Will the Enterprise version of nxlog work with Log Server or can we only use the CE version?

[cdienger]

The enterprise edition should work as well.