Page 1 of 2
Defaults missing - again
Posted: Thu May 14, 2015 7:40 pm
by Fred Kroeger
Well I filled up the root disk so decided to Change the data store path to a couple of separate disks as per the Nagios Doc.
And had the same problems as last time (see my previous post about "Defaults missing" ). After I changed the data store path in the config file, the elasticsearch service starts but then dies.
I eventually resolved it by leaving the original path and then adding the new paths. At least the elasticsearch service now stays running. I then was able to remove the original deafult path.
However I could no longer login with my account or the nagiosadmin account - same as I did last time.
I used the command that I was given last time to recreate the nagiosadmin account & password - so now I can login as nagiosadmin.
But once again *all* the user accounst I created are missing, as well as all the default dashboards & queries and all the ones I had created.
My solution last time was to rebuild both logservers but I would rather solve this problem now as I can't afford to keep rebuilding these VMs
The original install was working fine & I have upgraded in April to NLS 2015R1.4
Regards... Fred
Re: Defaults missing - again
Posted: Fri May 15, 2015 10:51 am
by jolson
What is the path that you're trying to use for your new datastore? Can you please give us an exact list of the commands that you went through to get to this state?
It's possible that there's a bug here, but many people have changed their datastore locations without issue.
What kind of storage medium are you using, and where is it mounted? Are the permissions read/writeable by the 'nagios' user?
If these are network disks, what protocol are you using to share them?
Please answer the above questions, I'd like to get this hammered out if there is something wrong with our document/procedure.
Re: Defaults missing - again
Posted: Sun May 17, 2015 6:44 pm
by Fred Kroeger
Hi
NagiosLS is a VM and the two logstore disks are both virtual disks mounted as a UNIX filesystem with the correct ownership/permissions.
My greatest concern is why have I lost the user accounts and all the dashboards again. What is the relationship between these and the elasticsearch paths?
regards... Fred
Code: Select all
# mount
rootfs on / type rootfs (rw)
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
devtmpfs on /dev type devtmpfs (rw,relatime,size=1018428k,nr_inodes=254607,mode=755)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /dev/shm type tmpfs (rw,relatime)
/dev/sda1 on / type ext4 (rw,noatime,barrier=1,data=ordered)
/proc/bus/usb on /proc/bus/usb type usbfs (rw,relatime)
/dev/mapper/VolGroup01-LogVol00 on /logstore-1 type ext4 (rw,relatime,barrier=1,data=ordered)
/dev/mapper/VolGroup02-LogVol00 on /logstore-2 type ext4 (rw,relatime,barrier=1,data=ordered)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
# ls -ld /logstore-1/data
drwxr-xr-x 3 nagios users 4096 May 15 16:13 /logstore-1/data
# ls -ld /logstore-2/data
drwxr-xr-x 3 nagios users 4096 May 15 16:13 /logstore-2/data
[root@bdcvlog042 ~]#
Re: Defaults missing - again
Posted: Mon May 18, 2015 12:50 pm
by jolson
I'm curious why you have two datastores mounted per node:
Code: Select all
mount /logstore-1
mount /logstore-2
Are you using the 'software-raid' like functionality of elasticsearch?
For clarity, each node will need an independent datastore - the nodes cannot use the
same repository for their own storage. For instance, you have Node A and Node B.
Node A uses a datastore at /logstore-1 which connects to nfs.example.local's /logstore-1 exported share.
Node B uses a datastore at /logstore-1 which connects to the same nfs.example.local /logstore-1 share.
There will be problems with the above config - each node needs its own seperate place to store data, as the data is going to be different between nodes. If I'm completely off-base here, let me know.
Properly:
Node A:/logstore-1 will use nfs.example.local:/logstore-1
Node B:/logstore-1 will use nfs.example.local:/logstore-2
My greatest concern is why have I lost the user accounts and all the dashboards again. What is the relationship between these and the elasticsearch paths?
This is something I'll need to look into.
Let me know. Thanks!
Re: Defaults missing - again
Posted: Mon May 18, 2015 9:30 pm
by Fred Kroeger
Each datastore is a separate virtual disk. They are not being shared across VMs. So each VM has it's own /logstore-1 & /logstore-2 .
The rationale behind using two datastore paths are because they are virtual disks therefore slower than physical disk. Technically, the Virtual disks can be located on different ESX datastores to improve I/O.
My understanding from the doco is that they are separate paths and the data is written to the least used path - this isn't really "software raid" ?
I ended up deleting all the data files in the elasticsearch disks and re-started clean. So far it is working, however I still have the issue where all my user accounts and default dashboards/querires no longer exist - plus the nagiosadmin account also being deleted.
regards... Fred
Re: Defaults missing - again
Posted: Tue May 19, 2015 3:31 pm
by tmcdonald
Please run the following as root and post the (sanitized) results in code tags:
Code: Select all
curl -XGET 'http://localhost:9200/nagioslogserver/_search?type=user&pretty'
It will contain password information, which will be a hash but still should be sanitized.
Re: Defaults missing - again
Posted: Tue May 19, 2015 5:53 pm
by Fred Kroeger
Code: Select all
[1]# curl -XGET 'http://localhost:9200/nagioslogserver/_search?type=user&pretty'
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 1.0,
"hits" : [ {
"_index" : "nagioslogserver",
"_type" : "user",
"_id" : "2",
"_score" : 1.0,
"_source":{"username":"nagiosadmin","password":"c678bcf3b5138b9263a95c44d28097f22c2e02877193d2c25313478821d45c19","auth_type":"admin","email":"[email protected]","language":"default","apiaccess":"1","apikey":"1396e08757545557073844695e5b64caa0bd3ad3","created":"2015-01-23 10:00:00","created_by":0,"default_dashboard":"/dashboard/elasticsearch/default"}
} ]
}
}
Code: Select all
[2]# curl -XGET 'http://localhost:9200/nagioslogserver/_search?type=user&pretty'
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 1.0,
"hits" : [ {
"_index" : "nagioslogserver",
"_type" : "user",
"_id" : "2",
"_score" : 1.0,
"_source":{"username":"nagiosadmin","password":"c678bcf3b5138b9263a95c44d28097f22c2e02877193d2c25313478821d45c19","auth_type":"admin","email":"[email protected]","language":"default","apiaccess":"1","apikey":"1396e08757545557073844695e5b64caa0bd3ad3","created":"2015-01-23 10:00:00","created_by":0,"default_dashboard":"/dashboard/elasticsearch/default"}
} ]
}
}
Re: Defaults missing - again
Posted: Wed May 20, 2015 9:19 am
by jolson
Is the nagiosadmin account accessible? How many users do you expect to see?
I ended up deleting all the data files in the elasticsearch disks and re-started clean
Which data files did you delete?
Your data directory should contain the following information:
Code: Select all
[root@localhost ~]# ls -l /usr/local/nagioslogserver/elasticsearch/data/clusterUUID/nodes/0/indices/
drwxrwxr-x 8 nagios nagios 4096 Apr 6 15:04 kibana-int
drwxr-xr-x 8 nagios users 4096 May 18 14:49 logstash-2015.05.18
drwxr-xr-x 8 nagios users 4096 May 19 00:00 logstash-2015.05.19
drwxr-xr-x 8 nagios users 4096 May 20 00:00 logstash-2015.05.20
drwxrwxr-x 4 nagios nagios 4096 Apr 6 15:04 nagioslogserver
drwxrwxr-x 8 nagios nagios 4096 Apr 6 15:06 nagioslogserver_log
Please get us a long listing of your current working data directories. The kibana-int database stores dashboards, and the nagioslogserver database stores queries/users.
Code: Select all
ls -l /logstore-1/data
ls -l /logstore-2/data
Re: Defaults missing - again
Posted: Wed May 20, 2015 8:08 pm
by Fred Kroeger
Well , you're explanation is making things a bit clearer. I deleted *everything* under /logstore-[1|2]/data as I was expecting that *only* elasticsearch files would be stored there.
So that explains why the useraccounts and dashboard/queries disappeared.
Tracking back further, I think I found why my initial move of the datapath failed. I had a typo in the directory name I entered in the config file - so when NLS started, it couldn't find the folder and failed.
So don't waste anymore time on checking this issue ... sorry...
The datastore paths now look exactly as you describe with the kibana-int & nagioslogserver folders as well as the logstash folders.
So now the question is how can I restore the default dashboards/queries again without rebuilding the server. Would applying the latest update add those again or would I need to remove some files/folders to restore the default database?
regards... Fred
Re: Defaults missing - again
Posted: Thu May 21, 2015 9:59 am
by jolson
So now the question is how can I restore the default dashboards/queries again without rebuilding the server.
Nagios Log Server takes backups daily by default, so if we're lucky your information was stored in a backup at some point. These backups are located at /store/backups/nagioslogserver.
To perform a restore, use the following commands:
Code: Select all
cd /usr/local/nagioslogserver/scripts
./restore_backup /store/backups/nagioslogserver/<backupname>.tar.gz