Nagios Support Forum

Support for Nagios products and services
https://support.nagios.com/forum/

Settings on the Backup and Maintenance are not staying

https://support.nagios.com/forum/viewtopic.php?t=45085

Page 1 of 1

Settings on the Backup and Maintenance are not staying

Posted: Wed Aug 09, 2017 9:30 am
by kseasolutions
When I am making changes to the settings on the Backup/Maintenance interface on Nagios Logging server 1.4.4 When I click Save it does not stay persistent. It goes back to the default settings. In addition bloom filter cache is missing from the user interface as well.

Any help would be great.

Re: Settings on the Backup and Maintenance are not staying

Posted: Wed Aug 09, 2017 2:33 pm
by cdienger
Sounds like a possible issue with the nagioslogserver index. Please run the following and post the output:

Code: Select all

curl -XGET 'http://localhost:9200/_cluster/health/*?level=shards&pretty'
I'd also like you run:

Code: Select all

tail -f /var/log/elasticsearch/<CLUSTER_ID>.log
Where the CLUSTER_ID can be found by looking at the second line of output from the previous command. Let this run while you try to write and save the changes again in the web UI. Provide anything that may be generated on the cli when you do.

Re: Settings on the Backup and Maintenance are not staying

Posted: Wed Aug 09, 2017 2:44 pm
by kseasolutions
curl -XGET 'http://localhost:9200/_cluster/health/* ... rds&pretty'

Code: Select all

{
  "cluster_name" : "redacted----",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 25,
  "active_shards" : 25,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 25,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "indices" : {
    "nagioslogserver_log" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5,
      "shards" : {
        "0" : {
          "status" : "yellow",
          "primary_active" : true,
          "active_shards" : 1,
          "relocating_shards" : 0,
          "initializing_shards" : 0,
          "unassigned_shards" : 1
        },
        "1" : {
          "status" : "yellow",
          "primary_active" : true,
          "active_shards" : 1,
          "relocating_shards" : 0,
          "initializing_shards" : 0,
          "unassigned_shards" : 1
        },
        "2" : {
          "status" : "yellow",
          "primary_active" : true,
          "active_shards" : 1,
          "relocating_shards" : 0,
          "initializing_shards" : 0,
          "unassigned_shards" : 1
        },
        "3" : {
          "status" : "yellow",
          "primary_active" : true,
          "active_shards" : 1,
          "relocating_shards" : 0,
          "initializing_shards" : 0,
          "unassigned_shards" : 1
        },
        "4" : {
          "status" : "yellow",
          "primary_active" : true,
          "active_shards" : 1,
          "relocating_shards" : 0,
          "initializing_shards" : 0,
          "unassigned_shards" : 1
        }
      }
    },
    "nagioslogserver" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5,
      "shards" : {
        "0" : {
          "status" : "yellow",
          "primary_active" : true,
          "active_shards" : 1,
          "relocating_shards" : 0,
          "initializing_shards" : 0,
          "unassigned_shards" : 1
        },
        "1" : {
          "status" : "yellow",
          "primary_active" : true,
          "active_shards" : 1,
          "relocating_shards" : 0,
          "initializing_shards" : 0,
          "unassigned_shards" : 1
        },
        "2" : {
          "status" : "yellow",
          "primary_active" : true,
          "active_shards" : 1,
          "relocating_shards" : 0,
          "initializing_shards" : 0,
          "unassigned_shards" : 1
        },
        "3" : {
          "status" : "yellow",
          "primary_active" : true,
          "active_shards" : 1,
          "relocating_shards" : 0,
          "initializing_shards" : 0,
          "unassigned_shards" : 1
        },
        "4" : {
          "status" : "yellow",
          "primary_active" : true,
          "active_shards" : 1,
          "relocating_shards" : 0,
          "initializing_shards" : 0,
          "unassigned_shards" : 1
        }
      }
    },
    "kibana-int" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5,
      "shards" : {
        "0" : {
          "status" : "yellow",
          "primary_active" : true,
          "active_shards" : 1,
          "relocating_shards" : 0,
          "initializing_shards" : 0,
          "unassigned_shards" : 1
        },
        "1" : {
          "status" : "yellow",
          "primary_active" : true,
          "active_shards" : 1,
          "relocating_shards" : 0,
          "initializing_shards" : 0,
          "unassigned_shards" : 1
        },
        "2" : {
          "status" : "yellow",
          "primary_active" : true,
          "active_shards" : 1,
          "relocating_shards" : 0,
          "initializing_shards" : 0,
          "unassigned_shards" : 1
        },
        "3" : {
          "status" : "yellow",
          "primary_active" : true,
          "active_shards" : 1,
          "relocating_shards" : 0,
          "initializing_shards" : 0,
          "unassigned_shards" : 1
        },
        "4" : {
          "status" : "yellow",
          "primary_active" : true,
          "active_shards" : 1,
          "relocating_shards" : 0,
          "initializing_shards" : 0,
          "unassigned_shards" : 1
        }
      }
    },
    "logstash-2017.08.08" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5,
      "shards" : {
        "0" : {
          "status" : "yellow",
          "primary_active" : true,
          "active_shards" : 1,
          "relocating_shards" : 0,
          "initializing_shards" : 0,
          "unassigned_shards" : 1
        },
        "1" : {
          "status" : "yellow",
          "primary_active" : true,
          "active_shards" : 1,
          "relocating_shards" : 0,
          "initializing_shards" : 0,
          "unassigned_shards" : 1
        },
        "2" : {
          "status" : "yellow",
          "primary_active" : true,
          "active_shards" : 1,
          "relocating_shards" : 0,
          "initializing_shards" : 0,
          "unassigned_shards" : 1
        },
        "3" : {
          "status" : "yellow",
          "primary_active" : true,
          "active_shards" : 1,
          "relocating_shards" : 0,
          "initializing_shards" : 0,
          "unassigned_shards" : 1
        },
        "4" : {
          "status" : "yellow",
          "primary_active" : true,
          "active_shards" : 1,
          "relocating_shards" : 0,
          "initializing_shards" : 0,
          "unassigned_shards" : 1
        }
      }
    },
    "logstash-2017.08.09" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5,
      "shards" : {
        "0" : {
          "status" : "yellow",
          "primary_active" : true,
          "active_shards" : 1,
          "relocating_shards" : 0,
          "initializing_shards" : 0,
          "unassigned_shards" : 1
        },
        "1" : {
          "status" : "yellow",
          "primary_active" : true,
          "active_shards" : 1,
          "relocating_shards" : 0,
          "initializing_shards" : 0,
          "unassigned_shards" : 1
        },
        "2" : {
          "status" : "yellow",
          "primary_active" : true,
          "active_shards" : 1,
          "relocating_shards" : 0,
          "initializing_shards" : 0,
          "unassigned_shards" : 1
        },
        "3" : {
          "status" : "yellow",
          "primary_active" : true,
          "active_shards" : 1,
          "relocating_shards" : 0,
          "initializing_shards" : 0,
          "unassigned_shards" : 1
        },
        "4" : {
          "status" : "yellow",
          "primary_active" : true,
          "active_shards" : 1,
          "relocating_shards" : 0,
          "initializing_shards" : 0,
          "unassigned_shards" : 1
        }
      }
    }
  }
}

Re: Settings on the Backup and Maintenance are not staying

Posted: Wed Aug 09, 2017 2:49 pm
by kseasolutions
From the tail I can see the error fairly clear

Code: Select all

[2017-08-09 15:45:18,741][DEBUG][action.index             ] [--redact--] [nagioslogserver][1], node[--redact--], [P], s[STARTED]: [color=#FF0000]Failed to execute [index {[nagioslogserver][cf_option][maintenance_settings], source[{"created":"2017-01-03 22:55:13","created_by":0,"value":"a:6:{s:13:\"optimize_time\";s:1:\"5\";s:10:\"close_time\";s:1:\"7\";s:11:\"delete_time\";s:2:\"30\";s:10:\"repository\";s:15:\"Archive_indexes\";s:20:\"delete_snapshot_time\";s:2:\"90\";s:6:\"active\";s:1:\"1\";}","modified_by":"AVz2hp3mHpAy0Akc-9b_"}]}][/color]
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [modified_by]
        at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:411)
        at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:706)
        at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:497)
        at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:544)
        at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493)
        at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:493)
        at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:192)
        at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574)
        at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.routeRequestOrPerformLocally(TransportShardReplicationOperationAction.java:444)
        at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.doRun(TransportShardReplicationOperationAction.java:370)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
        at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction.doExecute(TransportShardReplicationOperationAction.java:112)
        at org.elasticsearch.action.index.TransportIndexAction.innerExecute(TransportIndexAction.java:136)
        at org.elasticsearch.action.index.TransportIndexAction.doExecute(TransportIndexAction.java:114)
        at org.elasticsearch.action.index.TransportIndexAction.doExecute(TransportIndexAction.java:63)
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
        at org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:217)
        at org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:170)
        at org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$AsyncSingleAction$1.run(TransportInstanceSingleOperationAction.java:187)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NumberFormatException: For input string: "AVz2hp3mHpAy0Akc-9b_"
        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
        at java.lang.Long.parseLong(Long.java:441)
        at java.lang.Long.parseLong(Long.java:483)
        at org.elasticsearch.common.xcontent.support.AbstractXContentParser.longValue(AbstractXContentParser.java:145)
        at org.elasticsearch.index.mapper.core.LongFieldMapper.innerParseCreateField(LongFieldMapper.java:288)
        at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:239)
        at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:401)
        ... 21 more

Re: Settings on the Backup and Maintenance are not staying

Posted: Thu Aug 10, 2017 12:08 pm
by mcapra
I'm betting modified_by and created_by are both typed as longs rather than strings. I believe this is a known issue within NLS, though I don't have a reference number for the bug.

Can you share the full output of the following command executed from the CLI of one of your Nagios Log Server machines:

Code: Select all

curl -XGET 'http://localhost:9200/nagioslogserver/_mapping/'
Please wrap it in

Code: Select all

 tags as the output will be quite long.

Also, which account are you making these changes from? I'm betting the default nagiosadmin account is able to make the changes without issue.

Re: Settings on the Backup and Maintenance are not staying

Posted: Thu Aug 10, 2017 12:39 pm
by kseasolutions
curl -XGET 'http://localhost:9200/nagioslogserver/_mapping/'

Code: Select all

{"nagioslogserver":{"mappings":{"user":{"properties":{"apiaccess":{"type":"string"},"apikey":{"type":"string"},"auth_settings":{"properties":{"ad_username":{"type":"string"},"auth_server_id":{"type":"string"},"dn":{"type":"string"},"type":{"type":"string"}}},"auth_type":{"type":"string"},"created":{"type":"string"},"created_by":{"type":"long"},"default_dashboard":{"type":"string"},"email":{"type":"string"},"lang":{"type":"string"},"language":{"type":"string"},"name":{"type":"string"},"password":{"type":"string"},"username":{"type":"string"}}},"node":{"properties":{"address":{"type":"string"},"config_filters":{"properties":{"active":{"type":"long"},"name":{"type":"string"},"raw":{"type":"string"}}},"config_inputs":{"properties":{"active":{"type":"long"},"name":{"type":"string"},"raw":{"type":"string"}}},"elasticsearch":{"properties":{"message":{"type":"string"},"pid":{"type":"string"},"status":{"type":"string"}}},"hostname":{"type":"string"},"last_updated":{"type":"long"},"logstash":{"properties":{"message":{"type":"string"},"pid":{"type":"string"},"status":{"type":"string"}}},"ls_release":{"type":"long"},"ls_version":{"type":"string"}}},"snapshot":{"properties":{"auto":{"type":"long"},"clean_filename":{"type":"string"},"created":{"type":"long"},"filename":{"type":"string"},"name":{"type":"string"},"path":{"type":"string"}}},"cf_option":{"properties":{"created":{"type":"string"},"created_by":{"type":"long"},"modified_by":{"type":"long"},"value":{"type":"string"}}},"commands":{"properties":{"active":{"type":"long"},"args":{"properties":{"path":{"type":"string"},"sh_created":{"type":"long"},"sh_id":{"type":"string"},"timezone":{"type":"string"}}},"command":{"type":"string"},"created":{"type":"string"},"created_by":{"type":"string"},"frequency":{"type":"string"},"last_run_output":{"type":"string"},"last_run_status":{"type":"string"},"last_run_time":{"type":"string"},"node":{"type":"string"},"run_time":{"type":"long"},"status":{"type":"string"},"type":{"type":"string"}}},"query":{"properties":{"created_by":{"type":"string"},"created_id":{"type":"string"},"edit_type":{"type":"string"},"imported":{"type":"long"},"name":{"type":"string"},"raw":{"type":"string"},"services":{"type":"string"},"show_everyone":{"type":"long"},"title":{"type":"string"}}}}}}

Re: Settings on the Backup and Maintenance are not staying

Posted: Thu Aug 10, 2017 12:41 pm
by mcapra
Yup:

Code: Select all

"cf_option": {
				"properties": {
					"created": {
						"type": "string"
					},
					"created_by": {
						"type": "long"
					},
					"modified_by": {
						"type": "long"
					},
					"value": {
						"type": "string"
					}
				}
			}
Have you tried making the changes by using the nagiosadmin user? I'll defer to a tech in terms of a fix, since I don't recall specifically what the "official" fix was.

Re: Settings on the Backup and Maintenance are not staying

Posted: Thu Aug 10, 2017 12:48 pm
by kseasolutions
I just made the changes as the official Nagiosadmin user and they stuck this time (though bloom is still not an option)

Previously, I was in an admin account, but not nagiosadmin.

Re: Settings on the Backup and Maintenance are not staying

Posted: Thu Aug 10, 2017 12:55 pm
by kseasolutions
Thank you for the post @mcapra. I will wait to see what the technical fix is on this.

Thank you once again.

Re: Settings on the Backup and Maintenance are not staying

Posted: Thu Aug 10, 2017 4:10 pm
by scottwilkerson
The bloom filter has been removed in the current version as it is no longer used.

This is a remnant of a previous bug, to fix the mapping problem you will want to do the following

create a new system backup

Code: Select all

/usr/local/nagioslogserver/scripts/create_backup.sh
The restore this backup
find the newest file file

Code: Select all

ls -l /store/backups/nagioslogserver/
use that to restore the backup

Code: Select all

/usr/local/nagioslogserver/scripts/restore_backup.sh /store/backups/nagioslogserver/nagioslogserver.2017-08-10.XXXXXXX.tar.gz
All times are UTC-05:00
Page 1 of 1

Powered by phpBB® Forum Software © phpBB Limited