Nagios Support Forum

jolson wrote:

If you guys have any other ideas in the meantime let me know thanks.

Until the memory is added, I can't think of anything off of the top of my head. Thanks for the detailed information, as always.

The fact that your jobs system is backing up on the CA server is concerning to me. I understand that your CA server has more alerts than the US server - do the alerts also look back over large periods of time? It's possible that the system is backing up because long lookups cost a decent amount of server time.

Part of me is tempted to just blow up this cluster and make a new one and reload all the alerts into it.

I cant see anything different but we will see what the additional memory will do.

jolson wrote:

Part of me is tempted to just blow up this cluster and make a new one and reload all the alerts into it.

I'm tempted to agree with you since the slowness is causing problems. I can give you exact instructions on how to approach this in the most sane way possible if you're interested.

If there a way to export them and import them into a different instance that would be ideal.

cd /usr/local/nagioslogserver/scripts
./restore_backup.sh /store/backups/nagioslogserver/nagioslogserver.2015-10-27.1445979677.tar.gz

Also on a side note do you guys have an eta for a hotfix or something for the alert skipping issue we are experiencing? My management has pretty much put this project on hold until this is resolved. Thank you!

I spoke with a developer, and have word that the fix will likely be out within the month.

jolson wrote:This hot fix is not available yet - I will get you an updated roadmap, but the developers have left for the day. I will update this thread tomorrow with the relevant information. Thanks!

Alert 1
{
  "_index": "nagioslogserver_log",
  "_type": "ALERT",
  "_id": "AVDzc4FlCzVc0qk_F3xY",
  "_score": null,
  "_source": {
    "created": 1447193444708,
    "type": "ALERT",
    "message": "Alert Name 159401 CANADA: Work Process Service - Error   alert, Priority 3/2 returned CRITICAL: 1 matching entries found |logs=1;0;0",
    "source": "Nagios Log Server"
  },
  "sort": [
    1447193444708,
    1447193444708
  ]
}

Alert 2
{
  "_index": "nagioslogserver_log",
  "_type": "ALERT",
  "_id": "AVDzOPPFCzVc0qk_F3Ic",
  "_score": null,
  "_source": {
    "created": 1447189607365,
    "type": "ALERT",
    "message": "Alert Name 106929 CANADA: StdAssignmentDeliveryWF Error -   - Priority 3/2 returned CRITICAL: 1 matching entries found |logs=1;0;0",
    "source": "Nagios Log Server"
  },
  "sort": [
    1447189607365,
    1447189607365
  ]
}

Jklre wrote:I also found something odd looking through log files /var/log/logstash/logstash.log has been ballooning. Today's log is 37000KB as opposed to the US 4000KB.

not sure if this would be related though. They all are messages that state ":message=>"failed action with response of 400, dropping action:" and a whole few pages more details per line.