Email Response Command - missing hash data

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
MediaworksNZ
Posts: 29
Joined: Tue Aug 05, 2014 5:22 pm

Email Response Command - missing hash data

Post by MediaworksNZ »

HI,

I have configured email sending and receiving and it is working as expected.
I can acknowledge and submit downtime without any issues if they are separate and not parsed as one email.

When I try to send the following reply:
delay
5 hour

I get the following error:
Could not submit command due to missing hash data. Did you reply with the full email?
- Nagios XI System

Not sure what this means.

Also, when I try to send this email response, I also get an error in the logs but no email Response Error:

ack, downtime
now - 11:50
Someone knocked the plug out!

The system will acknowledge and add the comment, but will not apply the scheduled downtime.
Error: External command failed -> SCHEDULE_SVC_DOWNTIME;DAKLBKP1-TEST;Veeam Cloud Connect Service;0;0;0;0;0;myusername;On Thu, 20 Jun 2019 at 11:53, <a valid nagios email address> wrote:

Your help will be much appreciated.
I'm running on the latest version of NagiosXi with Centos 7.
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: Email Response Command - missing hash data

Post by ssax »

Please run this command, replicate BOTH errors, and then send us the full output of the tail command once you receive the error:

Code: Select all

tail -Fn0 /var/log/httpd/error_log /var/log/httpd/ssl_error_log /usr/local/nagiosxi/var/cmdsubsys.log /usr/local/nagiosxi/var/eventman.log
User avatar
cdienger
Support Tech
Posts: 5045
Joined: Tue Feb 07, 2017 11:26 am

Re: Email Response Command - missing hash data

Post by cdienger »

I would also recommend checking your email client's settings to make sure that it is sending text emails since html can cause problems.

We can also enable some debug logging by editing /usr/local/nagiosxi/html/includes/components/xicore/xicore.inc.php and adding "$debug =1;" around line 520:

Code: Select all

function process_incoming_mail()
{
    $debug = get_option("enable_subsystem_logging", true);
    $debugmsg = "";
        $debug = 1;
    // Check if we are supposed to be processing
    $process = get_option("mail_inbound_process", 0);
    if (!$process) {
        return;
    }
Then tail the cleaner.log while you send replies and we should see some details about how it is getting handled:

Code: Select all

tail -f /usr/local/nagiosxi/var/cleaner.log
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
MediaworksNZ
Posts: 29
Joined: Tue Aug 05, 2014 5:22 pm

Re: Email Response Command - missing hash data

Post by MediaworksNZ »

Hi,

@ssax, I have PM'd you the logs.
@cdienger I will try your solution in the next couple of days.

Thanks for all your help.
Craig
MediaworksNZ
Posts: 29
Joined: Tue Aug 05, 2014 5:22 pm

Re: Email Response Command - missing hash data

Post by MediaworksNZ »

Right,

Here is the output after the debug change was made.

ack, downtime
now - 14:50
Someone knocked the plug out!
---------------------------------------
delay
1 hour

[root@twnagios3 ~]# tail -f /usr/local/nagiosxi/var/cleaner.log
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors
NUMFOUND: 0
KEEPING ALL ERROR CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagiosxi
NUMFOUND: 20
KEEPING ALL SNAPSHOTS
----------------------------------
Running callbacks:
Processed 0 incoming emails
----------------------------------
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore
NUMFOUND: 20
KEEPING ALL GOOD CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors
ls: cannot access /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors/*.gz: No such file or directory
NUMFOUND: 0
KEEPING ALL ERROR CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagiosxi
NUMFOUND: 20
KEEPING ALL SNAPSHOTS
----------------------------------
Running callbacks:
Processed 0 incoming emails
----------------------------------
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore
NUMFOUND: 20
KEEPING ALL GOOD CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors
ls: cannot access /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors/*.gz: No such file or directory
NUMFOUND: 0
KEEPING ALL ERROR CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagiosxi
NUMFOUND: 20
KEEPING ALL SNAPSHOTS
----------------------------------
Running callbacks:
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore
NUMFOUND: 20
KEEPING ALL GOOD CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors
ls: cannot access /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors/*.gz: No such file or directory
NUMFOUND: 0
KEEPING ALL ERROR CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagiosxi
NUMFOUND: 20
KEEPING ALL SNAPSHOTS
----------------------------------
Running callbacks:
Processed 0 incoming emails
----------------------------------
Error - Could not find a host object in the hash provided
Error - Could not find a host object in the hash provided
Processed 2 incoming emails
----------------------------------
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore
NUMFOUND: 20
KEEPING ALL GOOD CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors
ls: cannot access /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors/*.gz: No such file or directory
NUMFOUND: 0
KEEPING ALL ERROR CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagiosxi
NUMFOUND: 20
KEEPING ALL SNAPSHOTS
----------------------------------
Running callbacks:
Processed 0 incoming emails
----------------------------------
^C
MediaworksNZ
Posts: 29
Joined: Tue Aug 05, 2014 5:22 pm

Re: Email Response Command - missing hash data

Post by MediaworksNZ »

Right,

Here is another one with a bit more details.
It did not create the Scheduled downtime.

-----------------------------------------------------------------

DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore
NUMFOUND: 20
KEEPING ALL GOOD CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors
ls: cannot access /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors/*.gz: No such file or directory
NUMFOUND: 0
KEEPING ALL ERROR CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagiosxi
NUMFOUND: 20
KEEPING ALL SNAPSHOTS
----------------------------------
Running callbacks:
msh_end: 129
msg_lines_count: 7
Array
(
[0] => ack, downtime
[1] => now - 15:30
[2] => Someone knocked the plug out!
[3] =>
[4] => On Tue, 25 Jun 2019 at 15:00, <blankedout> wrote:
[5] =>
[6] => >
)
msg_lines: 1
msg:ack, downtime
now - 15:30
Someone knocked the plug out!

On Tue, 25 Jun 2019 at 15:00, <blankedout> wrote:

>
Array
(
[0] => ack
[1] => downtime
)
Comment line: now - 15:30 Someone knocked the plug out!
ACKNOWLEDGE_SVC_PROBLEM;DAKLSQL1;Memory Usage;1;1;0;my username;now - 15:30 Someone knocked the plug out!
Array
(
[host_name] => DAKLSQL1
[comment_author] => myusername
[service_name] => Memory Usage
[persistent_comment] => 0
[comment_data] => now - 15:30 Someone knocked the plug out!
)
1
Command submitted
Comment line: On Tue, 25 Jun 2019 at 15:00, <blankedout> wrote:
SCHEDULE_SVC_DOWNTIME;DAKLSQL1;Memory Usage;0;0;0;0;0;myusername;On Tue, 25 Jun 2019 at 15:00, <blankedout> wrote:
Array
(
[host_name] => DAKLSQL1
[comment_author] => my username
[service_name] => Memory Usage
[comment_data] => On Tue, 25 Jun 2019 at 15:00, <blankedout> wrote:
)
1
Command submitted
Processed 1 incoming emails




----------------------------------
Running callbacks:
msh_end: 85
msg_lines_count: 6
Array
(
[0] => delay
[1] => 1 hour
[2] =>
[3] => On Tue, 25 Jun 2019 at 15:08, <blankedout> wrote:
[4] =>
[5] => >
)
msg_lines: 1
msg:delay
1 hour

On Tue, 25 Jun 2019 at 15:08, <blankedout> wrote:

>
Array
(
[0] => delay
)
DELAY_SVC_NOTIFICATION;DAKLFIN1;CPU Usage;1561435748
Array
(
[host_name] => DAKLFIN1
[comment_author] => blankedout
[service_name] => CPU Usage
[notification_time] => 1561435748
)
1
Command submitted
Processed 1 incoming emails
----------------------------------
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore
NUMFOUND: 20
KEEPING ALL GOOD CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors
ls: cannot access /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors/*.gz: No such file or directory
NUMFOUND: 0
KEEPING ALL ERROR CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagiosxi
NUMFOUND: 20
KEEPING ALL SNAPSHOTS
----------------------------------
Running callbacks:
Processed 0 incoming emails
----------------------------------
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore
NUMFOUND: 20
KEEPING ALL GOOD CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors
ls: cannot access /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors/*.gz: No such file or directory
NUMFOUND: 0
KEEPING ALL ERROR CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagiosxi
NUMFOUND: 20
KEEPING ALL SNAPSHOTS
----------------------------------
Running callbacks:
msh_end: 109
msg_lines_count: 6
Array
(
[0] => down
[1] => I'm looking into this right now
[2] =>
[3] => On Tue, 25 Jun 2019 at 15:09, <blankedout> wrote:
[4] =>
[5] => >
)
msg_lines: 1
msg:down
I'm looking into this right now

On Tue, 25 Jun 2019 at 15:09, <blankedout> wrote:

>
Array
(
[0] => down
)
SCHEDULE_SVC_DOWNTIME;DAKLSQL1;Memory Usage;0;0;0;0;0;blankedout;
Array
(
[host_name] => DAKLSQL1
[comment_author] => blankedout
[service_name] => Memory Usage
)
1
Command submitted
Processed 1 incoming emails
----------------------------------
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore
NUMFOUND: 20
KEEPING ALL GOOD CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors
ls: cannot access /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors/*.gz: No such file or directory
NUMFOUND: 0
KEEPING ALL ERROR CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagiosxi
NUMFOUND: 20
KEEPING ALL SNAPSHOTS
----------------------------------
Running callbacks:
Processed 0 incoming emails
User avatar
cdienger
Support Tech
Posts: 5045
Joined: Tue Feb 07, 2017 11:26 am

Re: Email Response Command - missing hash data

Post by cdienger »

The parsing is off somehow - the timeframe(now - 15:30) gets added to the comment instead of passed to the SCHEDULE_SVC_DOWNTIME command.

Try replying and put an extra line between things. Something like:

Code: Select all

ack, downtime

now - 14:50

Someone knocked the plug out!
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
MediaworksNZ
Posts: 29
Joined: Tue Aug 05, 2014 5:22 pm

Re: Email Response Command - missing hash data

Post by MediaworksNZ »

Hi @cdienger

Thanks for the suggestion.

I have some mixed results using your suggestion which is interesting.

I'll try to simplify things.
First of, I got the same results when sending replies via plain text or rich text.
I added a new server (not migrated from production nagios) to monitor with two different services to monitor: Memory and CPU
This server is a fake server which would always alert.

This is where it starts to get interesting:
When Acknowledging Memory Usage, Nagios could not find a host object in the hash provided.
Replies was tried with Plain Text and Rich Text
BTW, I was using your suggestion about adding an extra line in.

When Acknowledging CPU Usage, the email reply was executed correctly and the service was ACK and Scheduled as down.
So this proved that the extra line is needed as a tried again without the extra line and it ignored the Schedule timeout.

But this now makes me wonder, what is the difference between these two services that nagios will work with one, but not the other?

----------------------------------------------------------------------
Reply executed as expected from leaner Log
PROBLEM Service Alert - DAKLBKP1-TEST/CPU Usage is CRITICAL

Code: Select all

Running callbacks:
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore
NUMFOUND: 20
KEEPING ALL GOOD CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors
ls: cannot access /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors/*.gz: No such file or directory
NUMFOUND: 0
KEEPING ALL ERROR CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagiosxi
NUMFOUND: 20
KEEPING ALL SNAPSHOTS
----------------------------------
Running callbacks:
msh_end: 126
msg_lines_count: 9
Array
(
    [0] => ack, downtime
    [1] =>
    [2] => now - 14:50
    [3] =>
    [4] => Replied with Rich Text
    [5] =>
    [6] => On Wed, 26 Jun 2019 at 10:30, <****> wrote:
    [7] =>
    [8] => >
)
msg_lines: 1
msg:ack, downtime

now - 14:50

Replied with Rich Text

On Wed, 26 Jun 2019 at 10:30, <****> wrote:

>
Array
(
    [0] => ack
    [1] => downtime
)
ACKNOWLEDGE_SVC_PROBLEM;DAKLBKP1-TEST;CPU Usage;1;1;0;****;Problem was acknowledged by email response
Array
(
    [host_name] => DAKLBKP1-TEST
    [comment_author] => ****
    [service_name] => CPU Usage
    [persistent_comment] => 0
    [comment_data] => Problem was acknowledged by email response
)
1
Command submitted
SCHEDULE_SVC_DOWNTIME;DAKLBKP1-TEST;CPU Usage;1561501868;1561517400;1;0;15532;****;Downtime scheduled by email response
Array
(
    [host_name] => DAKLBKP1-TEST
    [comment_author] => ****
    [service_name] => CPU Usage
    [start_time] => 1561501868
    [end_time] => 1561517400
    [duration] => 15532
    [fixed] => 1
    [comment_data] => Downtime scheduled by email response
)
1
Command submitted
Processed 1 incoming emails

Reply NOT executed as expected from leaner Log
PROBLEM Service Alert - DAKLBKP1-TEST/CPU Usage is CRITICAL
This was a reply without the extra line spaces which did not work.

Code: Select all

Running callbacks:
msh_end: 129
msg_lines_count: 7
Array
(
    [0] => ack, downtime
    [1] => now - 14:50
    [2] => Replied with Plain Text Text
    [3] =>
    [4] => On Wed, 26 Jun 2019 at 10:33, <***> wrote:
    [5] => >
    [6] => >
)
msg_lines: 1
msg:ack, downtime
now - 14:50
Replied with Plain Text Text

On Wed, 26 Jun 2019 at 10:33, <****> wrote:
>
>
Array
(
    [0] => ack
    [1] => downtime
)
Comment line: now - 14:50 Replied with Plain Text Text
ACKNOWLEDGE_SVC_PROBLEM;DAKLBKP1-TEST;CPU Usage;1;1;0;***;now - 14:50 Replied with Plain Text Text
Array
(
    [host_name] => DAKLBKP1-TEST
    [comment_author] => ****
    [service_name] => CPU Usage
    [persistent_comment] => 0
    [comment_data] => now - 14:50 Replied with Plain Text Text
)
1
Command submitted
PHP Notice:  Undefined offset: 7 in /usr/local/nagiosxi/html/includes/components/xicore/xicore.inc.php on line 763
Comment line: On Wed, 26 Jun 2019 at 10:33, <****> wrote: > >
SCHEDULE_SVC_DOWNTIME;DAKLBKP1-TEST;CPU Usage;0;0;0;0;0;****;On Wed, 26 Jun 2019 at 10:33, <nagios@mediaworks.co.nz> wrote: > >
Array
(
    [host_name] => DAKLBKP1-TEST
    [comment_author] => ****
    [service_name] => CPU Usage
    [comment_data] => On Wed, 26 Jun 2019 at 10:33, <****> wrote: > >
)
1
Command submitted
Processed 1 incoming emails
jomann
Development Lead
Posts: 611
Joined: Mon Apr 22, 2013 10:06 am
Location: Nagios Enterprises

Re: Email Response Command - missing hash data

Post by jomann »

There was a bug fixed for 5.6.4 which relates to the downtime scheduling when you are submitting two different commands. I believe that is what you are hitting below. Essentially you get 0's in the schedule downtime command because it already read the data for the acknowledgement. The update should be coming out soon. If you'd like to try a patch I can give you the fixed file to try.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
MediaworksNZ
Posts: 29
Joined: Tue Aug 05, 2014 5:22 pm

Re: Email Response Command - missing hash data

Post by MediaworksNZ »

Hi @jomann.

That would be great if you could send me the file.
I'm reaching the deadline to get this server into production and this would get me one step closer.

Thanks.
Locked