Email Response Command - missing hash data

This board serves as an open discussion and support collaboration point for Nagios XI. NOTE: Nagios XI customers should use the Customer Support forum to obtain expedited support.

Email Response Command - missing hash data

Postby MediaworksNZ » Wed Jun 19, 2019 8:08 pm

HI,

I have configured email sending and receiving and it is working as expected.
I can acknowledge and submit downtime without any issues if they are separate and not parsed as one email.

When I try to send the following reply:
delay
5 hour

I get the following error:
Could not submit command due to missing hash data. Did you reply with the full email?
- Nagios XI System

Not sure what this means.

Also, when I try to send this email response, I also get an error in the logs but no email Response Error:

ack, downtime
now - 11:50
Someone knocked the plug out!

The system will acknowledge and add the comment, but will not apply the scheduled downtime.
Error: External command failed -> SCHEDULE_SVC_DOWNTIME;DAKLBKP1-TEST;Veeam Cloud Connect Service;0;0;0;0;0;myusername;On Thu, 20 Jun 2019 at 11:53, <a valid nagios email address> wrote:

Your help will be much appreciated.
I'm running on the latest version of NagiosXi with Centos 7.
MediaworksNZ
 
Posts: 22
Joined: Tue Aug 05, 2014 5:22 pm

Re: Email Response Command - missing hash data

Postby ssax » Thu Jun 20, 2019 4:18 pm

Please run this command, replicate BOTH errors, and then send us the full output of the tail command once you receive the error:

Code: Select all
tail -Fn0 /var/log/httpd/error_log /var/log/httpd/ssl_error_log /usr/local/nagiosxi/var/cmdsubsys.log /usr/local/nagiosxi/var/eventman.log
Be sure to check out our Knowledgebase for helpful articles and solutions!
User avatar
ssax
Dreams In Code
 
Posts: 4108
Joined: Wed Feb 11, 2015 12:54 pm

Re: Email Response Command - missing hash data

Postby cdienger » Thu Jun 20, 2019 4:38 pm

I would also recommend checking your email client's settings to make sure that it is sending text emails since html can cause problems.

We can also enable some debug logging by editing /usr/local/nagiosxi/html/includes/components/xicore/xicore.inc.php and adding "$debug =1;" around line 520:

Code: Select all
function process_incoming_mail()
{
    $debug = get_option("enable_subsystem_logging", true);
    $debugmsg = "";
        $debug = 1;
    // Check if we are supposed to be processing
    $process = get_option("mail_inbound_process", 0);
    if (!$process) {
        return;
    }


Then tail the cleaner.log while you send replies and we should see some details about how it is getting handled:

Code: Select all
tail -f /usr/local/nagiosxi/var/cleaner.log
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
User avatar
cdienger
Support Tech
 
Posts: 3328
Joined: Tue Feb 07, 2017 11:26 am

Re: Email Response Command - missing hash data

Postby MediaworksNZ » Mon Jun 24, 2019 9:28 pm

Hi,

@ssax, I have PM'd you the logs.
@cdienger I will try your solution in the next couple of days.

Thanks for all your help.
Craig
MediaworksNZ
 
Posts: 22
Joined: Tue Aug 05, 2014 5:22 pm

Re: Email Response Command - missing hash data

Postby MediaworksNZ » Mon Jun 24, 2019 9:51 pm

Right,

Here is the output after the debug change was made.

ack, downtime
now - 14:50
Someone knocked the plug out!
---------------------------------------
delay
1 hour

[root@twnagios3 ~]# tail -f /usr/local/nagiosxi/var/cleaner.log
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors
NUMFOUND: 0
KEEPING ALL ERROR CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagiosxi
NUMFOUND: 20
KEEPING ALL SNAPSHOTS
----------------------------------
Running callbacks:
Processed 0 incoming emails
----------------------------------
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore
NUMFOUND: 20
KEEPING ALL GOOD CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors
ls: cannot access /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors/*.gz: No such file or directory
NUMFOUND: 0
KEEPING ALL ERROR CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagiosxi
NUMFOUND: 20
KEEPING ALL SNAPSHOTS
----------------------------------
Running callbacks:
Processed 0 incoming emails
----------------------------------
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore
NUMFOUND: 20
KEEPING ALL GOOD CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors
ls: cannot access /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors/*.gz: No such file or directory
NUMFOUND: 0
KEEPING ALL ERROR CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagiosxi
NUMFOUND: 20
KEEPING ALL SNAPSHOTS
----------------------------------
Running callbacks:
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore
NUMFOUND: 20
KEEPING ALL GOOD CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors
ls: cannot access /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors/*.gz: No such file or directory
NUMFOUND: 0
KEEPING ALL ERROR CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagiosxi
NUMFOUND: 20
KEEPING ALL SNAPSHOTS
----------------------------------
Running callbacks:
Processed 0 incoming emails
----------------------------------
Error - Could not find a host object in the hash provided
Error - Could not find a host object in the hash provided
Processed 2 incoming emails
----------------------------------
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore
NUMFOUND: 20
KEEPING ALL GOOD CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors
ls: cannot access /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors/*.gz: No such file or directory
NUMFOUND: 0
KEEPING ALL ERROR CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagiosxi
NUMFOUND: 20
KEEPING ALL SNAPSHOTS
----------------------------------
Running callbacks:
Processed 0 incoming emails
----------------------------------
^C
MediaworksNZ
 
Posts: 22
Joined: Tue Aug 05, 2014 5:22 pm

Re: Email Response Command - missing hash data

Postby MediaworksNZ » Mon Jun 24, 2019 10:07 pm

Right,

Here is another one with a bit more details.
It did not create the Scheduled downtime.

-----------------------------------------------------------------

DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore
NUMFOUND: 20
KEEPING ALL GOOD CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors
ls: cannot access /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors/*.gz: No such file or directory
NUMFOUND: 0
KEEPING ALL ERROR CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagiosxi
NUMFOUND: 20
KEEPING ALL SNAPSHOTS
----------------------------------
Running callbacks:
msh_end: 129
msg_lines_count: 7
Array
(
[0] => ack, downtime
[1] => now - 15:30
[2] => Someone knocked the plug out!
[3] =>
[4] => On Tue, 25 Jun 2019 at 15:00, <blankedout> wrote:
[5] =>
[6] => >
)
msg_lines: 1
msg:ack, downtime
now - 15:30
Someone knocked the plug out!

On Tue, 25 Jun 2019 at 15:00, <blankedout> wrote:

>
Array
(
[0] => ack
[1] => downtime
)
Comment line: now - 15:30 Someone knocked the plug out!
ACKNOWLEDGE_SVC_PROBLEM;DAKLSQL1;Memory Usage;1;1;0;my username;now - 15:30 Someone knocked the plug out!
Array
(
[host_name] => DAKLSQL1
[comment_author] => myusername
[service_name] => Memory Usage
[persistent_comment] => 0
[comment_data] => now - 15:30 Someone knocked the plug out!
)
1
Command submitted
Comment line: On Tue, 25 Jun 2019 at 15:00, <blankedout> wrote:
SCHEDULE_SVC_DOWNTIME;DAKLSQL1;Memory Usage;0;0;0;0;0;myusername;On Tue, 25 Jun 2019 at 15:00, <blankedout> wrote:
Array
(
[host_name] => DAKLSQL1
[comment_author] => my username
[service_name] => Memory Usage
[comment_data] => On Tue, 25 Jun 2019 at 15:00, <blankedout> wrote:
)
1
Command submitted
Processed 1 incoming emails




----------------------------------
Running callbacks:
msh_end: 85
msg_lines_count: 6
Array
(
[0] => delay
[1] => 1 hour
[2] =>
[3] => On Tue, 25 Jun 2019 at 15:08, <blankedout> wrote:
[4] =>
[5] => >
)
msg_lines: 1
msg:delay
1 hour

On Tue, 25 Jun 2019 at 15:08, <blankedout> wrote:

>
Array
(
[0] => delay
)
DELAY_SVC_NOTIFICATION;DAKLFIN1;CPU Usage;1561435748
Array
(
[host_name] => DAKLFIN1
[comment_author] => blankedout
[service_name] => CPU Usage
[notification_time] => 1561435748
)
1
Command submitted
Processed 1 incoming emails
----------------------------------
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore
NUMFOUND: 20
KEEPING ALL GOOD CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors
ls: cannot access /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors/*.gz: No such file or directory
NUMFOUND: 0
KEEPING ALL ERROR CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagiosxi
NUMFOUND: 20
KEEPING ALL SNAPSHOTS
----------------------------------
Running callbacks:
Processed 0 incoming emails
----------------------------------
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore
NUMFOUND: 20
KEEPING ALL GOOD CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors
ls: cannot access /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors/*.gz: No such file or directory
NUMFOUND: 0
KEEPING ALL ERROR CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagiosxi
NUMFOUND: 20
KEEPING ALL SNAPSHOTS
----------------------------------
Running callbacks:
msh_end: 109
msg_lines_count: 6
Array
(
[0] => down
[1] => I'm looking into this right now
[2] =>
[3] => On Tue, 25 Jun 2019 at 15:09, <blankedout> wrote:
[4] =>
[5] => >
)
msg_lines: 1
msg:down
I'm looking into this right now

On Tue, 25 Jun 2019 at 15:09, <blankedout> wrote:

>
Array
(
[0] => down
)
SCHEDULE_SVC_DOWNTIME;DAKLSQL1;Memory Usage;0;0;0;0;0;blankedout;
Array
(
[host_name] => DAKLSQL1
[comment_author] => blankedout
[service_name] => Memory Usage
)
1
Command submitted
Processed 1 incoming emails
----------------------------------
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore
NUMFOUND: 20
KEEPING ALL GOOD CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors
ls: cannot access /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors/*.gz: No such file or directory
NUMFOUND: 0
KEEPING ALL ERROR CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagiosxi
NUMFOUND: 20
KEEPING ALL SNAPSHOTS
----------------------------------
Running callbacks:
Processed 0 incoming emails
MediaworksNZ
 
Posts: 22
Joined: Tue Aug 05, 2014 5:22 pm

Re: Email Response Command - missing hash data

Postby cdienger » Tue Jun 25, 2019 4:01 pm

The parsing is off somehow - the timeframe(now - 15:30) gets added to the comment instead of passed to the SCHEDULE_SVC_DOWNTIME command.

Try replying and put an extra line between things. Something like:

Code: Select all
ack, downtime

now - 14:50

Someone knocked the plug out!
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
User avatar
cdienger
Support Tech
 
Posts: 3328
Joined: Tue Feb 07, 2017 11:26 am

Re: Email Response Command - missing hash data

Postby MediaworksNZ » Tue Jun 25, 2019 5:57 pm

Hi @cdienger

Thanks for the suggestion.

I have some mixed results using your suggestion which is interesting.

I'll try to simplify things.
First of, I got the same results when sending replies via plain text or rich text.
I added a new server (not migrated from production nagios) to monitor with two different services to monitor: Memory and CPU
This server is a fake server which would always alert.

This is where it starts to get interesting:
When Acknowledging Memory Usage, Nagios could not find a host object in the hash provided.
Replies was tried with Plain Text and Rich Text
BTW, I was using your suggestion about adding an extra line in.

When Acknowledging CPU Usage, the email reply was executed correctly and the service was ACK and Scheduled as down.
So this proved that the extra line is needed as a tried again without the extra line and it ignored the Schedule timeout.

But this now makes me wonder, what is the difference between these two services that nagios will work with one, but not the other?

----------------------------------------------------------------------
Reply executed as expected from leaner Log
PROBLEM Service Alert - DAKLBKP1-TEST/CPU Usage is CRITICAL

Code: Select all
Running callbacks:
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore
NUMFOUND: 20
KEEPING ALL GOOD CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors
ls: cannot access /usr/local/nagiosxi/nom/checkpoints/nagioscore/errors/*.gz: No such file or directory
NUMFOUND: 0
KEEPING ALL ERROR CHECKPOINTS
DIR: /usr/local/nagiosxi/nom/checkpoints/nagiosxi
NUMFOUND: 20
KEEPING ALL SNAPSHOTS
----------------------------------
Running callbacks:
msh_end: 126
msg_lines_count: 9
Array
(
    [0] => ack, downtime
    [1] =>
    [2] => now - 14:50
    [3] =>
    [4] => Replied with Rich Text
    [5] =>
    [6] => On Wed, 26 Jun 2019 at 10:30, <****> wrote:
    [7] =>
    [8] => >
)
msg_lines: 1
msg:ack, downtime

now - 14:50

Replied with Rich Text

On Wed, 26 Jun 2019 at 10:30, <****> wrote:

>
Array
(
    [0] => ack
    [1] => downtime
)
ACKNOWLEDGE_SVC_PROBLEM;DAKLBKP1-TEST;CPU Usage;1;1;0;****;Problem was acknowledged by email response
Array
(
    [host_name] => DAKLBKP1-TEST
    [comment_author] => ****
    [service_name] => CPU Usage
    [persistent_comment] => 0
    [comment_data] => Problem was acknowledged by email response
)
1
Command submitted
SCHEDULE_SVC_DOWNTIME;DAKLBKP1-TEST;CPU Usage;1561501868;1561517400;1;0;15532;****;Downtime scheduled by email response
Array
(
    [host_name] => DAKLBKP1-TEST
    [comment_author] => ****
    [service_name] => CPU Usage
    [start_time] => 1561501868
    [end_time] => 1561517400
    [duration] => 15532
    [fixed] => 1
    [comment_data] => Downtime scheduled by email response
)
1
Command submitted
Processed 1 incoming emails



Reply NOT executed as expected from leaner Log
PROBLEM Service Alert - DAKLBKP1-TEST/CPU Usage is CRITICAL
This was a reply without the extra line spaces which did not work.

Code: Select all
Running callbacks:
msh_end: 129
msg_lines_count: 7
Array
(
    [0] => ack, downtime
    [1] => now - 14:50
    [2] => Replied with Plain Text Text
    [3] =>
    [4] => On Wed, 26 Jun 2019 at 10:33, <***> wrote:
    [5] => >
    [6] => >
)
msg_lines: 1
msg:ack, downtime
now - 14:50
Replied with Plain Text Text

On Wed, 26 Jun 2019 at 10:33, <****> wrote:
>
>
Array
(
    [0] => ack
    [1] => downtime
)
Comment line: now - 14:50 Replied with Plain Text Text
ACKNOWLEDGE_SVC_PROBLEM;DAKLBKP1-TEST;CPU Usage;1;1;0;***;now - 14:50 Replied with Plain Text Text
Array
(
    [host_name] => DAKLBKP1-TEST
    [comment_author] => ****
    [service_name] => CPU Usage
    [persistent_comment] => 0
    [comment_data] => now - 14:50 Replied with Plain Text Text
)
1
Command submitted
PHP Notice:  Undefined offset: 7 in /usr/local/nagiosxi/html/includes/components/xicore/xicore.inc.php on line 763
Comment line: On Wed, 26 Jun 2019 at 10:33, <****> wrote: > >
SCHEDULE_SVC_DOWNTIME;DAKLBKP1-TEST;CPU Usage;0;0;0;0;0;****;On Wed, 26 Jun 2019 at 10:33, <nagios@mediaworks.co.nz> wrote: > >
Array
(
    [host_name] => DAKLBKP1-TEST
    [comment_author] => ****
    [service_name] => CPU Usage
    [comment_data] => On Wed, 26 Jun 2019 at 10:33, <****> wrote: > >
)
1
Command submitted
Processed 1 incoming emails
MediaworksNZ
 
Posts: 22
Joined: Tue Aug 05, 2014 5:22 pm

Re: Email Response Command - missing hash data

Postby jomann » Wed Jun 26, 2019 4:25 pm

There was a bug fixed for 5.6.4 which relates to the downtime scheduling when you are submitting two different commands. I believe that is what you are hitting below. Essentially you get 0's in the schedule downtime command because it already read the data for the acknowledgement. The update should be coming out soon. If you'd like to try a patch I can give you the fixed file to try.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
User avatar
jomann
Development Lead
 
Posts: 589
Joined: Mon Apr 22, 2013 10:06 am
Location: Nagios Enterprises

Re: Email Response Command - missing hash data

Postby MediaworksNZ » Thu Jun 27, 2019 5:05 pm

Hi @jomann.

That would be great if you could send me the file.
I'm reaching the deadline to get this server into production and this would get me one step closer.

Thanks.
MediaworksNZ
 
Posts: 22
Joined: Tue Aug 05, 2014 5:22 pm

Next

Return to Nagios XI

Who is online

Users browsing this forum: No registered users and 11 guests