Inbound Mail hash errors: "^M" in cleaner.log

[benjaminsmith]

Hi,

I have an update on this issue, we've been able to repeat this here with Outlook and a bug report has been filed. It's a formatting issue of some kind and we should have this accounted for in the next maintenance release of Nagios XI.

[gdksc]

Hi, @benjaminsmith,

Sounds good. Thanks for the update.

[benjaminsmith]

HI @gdksc,

Your welcome!

[benjaminsmith]

Hi,

If you're interested in testing this out, I have a patch from development to resolve this issue. Open up the following file on the Nagios XI Server:

Code: Select all

/usr/local/nagiosxi/html/includes/components/xicore/xicore.inc.php

Then find line 587 and replace it with the following line:

Code: Select all

$msg = substr($email['body'], 0, $msg_end); with $msg = preg_replace("/[\\r\\n]+/", "\n", substr($email['body'], 0, $msg_end))

If you have time to changes this, let me know how it goes.

Thanks!
Benjamin

[gdksc]

Hi, @benjaminsmith,

I made the change to line 587, but still no luck.

Here's diff of the change I made and tested:

Code: Select all

# diff xicore.inc.php.orig xicore.inc.php
587c587
<             $msg = substr($email['body'], 0, $msg_end);
---
>             $msg = preg_replace("/[\\r\\n]+/", "\n", substr($email['body'], 0, $msg_end));

Here's sample result from cleaner.log:

Code: Select all

Running callbacks:

Error - Could not find a host object in the hash provided
Processed 1 incoming emails
----------------------------------

Given that error message, I'm not sure I'm getting to line 587.

Seems to fail the check starting at line 562:

Code: Select all

     
                  // Verify that at least a hostname is present
                  if (empty($obj->host)) {
                      echo "Error - Could not find a host object in the hash provided\n";
                      $error = true;
                  }

Something interesting, of note...

Previously, I'd added a line for debugging, around 563, to echo the email body:

Code: Select all

echo $email['body'];

That was where I was seeing the ^M in the hash, in the cleaner.log:

Code: Select all

-----^M
^M
To reply, keep the following data in your response email.^M
^M
##YTk2M2IzNmQ5NGRjMDU3YW1GVm5vNlNmdURYMzVHU0UxWHd1R0RweVpQdUJNTXliTy9EYThPe=^M
XVJQ0R4djRkVWxmbW5TLzY0ZSt0THppL1FuK21EOGVEZVF6eG40emJEQzZ5WmpRPT0=3D##^M
^M
Error - Could not find a host object in the hash provided
Processed 1 incoming emails

However, I've now removed the "echo email body" line replaced it with the following, starting at 561:

Code: Select all

               $hash = trim(str_replace(array('> ', '>', '=', "\n", "\t", "\r"), '', substr($email['body'], $start + 2, $end - $start - 2)));
               $obj = json_decode(decrypt_data($hash));
               echo "hash: " . $hash . "\n";
               echo "\$obj = json_decode(decrypt_data($hash))\n";
               $error = json_last_error();
               echo "json_last_error: " . $error . "\n";
               var_dump($obj);

Following are sample results from cleaner.log, after adding those lines:

Code: Select all

----------------------------------
Running callbacks:
hash: OTc2ZTAzMDgzNTI5ZmEyOTY5SHNRT3dSRi9wb012bTVaZEtLUjRqZG4rSVRlem8vS3hjT1pRaVpENkZWWXRmQnI1MG9lRHhqcmIrbEs1VzZReFBFN1lja1dHL2xsemg5NENObTVRPT03D
$obj = json_decode(decrypt_data(OTc2ZTAzMDgzNTI5ZmEyOTY5SHNRT3dSRi9wb012bTVaZEtLUjRqZG4rSVRlem8vS3hjT1pRaVpENkZWWXRmQnI1MG9lRHhqcmIrbEs1VzZReFBFN1lja1dHL2xsemg5NENObTVRPT03D))
json_last_error: 0
NULL
Error - Could not find a host object in the hash provided
Processed 1 incoming emails
----------------------------------

So, it seems the hash does not have the ^M, but the decode is not finding the host object, for some reason.

Additionally -- and this goes back to something observed early on -- all works OK for a host down alert, where the only noticeable difference is the # of characters in the hash.

A snip from the cleaner.log, for a host ack which has a smaller # of chars in the hash:

Code: Select all

----------------------------------
Running callbacks:
hash: YzdjNjFmNDNhOGUzMGM1ZUI5QkZ2TjVXN3U4MlYvQXUyWjhkRzBxQnkxRFNENTBySnZ6NTBCSmJXNmM9
$obj = json_decode(decrypt_data(YzdjNjFmNDNhOGUzMGM1ZUI5QkZ2TjVXN3U4MlYvQXUyWjhkRzBxQnkxRFNENTBySnZ6NTBCSmJXNmM9))
json_last_error: 0
object(stdClass)#8 (1) {
  ["host"]=>
  string(5) "dummy"
}
...

Is there any additional debugging we could add for original line 560, for the json_decode ?

Code: Select all

 $obj = json_decode(decrypt_data($hash));

To recap, here is the current state of my xicore.inc.php vs the original:

Code: Select all

# diff xicore.inc.php.orig xicore.inc.php
560a561,565
>                 echo "hash: " . $hash . "\n";
>                 echo "\$obj = json_decode(decrypt_data($hash))\n";
>                 $error = json_last_error();
>                 echo "json_last_error: " . $error . "\n";
>                 var_dump($obj);
587c592
<             $msg = substr($email['body'], 0, $msg_end);
---
>             $msg = preg_replace("/[\\r\\n]+/", "\n", substr($email['body'], 0, $msg_end));

Thanks again for your help with this.

[benjaminsmith]

Hi,

Thanks for testing this out and providing feedback, this is really helpful. I'm going to share this with the developer here and I'll follow up with you again.

Benjamin

[jomann]

Hello, I've been testing this out internally with my own Outlook account and can you please try the following?

Can you please replace the line:

Code: Select all

$msg = preg_replace("/[\\r\\n]+/", "\n", substr($email['body'], 0, $msg_end));

with the original line:

Code: Select all

$msg = substr($email['body'], 0, $msg_end);

I think what needs to happen is we need to convert all Windows line endings to Linux for the hash to properly verify. Can you try adding the following code:

Code: Select all

$email['body'] = preg_replace("/[\\r\\n]+/", "\n",$email['body']);

right before the line:

Code: Select all

$start = strpos($email['body'], '##');

[gdksc]

Hi, @jomann.

I added the line, so that it's the only change from the original file:

Code: Select all

# diff xicore.inc.php.orig xicore.inc.php
554a555
>             $email['body'] = preg_replace("/[\\r\\n]+/", "\n",$email['body']);

For a first test, I sent a reply email with an "ack" to an alert (e.g. Disk Usage on /var), which included the hash line:

Code: Select all

-----

To reply, keep the following data in your response email.

##M2Q1NGFhYmU1MDcyNzdiNlNxY051VjhVUFhVL0xYZWhyOGlSaGJLOVE5S05JUGFWQ3dPZi91enhiYk11MkZWSzBOYTZLWVpnaHRjZy9iOG94dndVQ3JBcmdXcXBjM1dhVUdpUG9nPT0=##

Unfortunately, I still received the error in the cleaner.log:

Code: Select all

----------------------------------
Running callbacks:
Error - Could not find a host object in the hash provided
Processed 1 incoming emails
----------------------------------

For a second test, I sent a reply email with an "ack" to a different alert (e.g. Host is Up), which included the hash line:

Code: Select all

-----

To reply, keep the following data in your response email.

##Nzk1OGZkNWZlM2RmMzY3MUFoMUdLTVYrdm9XZ0pYcFJzc0FkNEtmSlpKVDE5a1ltZVNqRDRwb2lZeFU9##

And the hash was decoded, per the cleaner.log:

Code: Select all

----------------------------------
Running callbacks:
msh_end: 386
msg_lines_count: 15
Array
(
    [0] => ack
...

For some additional tests, I re-added some "echo" statements to xicore.inc.php:

Code: Select all

# diff xicore.inc.php.orig xicore.inc.php
554a555,556
>             $email['body'] = preg_replace("/[\\r\\n]+/", "\n",$email['body']);
>             echo $email['body'];
560a563,569
>
>                  echo "hash: " . $hash . "\n";
>                  echo "\$obj = json_decode(decrypt_data($hash))\n";
>                  $error = json_last_error();
>                  echo "json_last_error: " . $error . "\n";
>                  var_dump($obj);
>

Then I re-sent the two "ack" reply emails.

From looking at the cleaner.log, ^M are being stripped from the email body, as expected. I no longer see the ^M in the cleaner.log, when I echo the email body.

However, I'm still having trouble decoding this hash:

Code: Select all

hash: M2Q1NGFhYmU1MDcyNzdiNlNxY051VjhVUFhVL0xYZWhyOGlSaGJLOVE5S05JUGFWQ3dPZi91enhiYk11MkZWSzBOYTZLWVpnaHRjZy9iOG94dndVQ3JBcmdXcXBjM1dhVUdpUG9nPT03D
$obj = json_decode(decrypt_data(M2Q1NGFhYmU1MDcyNzdiNlNxY051VjhVUFhVL0xYZWhyOGlSaGJLOVE5S05JUGFWQ3dPZi91enhiYk11MkZWSzBOYTZLWVpnaHRjZy9iOG94dndVQ3JBcmdXcXBjM1dhVUdpUG9nPT03D))
json_last_error: 0
NULL
Error - Could not find a host object in the hash provided
Processed 1 incoming emails
----------------------------------

While not having trouble decoding this hash:

Code: Select all

hash: Nzk1OGZkNWZlM2RmMzY3MUFoMUdLTVYrdm9XZ0pYcFJzc0FkNEtmSlpKVDE5a1ltZVNqRDRwb2lZeFU9
$obj = json_decode(decrypt_data(Nzk1OGZkNWZlM2RmMzY3MUFoMUdLTVYrdm9XZ0pYcFJzc0FkNEtmSlpKVDE5a1ltZVNqRDRwb2lZeFU9))
json_last_error: 0
object(stdClass)#8 (1) {
  ["host"]=>
  string(14) "fbv-nagios-d01"
}
msh_end: 386
msg_lines_count: 15
Array
(
    [0] => ack
...

Let me know if any additional debugging suggestions...

Thanks again for your help.

[jomann]

The first hash you are testing was changed to PT03D instead of PT0= which would definitely cause problems. There shouldn't be any numbers or letters after the = in the encrypted object data hash. In the example above the one you just posted, it shows the hash having the value TPO=3D inside the hash portion, which shouldn't happen when the object is encrypted.

Can you try decoding with the = still in the hash?

You can also try adjusting the line that creates the hash with str_replace to remove the '=' as a replaceable character. This line,

Code: Select all

$hash = trim(str_replace(array('> ', '>', '=', "\n", "\t", "\r"), '', substr($email['body'], $start + 2, $end - $start - 2)));

to this line instead

Code: Select all

$hash = trim(str_replace(array('> ', '>', "\n", "\t", "\r"), '', substr($email['body'], $start + 2, $end - $start - 2)));

[gdksc]

Very interesting...

The "=" is ASCII "3D".

I tested changing the hash in my reply email to be simply five "=", like this:

Code: Select all

-----

To reply, keep the following data in your response email.

##=====##

And here's what I received in the cleaner.log, given I still have the "echo" statements in there:

Code: Select all

-----
To reply, keep the following data in your response email.
##=3D=3D=3D=3D=3D##
PHP Warning:  openssl_decrypt(): IV passed is only 7 bytes long, cipher expects an IV of precisely 16 bytes, padding with \0 in /usr/local/nagiosxi/html/includes/utilsl-helpers.inc.php on line 0
hash: 3D3D3D3D3D
$obj = json_decode(decrypt_data(3D3D3D3D3D))
json_last_error: 0
NULL
Error - Could not find a host object in the hash provided
Processed 1 incoming emails
----------------------------------

Something seems to be converting the "=" in the hash to "3D"...