Notifications stopped - Event Manager red - PHPMailer.php Fatal error

[cdueck]

We've stopped receiving XI notifications -- other event types seem to still work.
The Event Manager 'System Component Status' is red.

Event Manager is running, per eventman.log, but every run contains another copy of the same notification message, ending with the below PHP error.
No other events get processed.

We've already dealt with the particular service issue. How can I flush this event from the queue to allow the rest of the events to process?

Version: XI 2024R1.1.1

Code: Select all

PHP Fatal error:  Uncaught ValueError: mail(): Argument #3 ($message) must not contain any null bytes in /usr/local/nagiosxi/html/includes/components/oauth2/vendor/phpmailer/phpmailer/src/PHPMailer.php:877
Stack trace:
#0 /usr/local/nagiosxi/html/includes/components/oauth2/vendor/phpmailer/phpmailer/src/PHPMailer.php(877): mail()
#1 /usr/local/nagiosxi/html/includes/components/oauth2/vendor/phpmailer/phpmailer/src/PHPMailer.php(1962): PHPMailer\PHPMailer\PHPMailer->mailPassthru()
#2 /usr/local/nagiosxi/html/includes/components/oauth2/vendor/phpmailer/phpmailer/src/PHPMailer.php(1683): PHPMailer\PHPMailer\PHPMailer->mailSend()
#3 /usr/local/nagiosxi/html/includes/components/oauth2/vendor/phpmailer/phpmailer/src/PHPMailer.php(1519): PHPMailer\PHPMailer\PHPMailer->postSend()
#4 /usr/local/nagiosxi/html/includes/utils-email.inc.php(445): PHPMailer\PHPMailer\PHPMailer->send()
#5 /usr/local/nagiosxi/html/includes/components/xicore/xicore.inc.php(325): send_email()
#6 /usr/local/nagiosxi/html/includes/components/xicore/xicore.inc.php(57): xicore_handle_notification_event()
#7 /usr/local/nagiosxi/html/includes/utilsl.inc.php(3610): xicore_eventhandler()
#8 /usr/local/nagiosxi/cron/eventman.php(171): do_callbacks()
#9 /usr/local/nagiosxi/cron/eventman.php(133): process_event()
#10 /usr/local/nagiosxi/cron/eventman.php(110): process_event_record()
#11 /usr/local/nagiosxi/cron/eventman.php(59): process_events()
#12 /usr/local/nagiosxi/cron/eventman.php(22): do_eventman_jobs()
#13 {main}
  thrown in /usr/local/nagiosxi/html/includes/components/oauth2/vendor/phpmailer/phpmailer/src/PHPMailer.php on line 877

[cdueck]

for what it's worth, I've:

• Run mysqlcheck and everything is OK (unlike viewtopic.php?p=357811)
• Found select count(1) from xi_events; returns thousands of events, but it seems like the problematic events get a new timestamp each time they try to process, with no other data I can discern to help identify them.

[nagios-dnelson]

@cduek

If you need to resolve this issue quickly I would recommend opening a ticket with our support department as they have experience with this type of issue. You can access the Customer Support Portal at https://support.nagios.com/ and open a ticket.

[cdueck]

Thanks. I was able to resolve this myself.

eventman.log contained the event_id of the troublesome event. And, as there were more events in the queue (one per contact) still with their original timestamp, I fetched the DateTime from the message text.

With these two values, I could find all the relevant alerts:

Code: Select all

mysq> select * from xi_events where event_id = XXXX or event_time = '2025-10-14 XX:XX:XX';

And then purge the events:

Code: Select all

mysq> DELETE from xi_events where event_id = XXXX or event_time = '2025-10-14 XX:XX:XX';

Would be nice if the Event Manager was more fault tolorant -- at least processing the rest of the queue instead of terminating.

Also, it's notable that the cause was an NRPE plugin returning \'0\' within the output.
XI web interface displayed this as \\'0\\', which is still fine.
But, as visible in eventman.log, Event Manager/PHPMailer strips the quotation marks before unescaping the backslashes, resulting in a \0 (null) character.
Oof.