check_log is not collecting queried string

[kn00567386]

Hello,
I am using check_log to query string "AMQP server on 127.0.0.1:5671 is unreachable". But it is giving output other that queried string. What should be wrong. Please help. I am suspecting it is taking each word from string and showing output. How to fix this.

Below is command used for it.

---> /usr/local/nagios/libexec/check_nrpe -H <IP> -t 30 -c check_log -a '-F /var/log/neutron/server.log -O /usr/lib64/nagios/plugins/500_log.txt -q 'AMQP server on 127.0.0.1:5671 is unreachable''
(4) < 2021-05-11 15:49:55.325 12342 DEBUG amqp [req-34a805d4-1b5f-4210-be2a-157cd8148a64 - - - - -] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2016 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@ifra04-pvc01', 'platform': 'Erlang/OTP', 'version': '3.6.6'}, mechanisms: ['PLAIN', 'AMQPLAIN'], locales: [u'en_US'] _on_start /usr/lib/python2.7/site-packages/amqp/connection.py:369|match=4;;;0

[ssax]

The output is likely getting truncated so you're not seeing all 4 entries or the entirety of the messages.

What version of the remote NRPE agent are you running?

Code: Select all

/usr/local/nagios/libexec/check_nrpe -H X.X.X.X

[kn00567386]

Please refer agent version below.
/usr/local/nagios/libexec/check_nrpe -H 192.168.4.136
NRPE v4.0.3


My worried point is why its not reporting only for queried string "AMQP server on 127.0.0.1:5671 is unreachable".

I dont see this string is occur and displayed in output. is it matching each word/letter for query string.

[kn00567386]

The output is likely getting truncated so you're not seeing all 4 entries or the entirety of the messages.

What version of the remote NRPE agent are you running?

Code: Select all

/usr/local/nagios/libexec/check_nrpe -H X.X.X.X

Please refer agent version below.
/usr/local/nagios/libexec/check_nrpe -H 192.168.4.136
NRPE v4.0.3


My worried point is why its not reporting only for queried string "AMQP server on 127.0.0.1:5671 is unreachable".

I dont see this string is occur and displayed in output. is it matching each word/letter for query string.

[benjaminsmith]

Hi,

I'm helping Sean out with this topic as he is out today.

Can you run the plugin locally (on the remote host) as the nagios user. If it's working locally, then this likely an issue with passing arguments.

Please note, the plugin only scans for new entries, it may be necessary to manually add the "AMQP server on 127.0.0.1:5671 is unreachable" to the end of the log for testing purposes.

Regards,
Benjamin

[kn00567386]

Strange thing is we don't see query string neither in source log file or old log file. Still we do see alerts for it.

[root@- ~]# grep “AMQP server on 127.0.0.1:5671 is unreachable” /var/log/neutron/server.log
[root@- ~]# grep “AMQP server on 127.0.0.1:5671 is unreachable” /usr/lib64/nagios/plugins/500_log.txt

[benjaminsmith]

Hi kn00567386,

That's very odd. When you run this locally, without the string in the file, are you getting valid output? Can you upload the nrpe.cfg file from the remote system, I'd like to see the command definition.

Thanks,
Benjamin

[kn00567386]

find the attached nrpe.cfg file.I tried to run it frequently on locally and i dont see it unwated error.

[root@-~]# for i in 1 2 3 4 5 6 7 8 9 10
> do
> /usr/lib64/nagios/plugins/check_log -F /var/log/neutron/server.log -O /usr/lib64/nagios/plugins/500_log.txt -q ‘AMQP server on^C27.0.0.1:5671 is unreachable’
> done
Log check ok - 0 pattern matches found|match=0;;;0
Log check ok - 0 pattern matches found|match=0;;;0
Log check ok - 0 pattern matches found|match=0;;;0
Log check ok - 0 pattern matches found|match=0;;;0
Log check ok - 0 pattern matches found|match=0;;;0
Log check ok - 0 pattern matches found|match=0;;;0
Log check ok - 0 pattern matches found|match=0;;;0
Log check ok - 0 pattern matches found|match=0;;;0
Log check ok - 0 pattern matches found|match=0;;;0
Log check ok - 0 pattern matches found|match=0;;;0

[benjaminsmith]

Hi @kn00567386,

A couple of items we noticed. One is that when running the plugin locally you are logged in as root, try logging in as the nagios user account, su - nagios, it's possible that this user cannot write an output file /usr/lib64/nagios/plugins/500_log.txt. Also, try using another file just to clear this out and start fresh.

Next, when running the check command from the XI server, enclose the query in double quotes and the whole argument in single quotes. For example:

Code: Select all

-a '-F /var/log/neutron/server.log -O /usr/lib64/nagios/plugins/500_log.txt -q "AMQP server on 127.0.0.1:5671 is unreachable"'

Let us know if that resolves the error.
Thanks,
Benjamin

[ssax]

Please run these commands as root on the system and send the full output:

Code: Select all

touch /tmp/testing.txt
/usr/local/nagios/libexec/check_log -F /tmp/testing.txt -O /tmp/testing.txt.log -q 'AMQP server on 127.0.0.1:5671 is unreachable'
echo 'test' >> /tmp/testing.txt
/usr/local/nagios/libexec/check_log -F /tmp/testing.txt -O /tmp/testing.txt.log -q 'AMQP server on 127.0.0.1:5671 is unreachable'
echo 'AMQP server on 127.0.0.1:5671 is unreachable' >> /tmp/testing.txt
/usr/local/nagios/libexec/check_log -F /tmp/testing.txt -O /tmp/testing.txt.log -q 'AMQP server on 127.0.0.1:5671 is unreachable'

It should look like this:

Code: Select all

[root@c77 ~]# touch /tmp/testing.txt
[root@c77 ~]# /usr/local/nagios/libexec/check_log -F /tmp/testing.txt -O /tmp/testing.txt.log -q 'AMQP server on 127.0.0.1:5671 is unreachable'
Log check data initialized...
[root@c77 ~]# echo 'test' >> /tmp/testing.txt
[root@c77 ~]# /usr/local/nagios/libexec/check_log -F /tmp/testing.txt -O /tmp/testing.txt.log -q 'AMQP server on 127.0.0.1:5671 is unreachable'
Log check ok - 0 pattern matches found|match=0;;;0
[root@c77 ~]# echo 'AMQP server on 127.0.0.1:5671 is unreachable' >> /tmp/testing.txt
[root@c77 ~]# /usr/local/nagios/libexec/check_log -F /tmp/testing.txt -O /tmp/testing.txt.log -q 'AMQP server on 127.0.0.1:5671 is unreachable'
(1) < AMQP server on 127.0.0.1:5671 is unreachable|match=1;;;0

What I was saying before is that this is not the entirely of that log messages:

Code: Select all

2021-05-11 15:49:55.325 12342 DEBUG amqp [req-34a805d4-1b5f-4210-be2a-157cd8148a64 - - - - -] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2016 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@ifra04-pvc01', 'platform': 'Erlang/OTP', 'version': '3.6.6'}, mechanisms: ['PLAIN', 'AMQPLAIN'], locales: [u'en_US'] _on_start /usr/lib/python2.7/site-packages/amqp/connection.py:369

There is likely more lines in that log and it's being cutoff, can you find that log entry in the log file and show us the entire log line from that file?