Page 1 of 1

check_mysql_query segfaults with mysql-connector-c

Posted: Thu Oct 04, 2018 2:35 am
by internethering
Hi,

Gentoo switched from mysql shipped client libs to mysql-connector-c (https://dev.mysql.com/downloads/connector/c/). This causes two problems with check_mysql_query:
/usr/lib64/nagios/plugins/check_mysql_query -q "SELECT CASE WHEN variable_value = 'ON' THEN 0 ELSE 2 END FROM information_schema.global_status WHERE variable_name='SLAVE_RUNNING'" -u munin -w 0 -c 1
Speicherzugriffsfehler (Speicherabzug geschrieben)
GDB tell me:

Code: Select all

           PID: 30491 (check_mysql_que)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Thu 2018-11-15 14:40:05 CET (2s ago)
  Command Line: /usr/lib64/nagios/plugins/check_mysql_query -q SELECT CASE WHEN variable_value = 'ON' THEN 0 ELSE 2 END FROM information_schema.global_status WHERE variable_name='SLAVE_RUNNING' -u munin -w 0 -c 1
    Executable: /usr/lib64/nagios/plugins/check_mysql_query
 Control Group: /user.slice/user-1000.slice/session-584.scope
          Unit: session-584.scope
         Slice: user-1000.slice
       Session: 584
     Owner UID: 1000 (rhering)
       Boot ID: 1402b547b1f842789ffb300c29f0e707
    Machine ID: 4a6906c2152dff421b9b5a385455c64e
      Hostname: alpha.mail
       Storage: /var/lib/systemd/coredump/core.check_mysql_que.0.1402b547b1f842789ffb300c29f0e707.30491.1542289205000000.lz4
       Message: Process 30491 (check_mysql_que) of user 0 dumped core.
                
                Stack trace of thread 30491:
                #0  0x00007f74fb668c47 memset (libcrypto.so.1.0.0)
                #1  0x00007f74fc4741ca my_aes_decrypt (libmysqlclient.so.18)
                #2  0x00007f74fc47063f n/a (libmysqlclient.so.18)
                #3  0x00007f74fc470b45 n/a (libmysqlclient.so.18)
                #4  0x00007f74fc470f6e my_search_option_files (libmysqlclient.so.18)
                #5  0x00007f74fc471735 my_load_defaults (libmysqlclient.so.18)
                #6  0x00007f74fc432ec8 mysql_read_default_options (libmysqlclient.so.18)
                #7  0x00007f74fc43a3fc mysql_real_connect (libmysqlclient.so.18)
                #8  0x00005571d9d80796 main (check_mysql_query)
                #9  0x00007f74fbe33ae7 __libc_start_main (libc.so.6)
                #10 0x00005571d9d80bfa _start (check_mysql_query)

GNU gdb (Gentoo 8.2 p1) 8.2
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib64/nagios/plugins/check_mysql_query...Reading symbols from /usr/lib64/debug//usr/lib64/nagios/plugins/check_mysql_query.debug...done.
done.

warning: core file may not match specified executable file.
[New LWP 30491]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/lib64/nagios/plugins/check_mysql_query -q SELECT CASE WHEN variable_value'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f74fb668c47 in memset (__len=168, __ch=0, __dest=0x0) at /usr/include/bits/string_fortified.h:71
71        return __builtin___memset_chk (__dest, __ch, __len, __bos0 (__dest));
(gdb) bt
#0  0x00007f74fb668c47 in memset (__len=168, __ch=0, __dest=0x0) at /usr/include/bits/string_fortified.h:71
#1  EVP_CIPHER_CTX_init (ctx=0x0) at evp_enc.c:82
#2  0x00007f74fc4741ca in my_aes_decrypt () from /usr/lib64/libmysqlclient.so.18
#3  0x00007f74fc47063f in ?? () from /usr/lib64/libmysqlclient.so.18
#4  0x00007f74fc470b45 in ?? () from /usr/lib64/libmysqlclient.so.18
#5  0x00007f74fc470f6e in my_search_option_files () from /usr/lib64/libmysqlclient.so.18
#6  0x00007f74fc471735 in my_load_defaults () from /usr/lib64/libmysqlclient.so.18
#7  0x00007f74fc432ec8 in mysql_read_default_options () from /usr/lib64/libmysqlclient.so.18
#8  0x00007f74fc43a3fc in mysql_real_connect () from /usr/lib64/libmysqlclient.so.18
#9  0x00005571d9d80796 in main (argc=<optimized out>, argv=<optimized out>) at check_mysql_query.c:100
This problem is gone, if you rename the .mylogin.cnf (see https://dev.mysql.com/doc/refman/8.0/en ... ditor.html). I've atm no idea howto solve it.

---

2nd problem:

Code: Select all

           PID: 21044 (check_mysql_que)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Thu 2018-11-15 14:45:56 CET (3s ago)
  Command Line: /usr/lib64/nagios/plugins/check_mysql_query -q SELECT CASE WHEN variable_value = 'ON' THEN 0 ELSE 2 END FROM information_schema.global_status WHERE variable_name='SLAVE_RUNNING' -u munin -w 0 -c 1
    Executable: /usr/lib64/nagios/plugins/check_mysql_query
 Control Group: /user.slice/user-1000.slice/session-586.scope
          Unit: session-586.scope
         Slice: user-1000.slice
       Session: 586
     Owner UID: 1000 (rhering)
       Boot ID: 1402b547b1f842789ffb300c29f0e707
    Machine ID: 4a6906c2152dff421b9b5a385455c64e
      Hostname: alpha.mail
       Storage: /var/lib/systemd/coredump/core.check_mysql_que.0.1402b547b1f842789ffb300c29f0e707.21044.1542289556000000.lz4
       Message: Process 21044 (check_mysql_que) of user 0 dumped core.
                
                Stack trace of thread 21044:
                #0  0x00007f7f8a455613 __rawmemchr_avx2 (libc.so.6)
                #1  0x00007f7f8a374192 _IO_str_init_static_internal (libc.so.6)
                #2  0x00007f7f8a3643fd _IO_vsscanf (libc.so.6)
                #3  0x00007f7f8a35d97c __sscanf (libc.so.6)
                #4  0x000055ba80c114c8 is_numeric (check_mysql_query)
                #5  0x000055ba80c1080a main (check_mysql_query)
                #6  0x00007f7f8a311ae7 __libc_start_main (libc.so.6)
                #7  0x000055ba80c10bfa _start (check_mysql_query)

GNU gdb (Gentoo 8.2 p1) 8.2
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib64/nagios/plugins/check_mysql_query...Reading symbols from /usr/lib64/debug//usr/lib64/nagios/plugins/check_mysql_query.debug...done.
done.

warning: core file may not match specified executable file.
[New LWP 21044]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/lib64/nagios/plugins/check_mysql_query -q SELECT CASE WHEN variable_value'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __rawmemchr_avx2 () at ../sysdeps/x86_64/multiarch/memchr-avx2.S:61
61              VPCMPEQ (%rdi), %ymm0, %ymm1
(gdb) bt
#0  __rawmemchr_avx2 () at ../sysdeps/x86_64/multiarch/memchr-avx2.S:61
#1  0x00007f7f8a374192 in _IO_str_init_static_internal (sf=sf@entry=0x7ffed8893cd0, ptr=ptr@entry=0x8f8f8f8f8f8f8f8f <error: Cannot access memory at address 0x8f8f8f8f8f8f8f8f>, size=size@entry=0, pstart=pstart@entry=0x0) at strops.c:41
#2  0x00007f7f8a3643fd in _IO_vsscanf (string=0x8f8f8f8f8f8f8f8f <error: Cannot access memory at address 0x8f8f8f8f8f8f8f8f>, format=0x55ba80c1805c "%f%c", args=args@entry=0x7ffed8893e00) at iovsscanf.c:40
#3  0x00007f7f8a35d97c in __sscanf (s=<optimized out>, format=format@entry=0x55ba80c1805c "%f%c") at sscanf.c:32
#4  0x000055ba80c114c8 in is_numeric (number=<optimized out>) at utils.c:245
#5  0x000055ba80c1080a in main (argc=<optimized out>, argv=<optimized out>) at check_mysql_query.c:148
I got this problem fixed:

Code: Select all

diff -urN a/plugins/check_mysql_query.c b/plugins/check_mysql_query.c
--- a/plugins/check_mysql_query.c       2018-11-15 14:25:30.110775355 +0100
+++ b/plugins/check_mysql_query.c       2018-11-15 14:25:49.477436877 +0100
@@ -136,18 +136,18 @@
                die (STATE_CRITICAL, "QUERY %s: Fetch row error - %s\n", _("CRITICAL"), error);
        }
 
-       /* free the result */
-       mysql_free_result (res);
-
-       /* close the connection */
-       mysql_close (&mysql);
-
        if (! is_numeric(row[0])) {
                die (STATE_CRITICAL, "QUERY %s: %s - '%s'\n", _("CRITICAL"), _("Is not a numeric"), row[0]);
        }
 
        value = strtod(row[0], NULL);
 
+       /* free the result */
+       mysql_free_result (res);
+
+       /* close the connection */
+       mysql_close (&mysql);
+
        if (verbose >= 3)
                printf("mysql result: %f\n", value);
some version info:
dev-db/mysql-connector-c: 6.1.11-r1
dev-db/mysql: 5.7.23-r2
net-analyzer/nagius-plugins: 2.2.1-r3

Re: check_mysql_query segfaults with mysql-connector-c

Posted: Thu Oct 04, 2018 3:23 pm
by scottwilkerson
Where did you get this plugin from?
internethering wrote: net-analyzer/monitoring-plugins: 2.2-r3
We are not the author of this, are you sure you are working with the nagios-plugins package and not something else?

Re: check_mysql_query segfaults with mysql-connector-c

Posted: Tue Oct 16, 2018 7:26 am
by internethering
scottwilkerson wrote:We are not the author of this, are you sure you are working with the nagios-plugins package and not something else?
Ups, you're right. Correct address: https://github.com/monitoring-plugins/m ... ssues/1557

Re: check_mysql_query segfaults with mysql-connector-c

Posted: Tue Oct 16, 2018 7:49 am
by scottwilkerson
internethering wrote:
scottwilkerson wrote:We are not the author of this, are you sure you are working with the nagios-plugins package and not something else?
Ups, you're right. Correct address: https://github.com/monitoring-plugins/m ... ssues/1557
Locking thread

Re: check_mysql_query segfaults with mysql-connector-c

Posted: Thu Nov 15, 2018 8:28 am
by scottwilkerson
Unlocked per user request

Re: check_mysql_query segfaults with mysql-connector-c

Posted: Thu Nov 15, 2018 8:49 am
by internethering
Thanks. The problem exists for check_mysql_query from nagios-plugins.org too. I updated my first post with actual backtraces and a patch for one problem.

By the way, check_mysql has the problem with .mylogin.cnf too:

Code: Select all

Core was generated by `/usr/lib64/nagios/plugins/check_mysql -H localhost -n'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f7f2250ac47 in memset (__len=168, __ch=0, __dest=0x0) at /usr/include/bits/string_fortified.h:71
71        return __builtin___memset_chk (__dest, __ch, __len, __bos0 (__dest));
(gdb) bt
#0  0x00007f7f2250ac47 in memset (__len=168, __ch=0, __dest=0x0) at /usr/include/bits/string_fortified.h:71
#1  EVP_CIPHER_CTX_init (ctx=0x0) at evp_enc.c:82
#2  0x00007f7f233161ca in my_aes_decrypt () from /usr/lib64/libmysqlclient.so.18
#3  0x00007f7f2331263f in ?? () from /usr/lib64/libmysqlclient.so.18
#4  0x00007f7f23312b45 in ?? () from /usr/lib64/libmysqlclient.so.18
#5  0x00007f7f23312f6e in my_search_option_files () from /usr/lib64/libmysqlclient.so.18
#6  0x00007f7f23313735 in my_load_defaults () from /usr/lib64/libmysqlclient.so.18
#7  0x00007f7f232d4ec8 in mysql_read_default_options () from /usr/lib64/libmysqlclient.so.18
#8  0x00007f7f232dc3fc in mysql_real_connect () from /usr/lib64/libmysqlclient.so.18
#9  0x000055d336ef2ca8 in main (argc=<optimized out>, argv=<optimized out>) at check_mysql.c:143

Re: check_mysql_query segfaults with mysql-connector-c

Posted: Thu Nov 15, 2018 5:17 pm
by scottwilkerson
Are you passing the -f flag with the full path to the file?

Code: Select all

 -f, --file=STRING
    Read from the specified client options file