ndoutils: problem importing historical data to DB with log2n

[kad]

Hello,

I am trying to import the nagios historical data from the logs into a DB using ndoutils, but something is amiss since the only table that is getting populated is nagios_logentries.

This is the command that I am using to import:

Code: Select all

log2ndo /var/log/nagios/archives/nagios-09-24-2013-00.log -d 127.0.0.1 -i "default" -t tcp -p 5668

And this is the configuration in my ndmod.cfg:

Code: Select all

instance_name=default
output_type=tcpsocket
output=127.0.0.1
tcp_port=5668
output_buffer_items=5000
buffer_file=/var/log/nagios/ndomod.tmp
file_rotation_interval=14400
file_rotation_timeout=60
reconnect_interval=15
reconnect_warning_interval=900
data_processing_options=-1
config_output_options=3

And the configuration in my ndo2db.cfg:

Code: Select all

ndo2db_user=nagios
ndo2db_group=nagios
socket_type=tcp
socket_name=/var/log/nagios/ndo.sock
tcp_port=5668
db_servertype=mysql
db_host=localhost
db_name=nagios
db_port=3306
db_prefix=nagios_
db_user=root
max_timedevents_age=1440
max_systemcommands_age=1440
max_servicechecks_age=1440
max_hostchecks_age=1440
max_eventhandlers_age=1440

Both nagios and the DB are in the same computer (using centos 5). I tried enabling debug but it did not log any error messages: not in the ndo2db.debug nor in the messages log. I also checked that data_processing_options=-1 is set, and nd2dbo is running: at least /etc/init.d/ndo2db status returns ndo2db (pid xxx) is running .... New data is getting inserted in the DB (for example the current status information when I modify the host configuration files to add new machines), and to sum it up everything with ndo is working perfectly except for the fact that I can figure out how to import the historic data.

As I said after every import a log file the number of entries in the table nagios_logentries increases, but the rest of the tables remain unaltered: therefore I got no information on downtimes, notifications, etc.
Please, any idea of what I did wrong and how could I fix it?

[sreinhardt]

Well, just to start off with, what version of NDO and log2ndo are you running? Have you validated that localhost:5668 is open and bound to?

[kad]

Hi sreinhardt,
I am using NDOutils 1.4, specifically:
log2ndo version: 1.4b9
ndo2db version: 1.4b9

My Nagios version is 3.4.4

As for 5668 being open and bound:

Code: Select all

>> netstat -apn |grep 5668
tcp        0      0 0.0.0.0:5668                0.0.0.0:*                   LISTEN      4523/ndo2db         
tcp        0      0 127.0.0.1:5668              127.0.0.1:56019             ESTABLISHED 4617/ndo2db         
tcp        0      0 127.0.0.1:56019             127.0.0.1:5668              ESTABLISHED 4493/nagios   

[scottwilkerson]

Correct me if I am wrong, but I thought log2ndo does only import the logentries.

The rest of the info gets in the DB from memory through the NEB, I don't believe there is a way to "import" that

[kad]

Correct me if I am wrong, but I thought log2ndo does only import the logentries.

The rest of the info gets in the DB from memory through the NEB, I don't believe there is a way to "import" that

Is that the case? The whole point (in my case) of using ndoutils is to have all that information in a BD so that I can build custom reports and charts. But when migrating an existing nagios install I may got months/years of data (that I am afraid my customer will not be happy to lose). How can I see that data in my reports then? Do I have to keep the nagios web GUI for viewing historical data? There is no way of rebuild that information in the DB from the logs? I though that was what data_processing_options was for.

If log2ndo cannot be used for that how are you approaching this issue?

[scottwilkerson]

The data_processing_options is for going forward, while NDO is running as a NEB.

One thing I will point out is that the reports in Core DO NOT pull the info from NDOUtils.

[kad]

One thing I will point out is that the reports in Core DO NOT pull the info from NDOUtils.

Right, if I undestand correctly the nagios core web gui is using a cgi to extract information from the archived logs.

I´d like thought to generate custom reports querying the data in the ndo DB. Since I got the nagios historic data since it was installed (the archived logs) I would like to have that data too in the BD so that I can generate my custom reports (the ones not in the nagios core gui) for any period of time.

Is that situation so unusual no one has needed to solve it before? From what you say I understand that my options at this point are:

1) Any new reports that I make may be also able to get data parsing the logs files
2) I lose historic data: reports will only have valid data from the point ndoutils was installed.

BTW, what if the NEB stops working and nobody realizes till a couple of weeks later? From what you say I understand the problem will be the same: data that has been lost, and that I will not be able to use in spite of having the original Nagios data files.

[abrist]

Is that situation so unusual no one has needed to solve it before?

I think most people adding ndo still use the core reports, so they see no change in behavior.

BTW, what if the NEB stops working and nobody realizes till a couple of weeks later?

You could probably monitor it. . .

I understand the problem will be the same: data that has been lost, and that I will not be able to use in spite of having the original Nagios data files.

Well, you could always use the core reports instead of you custom ones. . .

[kad]

OK, I see. Thank you for your help.

[tmcdonald]

Did that solve your issue, kad?