Static data is missing

[gimeb]

I have configured NLS to display static data from Windows server. This data is in the .txt format and generated on a monthly basis. Currently, I have 3 files for the month of Dec 2015, Jan 2016 and Feb 2016. The NLS dashboard is displaying data only for Jan 2016 and the rest is missing. Below is snippet from the nxlog.conf file. The full nxlog.conf file is attached. Please advise and let me know if you need additional information.

<Input itcamlogs>
Module im_file
File 'D:\ITCAM-Logs\runtraps*'
SavePos TRUE
Exec $Message = $raw_event;
</Input>

[jolson]

What happens if you append a line to one of those older logs? For example, open one of the older logs in notepad or similar and add a new line to the bottom of it. After adding the new line, save the document and check your Web GUI - does anything appear appropriately?

Is there a formatting difference in those old logs that could be causing a problem? Sometimes datestamps vary between logs, that's worth checking on.

[gimeb]

I have updated the files for 2015-12 and 2016-02. In was able to see data only from 2016-02 now. However, the data for 2015-12 is still not showing. I have attached document detailing the steps I have attempted and supporting screens. in addition, please let me know what would be another option to receive data automatically without going into each file and updating it

[gimeb]

I would like to provide an additional update. I have added a new line to the bottom of 2015-12 file. I also created a copy of it. After some time the data for 2015-12 was started to show. Please let me know what would be another option to receive data automatically without going into each file and updating it. We might receive similar type of data from multiple places and it would be time consuming to update each file on remote servers in order to get them into NLS.

[hsmith]

Can you see if any of this discussion from this website helps out at all? https://logstash.jira.com/browse/LOGSTASH-1506

[gimeb]

From the link you have provided I see that LOGSTASH-1636 - "Existing events not being imported" https://logstash.jira.com/browse/LOGSTASH-1636 could be pertaining to the issue I have encountered. However, I am not able to see what is the resolution. Please assist

[jolson]

I have verified the following:
* When nxlog is started and it has been assigned a file, it begins processing at the bottom of the log file, not the beginning.
* When a new log entry appears, nxlog processes it.
* A document can be re-parsed by changing your configuration in accordance with the following:

Default input:

Code: Select all

<Input test_log1>
    Module   im_file
    File     'C:\whaterver\test*'
    SavePos  TRUE
    Exec     $Message = $raw_event;
</Input>

Changed:

Code: Select all

<Input test_log1>
    Module   im_file
    File     'C:\whaterver\test*'
    SavePos  FALSE
    ReadFromLast FALSE
    Exec     $Message = $raw_event;
</Input>

Note that you will need to restart nxlog after making the above change.

You will see all of your logs that match C:\whaterver\test* arrive in Nagios Log Server. After that processing is complete, you should change your configuration back to the default:

Code: Select all

<Input test_log1>
    Module   im_file
    File     'C:\whaterver\test*'
    SavePos  TRUE
    Exec     $Message = $raw_event;
</Input>

Again, restart nxlog.

Hopefully this helps, thanks!

[gimeb]

Thank you for the resolving the issue. I after changing the nxlog default input I can see data for 10/2015, 11/2015 now along with what was there before (12/2015, 1/2016 and 2/2016). Please let me know the following:

1. The reason I should change the configuration back to the default
2. On the dashboard the furthest back I can select is 30 days. However, on my dashboard I still can see some data for 11/2015. Please explain.
3. My guess that NLS will not display data correctly if it goes back more than a month. Is this true?

[hsmith]

The reason I should change the configuration back to the default

You'll get spammed with old logs each time you restart. You'll end up with a massive amount of duplicates.

. On the dashboard the furthest back I can select is 30 days. However, on my dashboard I still can see some data for 11/2015. Please explain.

You send *all* of the log data to NLS today. The logs are held in NLS for the 30 days after the logs are received. Since they were all received today, NLS treats them like new logs.

My guess that NLS will not display data correctly if it goes back more than a month. Is this true?

If you open the index back up, it should be fine.

[gimeb]

Please clarify the following. You have indicated that "The logs are held in NLS for the 30 days after the logs are received".

1. What would happened after 30 days? Is this default setting or it can be adjusted?
2. On my Backup&Maintenance I have defined to "Close indexes older than 60 days". Should the dashboard show the option "Last 60 days" and not just "Last 30 days"
3. For statistical analysis we might need to go back beyond 60 days and longer. How I would I be able to accomplish that? Is this advisable?