Nagios Log server upgrade failure

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Post Reply
NMFSTeam
Posts: 88
Joined: Thu Nov 12, 2015 9:01 am

Nagios Log server upgrade failure

Post by NMFSTeam »

We are attempting to upgrade our Nagios Log Server 2.1.13 running on Ubuntu 20.04 LTS to the latest release, and it is never completing. Any advice would be appreciated.

Code: Select all

root@hqnaglogi1:/tmp# tar xzf nagioslogserver-latest.tar.gz
root@hqnaglogi1:/tmp# cd nagioslogserver/
root@hqnaglogi1:/tmp/nagioslogserver# ./upgrade 
Upgrading on Ubuntu 20.04
Nagios Log Server Upgrade Log
==========================
DATE: Fri Feb 23 21:30:11 UTC 2024

DISTRO INFO:
Ubuntu
20.04
x86_64

Old Version: 2113
New Version: 3001

Reading package lists...
Building dependency tree...
Reading state information...
php-dev is already the newest version (2:7.4+75).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Archive:  sourceguardian/loaders.linux-x86_64.zip
  inflating: /usr/lib/php/20190902/ixed.7.4.lin  
Sourceguardian extension found for PHP version 7.4
Sourceguardian extension already in php.ini
Sourceguardian extension already in php.ini
Upgrading Logstash...
Applying Nagios patches to Logstash...
Updating Logstash plugins...
Updating Logstash plugins...
Validating logstash-input-tcp-4.2.4-java.gem
Installing logstash-input-tcp
Installation successful
Validating logstash-filter-geoip-4.0.4-java.gem
Installing logstash-filter-geoip
Installation successful
Logstash upgraded OK
INSTALL: NCPA is being upgraded...
--2024-02-23 21:31:00--  https://repo.nagios.com/deb/focal/pool/main/ncpa-2.4.1.d9.amd64.deb
Resolving repo.nagios.com (repo.nagios.com)... 45.79.49.120, 2600:3c00::f03c:92ff:fef7:45ce
Connecting to repo.nagios.com (repo.nagios.com)|45.79.49.120|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10748198 (10M) [application/vnd.debian.binary-package]
Saving to: ‘ncpa-2.4.1.d9.amd64.deb’

ncpa-2.4.1.d9.amd64.deb                         100%[=======================================================================================================>]  10.25M  20.8MB/s    in 0.5s    

2024-02-23 21:31:00 (20.8 MB/s) - ‘ncpa-2.4.1.d9.amd64.deb’ saved [10748198/10748198]

(Reading database ... 1046351 files and directories currently installed.)
Preparing to unpack ncpa-2.4.1.d9.amd64.deb ...
Unpacking ncpa (2.4.1-1) over (2.4.1-1) ...
Setting up ncpa (2.4.1-1) ...
chown: cannot access '/usr/local/ncpa/etc/ncpa.cfg': No such file or directory
Processing triggers for libc-bin (2.31-0ubuntu9.14) ...
Processing triggers for systemd (245.4-4ubuntu3.23) ...
INSTALL: NCPA installed OK.
Reading package lists...
Building dependency tree...
Reading state information...
cron is already the newest version (3.0pl1-136ubuntu1).
snmp-mibs-downloader is already the newest version (1.2).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
--2024-02-23 21:31:09--  https://ftp-master.debian.org/keys/archive-key-11-security.asc
Resolving ftp-master.debian.org (ftp-master.debian.org)... 192.91.235.231
Connecting to ftp-master.debian.org (ftp-master.debian.org)|192.91.235.231|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 11873 (12K) [application/pgp-keys]
Saving to: ‘STDOUT’

-                                               100%[=======================================================================================================>]  11.59K  --.-KB/s    in 0.001s  

2024-02-23 21:31:09 (14.0 MB/s) - written to stdout [11873/11873]

--2024-02-23 21:31:09--  https://ftp-master.debian.org/keys/archive-key-11.asc
Resolving ftp-master.debian.org (ftp-master.debian.org)... 192.91.235.231
Connecting to ftp-master.debian.org (ftp-master.debian.org)|192.91.235.231|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 11861 (12K) [application/pgp-keys]
Saving to: ‘STDOUT’

-                                               100%[=======================================================================================================>]  11.58K  --.-KB/s    in 0.001s  

2024-02-23 21:31:09 (14.9 MB/s) - written to stdout [11861/11861]


WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Hit:1 http://deb.debian.org/debian-security bullseye-security InRelease
Hit:2 http://deb.debian.org/debian bullseye InRelease
Hit:3 http://us-east4.gce.archive.ubuntu.com/ubuntu focal InRelease
Hit:4 http://us-east4.gce.archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:5 http://us-east4.gce.archive.ubuntu.com/ubuntu focal-backports InRelease
Hit:6 https://packages.cloud.google.com/apt google-cloud-logging-focal-all InRelease
Hit:7 http://packages.cloud.google.com/apt google-compute-engine-focal-stable InRelease
Hit:8 https://packages.cloud.google.com/apt google-cloud-monitoring-focal-all InRelease
Hit:9 https://packages.cloud.google.com/apt google-cloud-ops-agent-focal-all InRelease
Hit:10 http://security.ubuntu.com/ubuntu focal-security InRelease
Hit:11 https://esm.ubuntu.com/apps/ubuntu focal-apps-security InRelease
Hit:12 https://esm.ubuntu.com/apps/ubuntu focal-apps-updates InRelease
Hit:13 https://esm.ubuntu.com/infra/ubuntu focal-infra-security InRelease
Hit:14 https://esm.ubuntu.com/infra/ubuntu focal-infra-updates InRelease
Reading package lists...
Building dependency tree...
Reading state information...
All packages are up to date.
W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/debian.list:1 and /etc/apt/sources.list.d/debian.list:3
W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/debian.list:1 and /etc/apt/sources.list.d/debian.list:3
W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/debian.list:1 and /etc/apt/sources.list.d/debian.list:3
W: Target CNF (main/cnf/Commands-amd64) is configured multiple times in /etc/apt/sources.list.d/debian.list:1 and /etc/apt/sources.list.d/debian.list:3
W: Target CNF (main/cnf/Commands-all) is configured multiple times in /etc/apt/sources.list.d/debian.list:1 and /etc/apt/sources.list.d/debian.list:3
W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/debian.list:1 and /etc/apt/sources.list.d/debian.list:5
W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/debian.list:1 and /etc/apt/sources.list.d/debian.list:5
W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/debian.list:1 and /etc/apt/sources.list.d/debian.list:5
W: Target CNF (main/cnf/Commands-amd64) is configured multiple times in /etc/apt/sources.list.d/debian.list:1 and /etc/apt/sources.list.d/debian.list:5
W: Target CNF (main/cnf/Commands-all) is configured multiple times in /etc/apt/sources.list.d/debian.list:1 and /etc/apt/sources.list.d/debian.list:5
W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/debian.list:1 and /etc/apt/sources.list.d/debian.list:7
W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/debian.list:1 and /etc/apt/sources.list.d/debian.list:7
W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/debian.list:1 and /etc/apt/sources.list.d/debian.list:7
W: Target CNF (main/cnf/Commands-amd64) is configured multiple times in /etc/apt/sources.list.d/debian.list:1 and /etc/apt/sources.list.d/debian.list:7
W: Target CNF (main/cnf/Commands-all) is configured multiple times in /etc/apt/sources.list.d/debian.list:1 and /etc/apt/sources.list.d/debian.list:7
W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/debian.list:2 and /etc/apt/sources.list.d/debian.list:4
W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/debian.list:2 and /etc/apt/sources.list.d/debian.list:4
W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/debian.list:2 and /etc/apt/sources.list.d/debian.list:4
W: Target CNF (main/cnf/Commands-amd64) is configured multiple times in /etc/apt/sources.list.d/debian.list:2 and /etc/apt/sources.list.d/debian.list:4
W: Target CNF (main/cnf/Commands-all) is configured multiple times in /etc/apt/sources.list.d/debian.list:2 and /etc/apt/sources.list.d/debian.list:4
W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/debian.list:2 and /etc/apt/sources.list.d/debian.list:6
W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/debian.list:2 and /etc/apt/sources.list.d/debian.list:6
W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/debian.list:2 and /etc/apt/sources.list.d/debian.list:6
W: Target CNF (main/cnf/Commands-amd64) is configured multiple times in /etc/apt/sources.list.d/debian.list:2 and /etc/apt/sources.list.d/debian.list:6
W: Target CNF (main/cnf/Commands-all) is configured multiple times in /etc/apt/sources.list.d/debian.list:2 and /etc/apt/sources.list.d/debian.list:6
W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/debian.list:2 and /etc/apt/sources.list.d/debian.list:8
W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/debian.list:2 and /etc/apt/sources.list.d/debian.list:8
W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/debian.list:2 and /etc/apt/sources.list.d/debian.list:8
W: Target CNF (main/cnf/Commands-amd64) is configured multiple times in /etc/apt/sources.list.d/debian.list:2 and /etc/apt/sources.list.d/debian.list:8
W: Target CNF (main/cnf/Commands-all) is configured multiple times in /etc/apt/sources.list.d/debian.list:2 and /etc/apt/sources.list.d/debian.list:8
W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/debian.list:1 and /etc/apt/sources.list.d/debian.list:3
W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/debian.list:1 and /etc/apt/sources.list.d/debian.list:3
W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/debian.list:1 and /etc/apt/sources.list.d/debian.list:3
W: Target CNF (main/cnf/Commands-amd64) is configured multiple times in /etc/apt/sources.list.d/debian.list:1 and /etc/apt/sources.list.d/debian.list:3
W: Target CNF (main/cnf/Commands-all) is configured multiple times in /etc/apt/sources.list.d/debian.list:1 and /etc/apt/sources.list.d/debian.list:3
W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/debian.list:1 and /etc/apt/sources.list.d/debian.list:5
W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/debian.list:1 and /etc/apt/sources.list.d/debian.list:5
W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/debian.list:1 and /etc/apt/sources.list.d/debian.list:5
W: Target CNF (main/cnf/Commands-amd64) is configured multiple times in /etc/apt/sources.list.d/debian.list:1 and /etc/apt/sources.list.d/debian.list:5
W: Target CNF (main/cnf/Commands-all) is configured multiple times in /etc/apt/sources.list.d/debian.list:1 and /etc/apt/sources.list.d/debian.list:5
W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/debian.list:1 and /etc/apt/sources.list.d/debian.list:7
W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/debian.list:1 and /etc/apt/sources.list.d/debian.list:7
W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/debian.list:1 and /etc/apt/sources.list.d/debian.list:7
W: Target CNF (main/cnf/Commands-amd64) is configured multiple times in /etc/apt/sources.list.d/debian.list:1 and /etc/apt/sources.list.d/debian.list:7
W: Target CNF (main/cnf/Commands-all) is configured multiple times in /etc/apt/sources.list.d/debian.list:1 and /etc/apt/sources.list.d/debian.list:7
W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/debian.list:2 and /etc/apt/sources.list.d/debian.list:4
W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/debian.list:2 and /etc/apt/sources.list.d/debian.list:4
W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/debian.list:2 and /etc/apt/sources.list.d/debian.list:4
W: Target CNF (main/cnf/Commands-amd64) is configured multiple times in /etc/apt/sources.list.d/debian.list:2 and /etc/apt/sources.list.d/debian.list:4
W: Target CNF (main/cnf/Commands-all) is configured multiple times in /etc/apt/sources.list.d/debian.list:2 and /etc/apt/sources.list.d/debian.list:4
W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/debian.list:2 and /etc/apt/sources.list.d/debian.list:6
W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/debian.list:2 and /etc/apt/sources.list.d/debian.list:6
W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/debian.list:2 and /etc/apt/sources.list.d/debian.list:6
W: Target CNF (main/cnf/Commands-amd64) is configured multiple times in /etc/apt/sources.list.d/debian.list:2 and /etc/apt/sources.list.d/debian.list:6
W: Target CNF (main/cnf/Commands-all) is configured multiple times in /etc/apt/sources.list.d/debian.list:2 and /etc/apt/sources.list.d/debian.list:6
W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/debian.list:2 and /etc/apt/sources.list.d/debian.list:8
W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/debian.list:2 and /etc/apt/sources.list.d/debian.list:8
W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/debian.list:2 and /etc/apt/sources.list.d/debian.list:8
W: Target CNF (main/cnf/Commands-amd64) is configured multiple times in /etc/apt/sources.list.d/debian.list:2 and /etc/apt/sources.list.d/debian.list:8
W: Target CNF (main/cnf/Commands-all) is configured multiple times in /etc/apt/sources.list.d/debian.list:2 and /etc/apt/sources.list.d/debian.list:8

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Reading package lists...
Building dependency tree...
Reading state information...
chromium is already the newest version (120.0.6099.224-1~deb11u1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
ln: failed to create symbolic link '/bin/chromium-browser': File exists
root@hqnaglogi1:/tmp/nagioslogserver#
Top
sgardil
Posts: 143
Joined: Wed Aug 09, 2023 9:58 am

Re: Nagios Log server upgrade failure

Post by sgardil »

Hey @NMFSTeam

From the error given it seems the issue is with the symlink to chromium. Have you checked to see if there is an active symlink already? You can do so by running

Code: Select all

ls -l /bin/chromium-browser
If there is a symlink you can try removing the link and try the upgrade again. Let us know if this helps.
Top
NMFSTeam
Posts: 88
Joined: Thu Nov 12, 2015 9:01 am

Re: Nagios Log server upgrade failure

Post by NMFSTeam »

Thank you for the quick reply! I just tried again after deleting the link, and now I am getting a different error.

Code: Select all

root@hqnaglogi1:/tmp/nagioslogserver# ./upgrade
Upgrading on Ubuntu 20.04
Nagios Log Server Upgrade Log
==========================
DATE: Mon Feb 26 21:05:40 UTC 2024

DISTRO INFO:
Ubuntu
20.04
x86_64

Old Version: 2113
New Version: 3001

Reading package lists...
Building dependency tree...
Reading state information...
php-dev is already the newest version (2:7.4+75).
0 upgraded, 0 newly installed, 0 to remove and 8 not upgraded.
Archive:  sourceguardian/loaders.linux-x86_64.zip
  inflating: /usr/lib/php/20190902/ixed.7.4.lin
Sourceguardian extension found for PHP version 7.4
Sourceguardian extension already in php.ini
Sourceguardian extension already in php.ini
Upgrading Logstash...
Applying Nagios patches to Logstash...
Updating Logstash plugins...
Updating Logstash plugins...
Validating logstash-input-tcp-4.2.4-java.gem
Installing logstash-input-tcp
Installation successful
Validating logstash-filter-geoip-4.0.4-java.gem
Installing logstash-filter-geoip
Installation successful
Logstash upgraded OK
INSTALL: NCPA is being upgraded...
--2024-02-26 21:06:40--  https://repo.nagios.com/deb/focal/pool/main/ncpa-2.4.1.d9.amd64.deb
Resolving repo.nagios.com (repo.nagios.com)... 45.79.49.120, 2600:3c00::f03c:92ff:fef7:45ce
Connecting to repo.nagios.com (repo.nagios.com)|45.79.49.120|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10748198 (10M) [application/vnd.debian.binary-package]
Saving to: ‘ncpa-2.4.1.d9.amd64.deb.1’

ncpa-2.4.1.d9.amd64.deb.1                       100%[====================================================================================================>]  10.25M  22.3MB/s    in 0.5s

2024-02-26 21:06:41 (22.3 MB/s) - ‘ncpa-2.4.1.d9.amd64.deb.1’ saved [10748198/10748198]

(Reading database ... 1046351 files and directories currently installed.)
Preparing to unpack ncpa-2.4.1.d9.amd64.deb ...
Unpacking ncpa (2.4.1-1) over (2.4.1-1) ...
Setting up ncpa (2.4.1-1) ...
chown: cannot access '/usr/local/ncpa/etc/ncpa.cfg': No such file or directory
Processing triggers for libc-bin (2.31-0ubuntu9.14) ...
Processing triggers for systemd (245.4-4ubuntu3.23) ...
INSTALL: NCPA installed OK.
Reading package lists...
Building dependency tree...
Reading state information...
cron is already the newest version (3.0pl1-136ubuntu1).
snmp-mibs-downloader is already the newest version (1.2).
0 upgraded, 0 newly installed, 0 to remove and 8 not upgraded.
--2024-02-26 21:06:50--  https://ftp-master.debian.org/keys/archive-key-11-security.asc
Resolving ftp-master.debian.org (ftp-master.debian.org)... 192.91.235.231
Connecting to ftp-master.debian.org (ftp-master.debian.org)|192.91.235.231|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 11873 (12K) [application/pgp-keys]
Saving to: ‘STDOUT’

-                                               100%[====================================================================================================>]  11.59K  --.-KB/s    in 0.001s

2024-02-26 21:06:50 (18.2 MB/s) - written to stdout [11873/11873]

--2024-02-26 21:06:51--  https://ftp-master.debian.org/keys/archive-key-11.asc
Resolving ftp-master.debian.org (ftp-master.debian.org)... 192.91.235.231
Connecting to ftp-master.debian.org (ftp-master.debian.org)|192.91.235.231|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 11861 (12K) [application/pgp-keys]
Saving to: ‘STDOUT’

-                                               100%[====================================================================================================>]  11.58K  --.-KB/s    in 0.001s

2024-02-26 21:06:51 (11.9 MB/s) - written to stdout [11861/11861]

mv: cannot stat './chromium_20.pref': No such file or directory
root@hqnaglogi1:/tmp/nagioslogserver#
Top
User avatar
jmichaelson
Posts: 117
Joined: Wed Aug 23, 2023 1:02 pm

Re: Nagios Log server upgrade failure

Post by jmichaelson »

Try untarring the tarball again. We expect a couple of things to be in place while installing chromium, that get moved. I'll open an internal issue for those.
Please let us know if you have any other questions or concerns.

-Jason
Top
NMFSTeam
Posts: 88
Joined: Thu Nov 12, 2015 9:01 am

Re: Nagios Log server upgrade failure

Post by NMFSTeam »

Thank you, that worked. Upgrade has been completed! Thanks again.
Top
NMFSTeam
Posts: 88
Joined: Thu Nov 12, 2015 9:01 am

Re: Nagios Log server upgrade failure

Post by NMFSTeam »

This system has never been the most responsive, but now everything seems to take a long time. Just logging in takes 3 to 5 minutes, and every click on a portion of the webUI takes the same amount of time to respond. It eventually does come up, just takes a while. Any tips on how we can troubleshoot this issue?

Thank you!
Top
bbahn
Posts: 112
Joined: Thu Jan 12, 2023 5:42 pm

Re: Nagios Log server upgrade failure

Post by bbahn »

@NMFSTeam,

To test the turbulence you're experiencing:
  • CPU and Memory Usage: Use tools like top, htop, or vmstat to monitor CPU and memory usage. High utilization might indicate that you need to allocate more resources.
  • Disk Space: Ensure there's enough disk space, especially where logs are stored. Use df -h to check disk usage.
  • Network Bandwidth: Check if the network is congested. Tools like iftop or nload can help you monitor network traffic.
If none of these is the source of your stress, then looking at the health of your Elasticsearch cluster and the Nagios & System logs may help
April: Actively advancing awesome answers with ardent alliteration, aptly addressing all ambiguities. Amplify your acumen and avail our amicable assistance. Eagerly awaiting your astute assessments of our advice.
Top
NMFSTeam
Posts: 88
Joined: Thu Nov 12, 2015 9:01 am

Re: Nagios Log server upgrade failure

Post by NMFSTeam »

The system in question has 8 vCPUs, and 64GB of memory allocated. Disk space is not an issue, has over 2TB free currently. Looks like perhaps memory is the issue? See the snippet from "top" below.

Code: Select all

top - 21:27:20 up 4 days, 23:55,  2 users,  load average: 1.54, 1.37, 1.75
Tasks: 196 total,   1 running, 195 sleeping,   0 stopped,   0 zombie
%Cpu(s):  3.9 us,  1.5 sy,  9.7 ni, 82.6 id,  0.4 wa,  0.0 hi,  1.8 si,  0.1 st
MiB Mem :  64297.1 total,   1040.4 free,  35564.7 used,  27692.0 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.  27961.6 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
1226601 root      39  19 6326764 645976  18128 S  83.1   1.0   1437:38 java
1226759 nagios    20   0   95.1g  32.7g 339092 S  36.9  52.1   1553:49 java
      1 root      20   0  171972  12708   7704 S   0.3   0.0  15:30.02 systemd
     14 root      20   0       0      0      0 I   0.3   0.0   1:19.98 rcu_sched
    616 root      16  -4   11364   1484   1288 S   0.3   0.0   1:00.31 auditd
1832984 root      20   0 1261036 253732   6784 S   0.3   0.4  10:07.17 nessus-agent-mo
Top
User avatar
jmichaelson
Posts: 117
Joined: Wed Aug 23, 2023 1:02 pm

Re: Nagios Log server upgrade failure

Post by jmichaelson »

Can you provide information as to which of those java processes is which?

ps-ax | grep java

I'm looking to see which one is the elasticsearch process and which is the logstash (I have my hunch I'd just like to confirm it).

Also, matching up that shot from top again if the process id's don't match up.

It may be a memory issue, for sure, but since it shows that despite the 95gb java process, there's 27gb taken up by "buffers" that doesn't seem all that likely.
Please let us know if you have any other questions or concerns.

-Jason
Top
NMFSTeam
Posts: 88
Joined: Thu Nov 12, 2015 9:01 am

Re: Nagios Log server upgrade failure

Post by NMFSTeam »

Here is a fresh pull from top:

Code: Select all

top - 14:15:20 up 9 days, 16:43,  2 users,  load average: 2.82, 2.54, 2.27
Tasks: 199 total,   2 running, 197 sleeping,   0 stopped,   0 zombie
%Cpu(s): 16.2 us,  1.0 sy,  8.3 ni, 71.9 id,  1.1 wa,  0.0 hi,  1.2 si,  0.4 st
MiB Mem :  64297.1 total,   1098.2 free,  35548.5 used,  27650.4 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.  27980.4 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 176751 root      20   0  300724 296848  15060 R  89.1   0.5 389:05.83 aide
1226601 root      39  19 6573004 685524  18104 S  70.2   1.0   4576:58 java
1226759 nagios    20   0   94.6g  32.4g  24824 S  44.7  51.7   3836:08 java
    131 root      20   0       0      0      0 S   0.7   0.0  37:50.02 kswapd0
  74787 root      20   0 1266604 251480   8520 S   0.7   0.4   4:46.86 nessus-agent-mo
 327584 root      20   0       0      0      0 I   0.3   0.0   0:00.24 kworker/u16:3-writeback
 331081 root      20   0   52160  13256  11316 R   0.3   0.0   0:00.08 top
      1 root      20   0  171972  11744   6740 S   0.0   0.0  34:07.02 systemd
And here are the results of the ps command:

Code: Select all

root@hqnaglogi1:~# ps -ax | grep java
 331290 pts/1    S+     0:00 grep --color=auto java
1226601 ?        SNl  4577:22 /bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Xmx500m -Xss2048k -Djffi.boot.library.path=/usr/local/nagioslogserver/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/nagioslogserver/logstash/heapdump.hprof -Xbootclasspath/a:/usr/local/nagioslogserver/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/local/nagioslogserver/logstash/vendor/jruby -Djruby.lib=/usr/local/nagioslogserver/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /usr/local/nagioslogserver/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/logstash/logstash.log -w 4
1226759 ?        SLl  3836:19 /bin/java -Xms32148m -Xmx32148m -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Des.cluster.name=c2f33fb7-081b-4317-abd1-643e2a8a2aff -Des.node.name=c676baf2-bd36-4dc9-9b4f-76d81e2ae08e -Des.discovery.zen.ping.unicast.hosts=localhost,10.27.25.5 -Des.path.repo=/ -Delasticsearch -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.path.home=/usr/local/nagioslogserver/elasticsearch -cp :/usr/local/nagioslogserver/elasticsearch/lib/elasticsearch-1.7.6.jar:/usr/local/nagioslogserver/elasticsearch/lib/*:/usr/local/nagioslogserver/elasticsearch/lib/sigar/* -Des.default.path.home=/usr/local/nagioslogserver/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/usr/local/nagioslogserver/elasticsearch/data -Des.default.path.work=/usr/local/nagioslogserver/tmp/elasticsearch -Des.default.path.conf=/usr/local/nagioslogserver/elasticsearch/config org.elasticsearch.bootstrap.Elasticsearch
root@hqnaglogi1:~#
Top
Post Reply

Return to “Nagios Log Server”