Nagios Log Server 2024R2 error when configuring snapshot repository

[IT_LAS]

Hi,
I'm configuring a snapshot repository on a Nagios Log Server 2024R2 2 server cluster and it keeps showing errors regarding some issue with the directory permissions.
I've checked everything possible and I cannot find a solution to the issue.
On the Opensearch logs I see the following error trace:

Code: Select all

org.opensearch.transport.RemoteTransportException: [node1][172.16.10.145:9300][cluster:admin/repository/put]
Caused by: org.opensearch.repositories.RepositoryVerificationException: [NLSPRO] cannot delete test data at
        at org.opensearch.repositories.blobstore.BlobStoreRepository.endVerification(BlobStoreRepository.java:2018) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.repositories.RepositoriesService$3.lambda$doRun$1(RepositoriesService.java:383) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854) ~[opensearch-2.14.0.jar:2.14.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: java.nio.file.DirectoryNotEmptyException: /mnt/nlspron/tests-hnoudV8AS5ab1icPpabRbg
        at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:289) ~[?:?]
        at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:104) ~[?:?]
        at java.nio.file.Files.delete(Files.java:1152) ~[?:?]
        at org.opensearch.common.blobstore.fs.FsBlobContainer$1.postVisitDirectory(FsBlobContainer.java:141) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.common.blobstore.fs.FsBlobContainer$1.postVisitDirectory(FsBlobContainer.java:137) ~[opensearch-2.14.0.jar:2.14.0]
        at java.nio.file.Files.walkFileTree(Files.java:2803) ~[?:?]
        at java.nio.file.Files.walkFileTree(Files.java:2857) ~[?:?]
        at org.opensearch.common.blobstore.fs.FsBlobContainer.delete(FsBlobContainer.java:137) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.repositories.blobstore.BlobStoreRepository.endVerification(BlobStoreRepository.java:2016) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.repositories.RepositoriesService$3.lambda$doRun$1(RepositoriesService.java:383) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854) ~[opensearch-2.14.0.jar:2.14.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]

The directory for the repository is a cifs-shared folder that's located on another server.
Is there any other thing that I should check?

I've made the same process on a standalone server cluster and it worked without errors.

[jmichaelson]

The first thing to check is whether the snapshot directory is registered and mounted on all the systems in your cluster. Each system has to have write access to the shared directory for snapshots. Then check the path.repo line in /usr/local/nagioslogserver/opensearch/config/opensearch.yml. It needs to contain the name of the mounted directory, which needs to be the same across each node in the cluster.

Assuming that all checks out, make sure that the nagios user can write a file to that directory (do a su - nagios and try writing something from each system to try).

[IT_LAS]

Both server have the directory correctly mounted and registered. On both servers I can write/delete files on the mounted directory with the nagios user. The directory is also configured on the "/usr/local/nagioslogserver/opensearch/config/opensearch.yml".
Also the default respository path seems to be undetectable even if it is configured in the yaml of the Opensearch service.

[jmichaelson]

Reading through that log its cmplaining about being unable to delete a file. DId OpenSearch put (and leave) anything in the shared repository?

[IT_LAS]

Hi,
Yes, it creates a folder named tests-xxxxxxx that seems to be used to verify if the service have write permissions on the folder.

For the record, in the end I've been able to configure the snapshot repository.
I had to reinstall everything from scratch, set up the snapshot repository first before adding the second node to the cluster
If I added the node first and then tried to set up the repository is where I encountered the nonsensical permissions issues.

[jmichaelson]

That's good to know. Thank you! I'll investigate that deeper both within Log Server and upstream with OpenSearch.