Cant open up sockets..

[antonisnyc94]

I dont understand as to why version 4.1.5 works fine and once i change the docker image to 4.2.0 I get this error. Im using the opendistro wazuh docker images. With netstat it seems like wazuh doesnt listen to anything.. Do you know any further troubleshooting steps i should take?

[cont-init.d] 2-manager: executing... 
2021/08/27 18:45:40 wazuh-analysisd: CRITICAL: (1210): Queue 'queue/sockets/queue' not accessible: 'No such file or directory'
wazuh-analysisd: Configuration error. Exiting
[cont-init.d] 2-manager: exited 1.

Thanks,
Tony

[antonisnyc94]

I have this implemented with kubernetes as found in wazuh-kubernetes. Note that i upgraded from 4.1.5 and due to all of the renaming I assume it must be a problem with the persistent volume storing old data or the new mount paths not being changed in the statefulset

[cleverer]

Same issue here…

[cleverer]

got it running by creating the following folders by hand:

docker-compose exec wazuh-manager bash
cd /var/ossec/queue
mkdir sockets
chown ossec:ossec sockets
chmod 770 sockets
mkdir logcollector
chown ossec:ossec logcollector
chmod 770 logcollector

[antonisnyc94]

I added the paths in k8s and now im getting this:

[cont-init.d] 2-manager: executing...
Starting Wazuh v4.2.0...
Started wazuh-apid...
Started wazuh-csyslogd...
Started wazuh-dbd...
2021/08/31 11:49:59 wazuh-integratord: INFO: Remote integrations not configured. Clean exit.
Started wazuh-integratord...
Started wazuh-agentlessd...
Started wazuh-db...
Started wazuh-execd...
Started wazuh-analysisd...
2021/08/31 11:50:02 wazuh-syscheckd: WARNING: The check_unixaudit option is deprecated in favor of the SCA module.
Started wazuh-syscheckd...
Started wazuh-remoted...
wazuh-logcollector did not start correctly.
[cont-init.d] 2-manager: exited 1.

k8s config update:
- name: wazuh-manager-master
mountPath: /var/ossec/queue/sockets
subPath: wazuh/var/ossec/queue/ossec
- name: wazuh-manager-master
mountPath: /var/ossec/queue/sockets/logcollector
subPath: wazuh/var/ossec/queue/ossec/logcollector

[DFolchA]

Hello, thank you for reporting this,

This issue seems to be related with the changes in the directory structure done in this issue: wazuh/wazuh#6885

It seems that the old directories are maintained in a permanent volume and the new ones are not created on upgrade.

[DFolchA]

Update Report

Working branch: https://github.com/wazuh/wazuh-docker/tree/fix-4.2-upgrade

Proposed and tested three different solutions:

1. Remove ossec_queue volume

By removing the volume from here:

wazuh-docker/build-from-sources.yml

Line 24 in 4b75a66

- ossec_queue:/var/ossec/queue

We can overwrite the old /var/ossec/queue/ with the new directory structure.

This solution was rejected due to data loss risk caused by the deletion of the databases.

2. Remove /queue/ossec from the persistent volume and add /queue/sockets and /queue/logcollector

By adding /queue/sockets and /queue/logcollector to the permanent data list (PERMANENT_DATA) these directories will be created when deploying wazuh-docker 4.2.0 additionally we can remove /queue/ossec by adding it to PERMANENT_DATA_DEL.

This solution was implemented on this commit: 6da1b19

This solution was rejected because it would be difficult to apply the same solution for future path name changes.

3. Create new function to rename paths

We added a new function to the init script that renames the files/directories indicated in a new list PERMANENT_DATA_MOVE.

This solution replicates the behaviour implemented in wazuh-packages, and solves the problems of the previous solutions, avoiding data loss and providing a method to modify the paths in the future.

This solution was implemented on this commit: 68bc08f

Wazuh service after upgrading using the new image:

dfolch@pop-os:~/wazuh-docker$ docker-compose exec wazuh  /var/ossec/bin/wazuh-control status
wazuh-clusterd not running...
wazuh-modulesd is running...
wazuh-monitord is running...
wazuh-logcollector is running...
wazuh-remoted is running...
wazuh-syscheckd is running...
wazuh-analysisd is running...
wazuh-maild not running...
wazuh-execd is running...
wazuh-db is running...
wazuh-authd is running...
wazuh-agentlessd not running...
wazuh-integratord not running...
wazuh-dbd not running...
wazuh-csyslogd not running...
wazuh-apid is running...
dfolch@pop-os:~/wazuh-docker$ docker-compose exec wazuh ls -l /var/ossec/queue
total 52
drwxrwx--- 2 root   ossec 4096 Apr 22 13:39 agent-groups
drwxr-x--- 2 ossec  ossec 4096 Apr 22 13:39 agentless
drwxrwx--- 2 ossec  ossec 4096 Sep  2 14:53 alerts
drwxrwx--- 2 ossec  ossec 4096 Apr 22 13:39 cluster
drwxr-x--- 2 ossec  ossec 4096 Sep  2 14:53 db
drwxr-x--- 3 ossec  ossec 4096 Sep  2 14:47 diff
drwxr-x--- 3 ossec  ossec 4096 Sep  2 09:31 fim
drwxr-x--- 2 ossec  ossec 4096 Sep  2 14:46 fts
drwxr-x--- 2 ossec  ossec 4096 Sep  2 14:53 logcollector
drwxrwx--- 2 ossecr ossec 4096 Sep  2 14:46 rids
drwxrwx--- 2 ossec  ossec 4096 Sep  2 14:53 sockets
drwxrwx--- 2 ossec  ossec 4096 Sep  2 14:53 tasks
drw-rw---- 3 root   ossec 4096 Sep  2 09:31 vulnerabilities
dfolch@pop-os:~/wazuh-docker$ docker-compose exec wazuh ls -l /var/ossec/logs
total 44
-rw-rw---- 1 ossec  ossec     0 Sep  2 09:31 active-responses.log
drwxr-x--- 3 ossec  ossec  4096 Sep  2 14:53 alerts
drwxr-x--- 2 ossec  ossec  4096 Apr 22 13:39 api
-rw-rw---- 1 ossec  ossec  1265 Sep  2 14:53 api.log
drwxr-x--- 3 ossec  ossec  4096 Sep  2 14:53 archives
drwxr-x--- 2 ossec  ossec  4096 Apr 22 13:39 cluster
-rw-rw---- 1 ossec  ossec     0 Sep  2 14:46 cluster.log
drwxr-x--- 3 ossec  ossec  4096 Sep  2 14:53 firewall
-rw-r----- 1 ossecm ossec     0 Sep  2 09:31 integrations.log
-rw-rw---- 1 root   ossec 13501 Sep  2 14:53 ossec.log
drwxr-x--- 2 ossec  ossec  4096 Apr 22 13:39 wazuh

Workaround

Until these changes are released a possible solution for this problem would be to manually replicate the third solution and rename the affected directories

docker-compose exec wazuh mv -f /var/ossec/queue/ossec /var/ossec/queue/sockets

docker-compose exec wazuh mkdir /var/ossec/queue/logcollector

docker-compose exec wazuh  chown ossec:ossec /var/ossec/queue/logcollector

docker-compose exec wazuh  chmod 750 /var/ossec/queue/logcollector

[DFolchA]

Update

We have uploaded the new images to Dockerhub and retagged v4.2.0

[okynos]

Hello!
I will close this issue due to the fix is done.
Feel free to open up if it appears again or there are something similar.

Regards.

[antonisnyc94]

@DFolchA @okynos
Can you update the wazuh-odfe image in Dockerhub as well?

Thank you for taking care of the issue!

Regards,
Tony

[DFolchA]

Hello @antonisnyc94,

We have uploaded the new wazuh-odfe image to Dockerhub, sorry for the inconveniences.

Regards.

[antonisnyc94]

@DFolchA Great thank you!

Please not that for upgrading wazuh in kubernets, I had to add a "chown ossec:ossec /var/ossec/queue/sockets" in /etc/cont-init.d/2-manager since i was getting permissions denied error during startup because the owner is root.

Adding the command in containers statup wont work, due to the containers nature in mounting the volumes after startup.