RabbitMQ cache grows really big

Troubleshooting
1

I am self-hosting a taiga instance with a single project, and I got various issues with available disk space. Upon inspecting /var/lib/docker/volumes it seems that there’s 3.6 GB of RabbitMQ data (2 GB volumes/taiga-docker_taiga-async-rabbitmq-data and 1.6 GB volumes/taiga-docker_taiga-events-rabbitmq-data) versus 102 MB of actual project data (91 MB volumes/taiga-docker_taiga-db-data and 11 MB volumes/taiga-docker_taiga-media-data), which is a 37 times difference. So,

  1. Are these RMQ volumes safe to remove? What do they even do?
  2. Is it expected that these volumes grow this quickly?
1 Like
2

Hi @Wizzerinus

That is an strange behavior on the platform. The resource consumption of rabbitmq should be less.

Maybe they are creating a different vhost each time the service starts. Without reusing the existing one. For example

ls /var/lib/docker/volumes/taiga-docker_taiga-events-rabbitmq-data/_data/mnesia/

rabbit@07a159b2d29a
rabbit@07a159b2d29a-feature_flags
rabbit@07a159b2d29a-plugins-expand
rabbit@c9779ca3fd93
rabbit@c9779ca3fd93-feature_flags
rabbit@c9779ca3fd93.pid
rabbit@c9779ca3fd93-plugins-expand

## BAD: There are two vhosts here

If this happens, make sure you have the hostname defined in your services in docker-compose.yml.

  ...            
  taiga-async-rabbitmq:                                                                                                                                                                       
     image: rabbitmq:3.8-management-alpine                                                                                                                                                     
     hostname: "taiga-async-rabbitmq"  # <- HERE
     ...            
  ...            
  taiga-events-rabbitmq:                                                                                                                                                                       
     image: rabbitmq:3.8-management-alpine                                                                                                                                                     
     hostname: "taiga-events-rabbitmq"   # <- HERE
     ...

If this doesn’t fix the problem:

  • Can you paste the info of your docker repo
    cd taiga-docker
git describe --long
    …and the possible modifications you have made to the docker-compose.yml file?
  • Can you check and share the logs for those resources?

The events are useful for the interface to work in real time on the boards (kanban and taskboard). It is also used to manage asynchronous tasks (sending emails, background calculations, imports and exports, integrations, webhooks…).

You can delete these volumes so that they can be recreated, some notification or some async task (some email, some request to an external service -webhooks, integrations- ) may be lost. Although it is not recommended to do it very often, especially with the async service.

Best regards

3

Thank you for the comprehensive response! ^^

Here’s the Docker-compose file I’m using:

In particular, the hostnames are set, but they’re different, in your case they’re both taiga-async-rabbitmq, in my case one is async and the other is events, idk if that’s intended but also I did not change this:

  taiga-async-rabbitmq:
    image: rabbitmq:3.8-management-alpine
    environment: ...
    hostname: "taiga-async-rabbitmq"
    ...
  taiga-events-rabbitmq:
    image: rabbitmq:3.8-management-alpine
    environment: ...
    hostname: "taiga-events-rabbitmq"
    ...

This is the output of the volume inspection command:

-> % sudo ls /var/lib/docker/volumes/taiga-docker_taiga-async-rabbitmq-data/_data/mnesia/ -l
total 64
drwxr-xr-x  4 100 101 4096 Oct 14  2022 rabbit@07d652149e8a
-rw-r--r--  1 100 101  148 Oct 14  2022 rabbit@07d652149e8a-feature_flags
drwxr-xr-x 10 100 101 4096 Oct 14  2022 rabbit@07d652149e8a-plugins-expand
drwxr-xr-x  4 100 101 4096 Oct 14  2022 rabbit@960ae3a757d4
-rw-r--r--  1 100 101  148 Oct 14  2022 rabbit@960ae3a757d4-feature_flags
drwxr-xr-x 10 100 101 4096 Oct 14  2022 rabbit@960ae3a757d4-plugins-expand
drwxr-xr-x  4 100 101 4096 Oct 14  2022 rabbit@abd9c6756945
-rw-r--r--  1 100 101  148 Oct 14  2022 rabbit@abd9c6756945-feature_flags
drwxr-xr-x 10 100 101 4096 Oct 14  2022 rabbit@abd9c6756945-plugins-expand
drwxr-xr-x  4 100 101 4096 Oct 14  2022 rabbit@cbd6e6ca3476
-rw-r--r--  1 100 101  148 Oct 14  2022 rabbit@cbd6e6ca3476-feature_flags
drwxr-xr-x 10 100 101 4096 Oct 14  2022 rabbit@cbd6e6ca3476-plugins-expand
drwxr-xr-x  4 100 101 4096 Jun 17 08:11 rabbit@taiga-async-rabbitmq
-rw-r--r--  1 100 101  148 May 18 18:49 rabbit@taiga-async-rabbitmq-feature_flags
-rw-r--r--  1 100 101    3 Jun 17 08:08 rabbit@taiga-async-rabbitmq.pid
drwxr-xr-x 10 100 101 4096 Jun 17 08:08 rabbit@taiga-async-rabbitmq-plugins-expand

It seems like there are volumes from an old borked installation which I tried to do last year. I am going to delete those volumes then.

I cannot upload the logs because they contain confidential information, but as I expected they have a bunch of emails for notifications in our project. Our installation does not use emails aside from registration (not even verification). So I guess I should look into disabling emails altogether.

4

Sorry, it was a typo, hostnames should be different: taiga-async-rabbitmq for taiga-async-rabbitmq service and taiga-events-rabbitmq for taiga-events-rabbitmq service.

According to your docker-compose file, your system is configured perfectly so you can delete the files of the vhosts that are no longer needed:

rm -rf rabbit@07d652149e8a*
rm -rf rabbit@960ae3a757d4*
rm -rf rabbit@abd9c6756945*
rm -rf rabbit@cbd6e6ca3476*
1 Like