High Celery CPU Usage in taiga-back-async

initiativist · August 10, 2023, 5:36am

Hi Daniel,

Discourse is quite cool, I’m going to look into it for my own purposes.

Most of the RabbitMQ steps I followed are outlined under the link you provided, no new information there.

After a significant amount of troubleshooting, I think I’ve found the issue relating to high CPU usage.

It wasn’t the telemetry after all, as you’ve indicated you found improbable. The reason why commenting it out stopped the issue from occurring was because I neglected to put pass in the function and so it was failing to work in the first place. ah well.

As for the actual solution to the issue.

I started by logging all the task executions and monitoring with flower. I quickly found that none of the native tasks were to blame for the high CPU usage.

I used py-spy to record the time spent in each function and found close_open_fds() to be taking up 100% of the CPU time.

see:


[user@host taiga-docker]# py-spy top --pid <pid>                                                                              
                                                                                                                                                                                    
Collecting samples from '/opt/venv/bin/python /opt/venv/bin/celery -A taiga.celery worker -B --concurrency 4 -l DEBUG -E' (python v3.11.4)                                          
Total Samples 1900                                                                                                                                                                  
GIL: 60.00%, Active: 100.00%, Threads: 1                                                                                                                                            
                                                                                                                                                                                    
  %Own   %Total  OwnTime  TotalTime  Function (filename)                                                                                                                            
100.00% 100.00%   19.00s    19.00s   close_open_fds (billiard/compat.py)                                                                                                            
  0.00% 100.00%   0.000s    19.00s   start (celery/worker/worker.py)                                                                                                                
  0.00% 100.00%   0.000s    19.00s   run (celery/beat.py)                                                                                                                           
  0.00% 100.00%   0.000s    19.00s   main (click/core.py)                                                                                                                           
  0.00% 100.00%   0.000s    19.00s   _Popen (billiard/context.py)                                                                                                                   
  0.00% 100.00%   0.000s    19.00s   __init__ (billiard/popen_fork.py)                                                                                                              
  0.00% 100.00%   0.000s    19.00s   start (celery/bootsteps.py)                                                                                                                    
  0.00% 100.00%   0.000s    19.00s   _launch (billiard/popen_fork.py)                                                                                                               
  0.00% 100.00%   0.000s    19.00s   main (celery/bin/celery.py)                                                                                                                    
  0.00% 100.00%   0.000s    19.00s   main (celery/__main__.py)                                                                                                                      
  0.00% 100.00%   0.000s    19.00s   invoke (click/core.py)                                                                                                                         
  0.00% 100.00%   0.000s    19.00s   __call__ (click/core.py)                                                                                                                       
  0.00% 100.00%   0.000s    19.00s   caller (celery/bin/base.py)                                                                                                                    
  0.00% 100.00%   0.000s    19.00s   <module> (celery)                                                                                                                              
  0.00% 100.00%   0.000s    19.00s   start (billiard/process.py)                                                                                                                    
  0.00% 100.00%   0.000s    19.00s   worker (celery/bin/worker.py)                                                                                                                  
  0.00% 100.00%   0.000s    19.00s   _bootstrap (billiard/process.py)                                                                                                               
  0.00% 100.00%   0.000s    19.00s   new_func (click/decorators.py)

That led me to this bug:

github.com/celery/celery

Celery worker using beat, stays at 100% cpu for long time when used in docker

opened 07:19PM - 10 Jun 23 UTC

vladmihaisima

Issue Type: Bug Report

When trying to invoke worker with beat (celery -A tasks worker -B), it starts bu…t one thread stays at 100% processor and the beat worker does not log anything. See minimal reproducing steps below, which I extracted from a larger project. The root cause is: when started in a docker environment where SC_OPEN_MAX is very large (my current docker sets it to 1073741816 by default) the function close_open_fds invoked at https://github.com/celery/celery/blob/5.2/celery/beat.py#L715 , which invokes https://github.com/celery/billiard/blob/v4.1.0/billiard/compat.py#L133 , will iterate through the 1 billion file descriptors before returning (so technically might finish but I did not wait longer than couple of minutes). While the root cause is in billiard, I realized that while trying to debug the celery issue, so might help other people having similar issues. I consider this a bug, as having a very large total possible number of open files should not incur a starting penalty of minutes - other mechanisms for closing potentially open files could be used (like iterating over /proc/self/fd). You can use the script below to reproduce: ``` #!/bin/bash # Start a broker docker run -d --network host -p 6379:6379 redis # Create a task file cat > tasks.py <<EOF from celery import Celery app = Celery('tasks', broker='redis://localhost') @app.task def add(x, y): return x + y EOF # Try to start celery worker with -B # This will keep 1 cpu at 100% and not display 'beat: Starting...'. # Note that while I set the ulimit to unreasonable values, my system # had this by default, this is just to make a reproducible case docker run \ --network host \ --ulimit nofile=1073741816:1073741816 \ -v $PWD:$PWD \ python:3.11 \ bash -c "pip3 install celery[redis]; cd $PWD; celery -A tasks worker -B --loglevel=DEBUG" # Workaround (which results in celery displaying immediately 'beat: Starting...') is to reduce nofile to reasonable values: # docker run \ # --network host \ # --ulimit nofile=2048:2048 \ # -v $PWD:$PWD \ # python:3.11 \ # bash -c "pip3 install celery[redis]; cd $PWD; celery -A tasks worker -B --loglevel=DEBUG" ```

The temporary solution I adopted was to alter taiga-back/docker/async_entrypoint.sh as follows:

#!/usr/bin/env bash

# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
#
# Copyright (c) 2021-present Kaleidos Ventures SL

set -euo pipefail

# Give permission to taiga:taiga after mounting volumes
echo "Give permission to taiga:taiga"
chown -R taiga:taiga /taiga-back

# Start Celery processes
echo "Starting Celery..."
exec gosu taiga celery -A taiga.celery worker \
    --concurrency 4 \
    -l DEBUG \
    -E \
    "$@" &

exec gosu taiga celery -A taiga.celery beat -l DEBUG

certainly the -l DEBUG and -E commands aren’t necessary, just the separation of the beat and the workers.

I hope this is helpful for you and your team.

Topic		Replies	Views
Unexpected CPU usage by RabbitMQ (beam.smp) under no load Self hosted Taiga docker	17	5606	December 11, 2023
Issues with IPV6 and taiga async docker Self hosted Taiga	2	462	October 4, 2023
High cpu load, and database not found Self hosted Taiga issues , docker	1	262	January 2, 2024
Installed Taiga according to the instructions, but it does not work. Error 500 Self hosted Taiga	20	3639	December 6, 2024
Different problem after installation from source code Self hosted Taiga	12	2748	January 17, 2024

High Celery CPU Usage in taiga-back-async

Related topics

Get Taiga for free