API HTTP 500 on every PUT/POST

I have just completed barebones installation but am receiving errors from the API when I try to add/edit anything in the UI. It is not an issue with the wss port it is https://taiga.mydomain.com/api/v1/

I can receive successful responses back from the api directly so it seems that it is not an issue with the nginx proxy.

When I monitor the output of “docker logs taiga-docker_taiga-back_1” I receive this:

amqp.exceptions.AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Subject: [Django] ERROR (EXTERNAL IP): Internal Server Error:
 /api/v1/userstories
From: root@localhost
To: example@example.com
Date: Wed, 20 Mar 2024 06:09:58 -0000
Message-ID: <171091499829.39.16516801897609192457@cdc8d7eebb8e>

Internal Server Error: /api/v1/userstories

Exception Value: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.
Request information:
USER: user01
-------------------------------------------------------------------------------
ERROR:2024-03-20 06:09:58,272: Internal Server Error: /api/v1/userstories
Traceback (most recent call last):
  File "/opt/venv/lib/python3.11/site-packages/kombu/utils/functional.py", line 30, in __call__
    return self.__value__
           ^^^^^^^^^^^^^^
AttributeError: 'ChannelPromise' object has no attribute '__value__'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.11/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/taiga-back/taiga/base/api/viewsets.py", line 95, in view
    return self.dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/taiga-back/taiga/base/api/views.py", line 449, in dispatch
    response = self.handle_exception(exc)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/taiga-back/taiga/base/api/views.py", line 447, in dispatch
    response = handler(request, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/contextlib.py", line 80, in inner
    with self._recreate_cm():
  File "/opt/venv/lib/python3.11/site-packages/django/db/transaction.py", line 290, in __exit__
    connection.set_autocommit(True)
  File "/opt/venv/lib/python3.11/site-packages/django/db/backends/base/base.py", line 420, in set_autocommit
    self.run_and_clear_commit_hooks()
  File "/opt/venv/lib/python3.11/site-packages/django/db/backends/base/base.py", line 653, in run_and_clear_commit_hooks
    func()
  File "/taiga-back/taiga/timeline/signals.py", line 28, in <lambda>
    connection.on_commit(lambda: push_to_timelines.delay(project_id,
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/celery/app/task.py", line 425, in delay
    return self.apply_async(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/celery/app/task.py", line 575, in apply_async
    return app.send_task(
           ^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/celery/app/base.py", line 788, in send_task
    amqp.send_task_message(P, name, message, **options)
  File "/opt/venv/lib/python3.11/site-packages/celery/app/amqp.py", line 510, in send_task_message
    ret = producer.publish(
          ^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/kombu/messaging.py", line 177, in publish
    return _publish(
           ^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/kombu/connection.py", line 523, in _ensured
    return fun(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/kombu/messaging.py", line 186, in _publish
    channel = self.channel
              ^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/kombu/messaging.py", line 209, in _get_channel
    channel = self._channel = channel()
                              ^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/kombu/utils/functional.py", line 32, in __call__
    value = self.__value__ = self.__contract__()
                             ^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/kombu/messaging.py", line 225, in <lambda>
    channel = ChannelPromise(lambda: connection.default_channel)
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/kombu/connection.py", line 895, in default_channel
    self._ensure_connection(**conn_opts)
  File "/opt/venv/lib/python3.11/site-packages/kombu/connection.py", line 433, in _ensure_connection
    return retry_over_time(
           ^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/kombu/utils/functional.py", line 312, in retry_over_time
    return fun(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/kombu/connection.py", line 877, in _connection_factory
    self._connection = self._establish_connection()
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/kombu/connection.py", line 812, in _establish_connection
    conn = self.transport.establish_connection()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/kombu/transport/pyamqp.py", line 201, in establish_connection
    conn.connect()
  File "/opt/venv/lib/python3.11/site-packages/amqp/connection.py", line 329, in connect
    self.drain_events(timeout=self.connect_timeout)
  File "/opt/venv/lib/python3.11/site-packages/amqp/connection.py", line 525, in drain_events
    while not self.blocking_read(timeout):
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/amqp/connection.py", line 531, in blocking_read
    return self.on_inbound_frame(frame)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/amqp/method_framing.py", line 53, in on_frame
    callback(channel, method_sig, buf, None)
  File "/opt/venv/lib/python3.11/site-packages/amqp/connection.py", line 537, in on_inbound_method
    return self.channels[channel_id].dispatch_method(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/amqp/abstract_channel.py", line 156, in dispatch_method
    listener(*args)
  File "/opt/venv/lib/python3.11/site-packages/amqp/connection.py", line 667, in _on_close
    raise error_for_code(reply_code, reply_text,
amqp.exceptions.AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.

Any suggestions on where to even look to figure out what is happening here would be appreciated!

Thanks,
J

I noticed an additional error in docker logs taiga-docker-taiga-events-rabbitmq-1

AMQPLAIN login refused: user 'taiga' - invalid credentials
2024-03-20 15:05:59.515 [info] <0.721.0> closing AMQP connection <0.721.0> (192.168.240.8:42036 -> 192.168.240.6:5672)
2024-03-20 15:05:59.516 [info] <0.723.0> Closing all channels from connection '192.168.240.8:42036 -> 192.168.240.6:5672' because it has been closed
2024-03-20 15:05:59.519 [info] <0.726.0> accepting AMQP connection <0.726.0> (192.168.240.8:42052 -> 192.168.240.6:5672)
2024-03-20 15:05:59.522 [error] <0.726.0> Error on AMQP connection <0.726.0> (192.168.240.8:42052 -> 192.168.240.6:5672, state: starting):

Since I’m just trying to get things working in general I haven’t made any changes to the .env file

# Taiga's RabbitMQ settings - Variables to leave messages for the realtime and asynchronous events
RABBITMQ_USER=taiga  # user to connect to RabbitMQ
RABBITMQ_PASS=taiga  # RabbitMQ user's password
RABBITMQ_VHOST=taiga  # RabbitMQ container name
RABBITMQ_ERLANG_COOKIE=secret-erlang-cookie  # unique value shared by any connected instance of RabbitMQ

Hi @jasonbdn

It seems that the backend, when it is going to send the asynchronous task that generates the new entry in the timeline, cannot connect to rabbitmq.

File "/taiga-back/taiga/timeline/signals.py", line 28, in <lambda>
    connection.on_commit(lambda: push_to_timelines.delay(project_id,
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(...)
  File "/opt/venv/lib/python3.11/site-packages/celery/app/base.py", line 788, in send_task
    amqp.send_task_message(P, name, message, **options)
(...)
  File "/opt/venv/lib/python3.11/site-packages/amqp/connection.py", line 667, in _on_close
    raise error_for_code(reply_code, reply_text,
amqp.exceptions.AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.

The error message is

(403) ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.

Check your settings value for CELERY_BROKER_URL (and EVENTS_PUSH_BACKEND_OPTIONS could be wrong too). And you can review the log files of rabbitmq too to get more info about the cause of the error during the authentication.

Best regards

Remove the containers of rabbitmq and relauch the docker compose again.