RabbitMQ Administration

The stack includes native support for RabbitMQ.

Setup

To enable it, use the following configuration settings within Magento:

Host: queue1.i
Port: 5672

If you have multiple servers, the host value will change too - for example, stack2 will use host queue2.i instead.

A RabbitMQ admin user and virtualhost / is created on the stack by default - this user's credentials are available within the Sonassi control panel.

You should create a separate RabbitMQ user and virtualhost for each of the stores on your stack, then specify these in your site's env.php configuration file - this helps to ensure each store's queues are fully isolated from one another (Similar to using a different MySQL or Redis database):

    'queue' => [
        'amqp' => [
            'host' => 'queue1.i',
            'port' => '5672',
            'user' => 'example_user',
            'password' => 'example_password',
            'virtualhost' => 'example_virtualhost'
        ],
        'consumers_wait_for_messages' => 1
    ],

After editing your store's env.php, run bin/magento setup:upgrade --keep-generated to generate the queues within RabbitMQ.

---

Additional queue configuration (per queue)

Please note that on older versions of Magento (Below 2.4.5) you may need to configure each queue that you'd like to use with RabbitMQ individually in your store's env.php - otherwise, they'll still default to using MySQL instead.

A quick way to check if this is required is to view one of the queue's queue_topology.xml config files, for example:

cat vendor/magento/module-catalog/etc/queue_topology.xml
<?xml version="1.0"?>
<!--
/**
 * Copyright © Magento, Inc. All rights reserved.
 * See COPYING.txt for license details.
 */
-->
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework-message-queue:etc/topology.xsd">
    <exchange name="magento-db" type="topic" connection="db">
        <binding id="updateBinding" topic="product_action_attribute.update" destinationType="queue" destination="product_action_attribute.update"/>
        <binding id="updateBindingWebsite" topic="product_action_attribute.website.update" destinationType="queue" destination="product_action_attribute.website.update"/>
    </exchange>
</config>

If the exchange name value is magento-db like in the above snippet, you'll need to add this additional configuration - if it's just magento then you shouldn't need to.

For reference on the configuration to add, please see Magento's documentation at:

https://developer.adobe.com/commerce/php/development/components/message-queues/#change-message-queue-from-mysql-to-amqp

This includes an example for the product_action_attribute.update queue, but you'd need to add additional configuration sections for each of the queues your store is using (based on the output of bin/magento queue:consumers:list).

---

Topics vs consumers

Please note that there are some slight differences for the exportProcessor and codegeneratorProcessor queue consumers - for these, both of their queue->topics and queue->config->publishers values are import_export.export and sales_rule.codegenerator respectively, so your config for one of these would be similar to:

[...]
    'queue' => [
        'consumers_wait_for_messages' => 1,
        'amqp' => [
            'host' => 'queue1.i',
            'port' => '5672',
            'user' => 'username',
            'password' => 'password',
            'virtualhost' => 'domain'
        ],
        'topics' => [
            'import_export.export' => [
                'publisher' => 'amqp-magento'
            ]
        ],
        'config' => [
            'publishers' => [
                'import_export.export' => [
                    'connections' => [
                        'amqp' => [
                            'name' => 'amqp',
                            'exchange' => 'magento',
                            'disabled' => false
                        ],
                        'db' => [
                            'name' => 'db',
                            'exchange' => 'magento',
                            'disabled' => true
                        ]
                    ]
                ]
            ]
        ],
        'consumers' => [
            'exportProcessor' => [
                'connection' => 'amqp',
            ],
        ],
    ],
[...]

---

Accessing the administration panel

The RabbitMQ Administration panel can be accessed at https://queue.magestack.com

If you have multiple servers, you should access the server's respective RabbitMQ instance:

i.e. https://queue.2.magestack.com

! You must be connected to the VPN to reach this administration panel.

---

Supervisor

We've found that Magento's default consumers_runner cron task may not always behave as expected, with issues such as:

• Not always reliably starting queue consumers, leaving messages stuck in a Pending state
• Leaving older queue consumer processes running after a site deployment, which then use stack resources unnecessarily

If you see similar issues to these, we would recommend using Supervisor to manage RabbitMQ message queues instead of Magento's consumers_runner.

To configure Supervisor as an example (using domain group example and vhost example.com):

• Confirm the message queues that are currently active on your store:

  cd /microcloud/domains/example/domains/example.com/http
  bin/magento queue:consumers:list

• Open /microcloud/domains/example/domains/example.com/http/app/etc/env.php, and disable cron_run:

  'cron_consumers_runner' => [
      'cron_run' => false,
      'consumers' => [
      ]
  ],

• Install the supervisor package as the root SSH user:

  su -l root
  apt install supervisor

• Create configuration files for each queue - for example, for the product_action_attribute.update queue, create /etc/supervisor/conf.d/queue-product_action_attribute.update.conf with:

  [program:queue-product_action_attribute.update]
  process_name=%(program_name)s_%(process_num)02d
  directory=/microcloud/domains/example/domains/example.com/http
  command=/usr/bin/fakechroot /usr/sbin/chroot /microcloud/domains/example /usr/bin/php /domains/example.com/http/bin/magento queue:consumers:start product_action_attribute.update --max-messages=100
  autostart=true
  autorestart=true
  user=www-data
  redirect_stderr=true
  stdout_logfile=/microcloud/domains/example/domains/example.com/http/var/log/queue.log

  Replace the example domain group and example.com vhost in the program, directory, command, and stdout_logfile variables for your store as necessary.

  The process_name variable can safely be left as-is: this controls the name that's assigned to the process by Supervisor - for the configuration file above, this would be queue-product_action_attribute.update_00 (Where program_name is mapped to queue-product_action_attribute.update, and process_num is mapped to 00)

• Create additional configuration files for each of the message queues for your store using the configuration template above - update the file name, and the program and command variables for each new configuration file.

• Start Supervisor

  /etc/init.d/supervisor start

Supervisor will now manage your store's message queues, rather than Magento.

You can then use Supervisor's supervisorctl to view the status of queues, or restart them - for example:

su -l root
supervisorctl status
supervisorctl restart all

If you add any new message queues, you will need to add a new Supervisor configuration file too.

After you've created a new Supervisor configuration file, you'll then need to let Supervisor know about it - you can do this with:

supervisorctl update