Index Management

In Magento there is a complex relationship between the data stored in the admin and what is presented on the frontend.

  • In the admin, you see non-indexed data, it is always up to date
  • On the frontend, you see indexed data, a snapshot in time of the product catalogue from when it was last indexed

The admin is uses non-indexed data so you can manage your catalogue and see accurate information at all times, but this comes with a penalty that loading non-indexed data takes substantially more time than loading indexed data. So on the frontend, it uses the much faster, indexed data - the penalty is that these indexes must be periodically updated.

Mode

Three modes exist for Magento indexes,

  • Manual
  • Update on save
  • Update on schedule (EE only)

The impact of using "Update on save" is extremely substantial, the effect is that every time a product changes (via API/admin/checkout etc.) - a number of critical areas of the frontend will cease to function until the reindex is complete. This means on every product change action - search will stop working, layered navigation will stop working and checkout will stop working.

Production stores

Only "Product Prices" and "Stock Status" should be set to "Update on save", all others must be manual. A cron should be configured to re-run indexes at an appropriate non-production hour.

The following will trigger a reindex of all indexes at 5am each day (you can adjust the command/time to suit your requirements).

0 5   * * * ( echo "cd /domains/example.com/http/shell/; php indexer.php reindexall" ) | /usr/bin/fakechroot /usr/sbin/chroot /microcloud/domains/example /bin/bash

Development/Staging stores

You may set them all to "Update on save"

Warning on manual reindexing

A facility exists in the Magento admin under Admin > System > Index Management and via SSH under ./shell/indexer.php to manually force a reindex to occur.

Under no circumstances should these facilities ever be used during production hours

Performing a manual reindex will cause immediate and prolonged downtime; your store will immediately cease to function and continue to be in this state until the reindex process has completed. For small catalogues, this can be a matter of minutes, for large catalogues, it could be hours.

The reindex time alone is also not just the issue. After a reindex starts, customer frontend requests being to queue and backlog, so even after the reindex completes, it can take minutes/hours for the backlog to clear. This compounding effect is what causes stores to be offline for several hours.

If you must reindex during production hours then put the store into maintenance mode first, then run the reindex, then remove maintenance mode on completion.

You can visualise the impact in real world terms, both for performance and sales, by reviewing the Kibana Local Performance dashboard.

Admin notices

One or more of the Indexes are not up to date. Click here to go to Index Management and rebuild required indexes.

You may often see this notice displayed at the top of your admin panel. This is an indicator that the data you see in the Magento admin differs from the data displayed on the frontend of the store, it is not a cause for alarm. You can and should disregard this message and not take any action.

3rd party indexing modules

It can appear desirable to "fix" this behavior in Magento by installing a 3rd party module. Unfortunately, this does not work and will lead to considerably more problems.

Popular extensions for asynchronous reindexing cause significant issues including erratic performance, instability and regular downtime.

Under no circumstances should you ever install a 3rd party module for reindexing. The most optimal approach and settings are those defined above; setting indexes to "Manual" with a cron configured for daily out-of-hours bulk reindex.