[4.x]: Very slow CMS and content management (with Neo and SuperTable)

[janreges]

What happened?

Hi,

for our projects we use Craft CMS (PRO) and usually we need the client to compose the individual pages himself from the individual visual blocks. We implemented the management of the structured content of these blocks using Neo and SuperTable plugins.

Note: I am creating this ticket in the Craft CMS, Neo and SuperTable GitHub repositories. I am interested in the opinion of the authors and architects of Craft CMS and these plugins. After investigating these performance issues, it is evident that some optimizations and solutions to these problems can be implemented in the Craft CMS code and then subsequently in the individual plugins.

Facts about our project

• 209 fields in Settings -> Fields section - 20 of them are of Asset type (typically images for visual components). We have all assets in only one common volume, which is on one local filesystem.
• 1 Neo field for the content editor, which contains 142 block types (65 are first level blocks and 77 are subblocks)
• 6 fields are of type SuperTable and these are also used within the Neo block types
• We have Craft CMS, Neo and SuperTable plugins in the latest versions

Issues we have and perceive from the Craft CMS perspective

• When a page containing the ahead mentioned Neo block is created or edited, the form takes 2.8 seconds to generate on the backend - for this, 2064 database queries are made, 123 MB of memory is allocated, 28127 events are generated and the log contains 4639 records. This is, of course, a huge overhead and makes the CMS difficult for users to use.
• Note: the duration times above are from a locally dockerized project on a powerful Ryzen processor (HEDT). PHP-FPM has an active opcache and an optimized composer autoload dump. If running on server CPUs (e.g. Intel Xeon E5-2699v4), the times are 2x worse. The reason is that the performance of the mentioned server CPU per 1 core is half compared to HEDT and the server has only 2400 MHz DDR4 ECC memory compared to 3600 MHz on the tested PC. In the case of the production server, it takes 6 seconds to load the form to create or edit a page, so it is understandably frustrating and very annoying.

Our investigation of the problems and findings

• If the SuperTable plugin is disabled, the form loads in 1.5 seconds, the number of DB queries is reduced from 2064 to 673, the memory allocation is reduced from 123 to 71 MB, the number of events is reduced from 28127 to 14352, and the number of logs is reduced from 4639 to 2409.
• If both SuperTable and Neo plugins are disabled, the form will load in 0.3 seconds, the number of DB queries will be reduced to 114, memory allocation to 17 MB, the number of events will be reduced to 1338 and logs to 297.
• So it is quite evident that if you have a larger number of fields and use both Neo and SuperTable, extreme and very often excessive numbers of DB queries, events and logs are executed, causing huge overhead. And unfortunately this happens even when loading a form to create a new page.

How can you help us

We would like to hear an informed opinion from the authors/architects of the existing code on the possible causes of these problems, but our investigation of the existing code shows that:

• The Neo and SuperTable plugins, and perhaps even the Craft CMS level fields, lack some form of internal instance cache to ensure that even if a field is used within a form dozens or hundreds of times within a single run of a PHP script (e.g. in Neo/SuperTable blocks), that they are only initialized once. That is, even if an instance of a field/element is instantiated hundreds of times in a run of a single instance of a PHP script, that initialization and retrieval of information from the DB or filesystem is done only once.
• Unfortunately, the Neo plugin is not tailored to higher tens or hundreds of block types. Editing such a Neo field (prescription of block types) freezes JavaScript in the browser for tens of seconds after loading. Managing and configuring these types of blocks is only done by the developers on the DEV environment, so the client doesn't come into contact with it, but it is annoying.
• In Craft CMS code, certain methods of some classes/services are called hundreds or thousands of times within one instance of PHP script execution - it depends on how one models content management. For example, we found that in our example, when creating a form to manage a page, the getRootFolderByVolumeId($volumeId) method is called over 1,000 times, causing over 1,000 identical queries to the database. We have therefore implemented a simple composer patch below that will reduce the number of related identical duplicate DB queries from 1,000 to units and likewise reduce the number of events or logs by thousands. A sample of this patch is at the end of this ticket. Even this one small optimization sped up page generation with page editing by 15-30%. However, this patch does not address the situation where the volume settings are modified as part of the script run, so it is not an ideal solution.
• For the Neo or SuperTable plugins, a form of "lazy loading" would be useful in some critical places, both at the block type configuration level (in the Settings section) and within the page content forms. Even though the user only uses 0 to 10% of the content types within the management of these blocks or content management on a given page, a complete initialization of all of them is performed with each request.
• The Craft CMS as a whole lacks a sophisticated cache that would be built on, for example, tagging the cache and then also have selective cache invalidation based on the tags. Before switching to Craft CMS, we used our own CMS built on the Nette framework, which also had a great taggable cache that worked with both storing to a local filesystem, e.g. Redis (https://doc.nette.org/en/caching#toc-invalidation-using-tags). In Craft CMS and YII, the cache options seem to be only expiration-based, which is insufficient for a large part of the scenarios inside the CMS.
• Because for most projects that use Craft CMS, the CMS administration part (Settings, fields, sections, etc.) is completely disabled on production projects (CMS configuration happens more on DEV environments and is versioned). Therefore, it is completely unnecessary to have a series of DB queries, event/log throws related to loading something that is practically unchangeable in the production CMS, while doing normal content management on most forms in the CMS. The invalidation of this cache could occur by default in cache-flush commands within deployments. However, for this cache to be truly meaningful, it is not just about caching DB queries, but rather entire serialized instances of some objects that must be created over and over again for most requests in the CMS during normal operation and take tens or hundreds of milliseconds.

How to proceed?

We really like the Craft CMS and a number of useful plugins and we are trying to promote them. Unfortunately we have now run into some very fundamental performance issues.

So the first thing we thought of was to start investigating these causes ourselves, think of and implement some optimizations and then send pull requests.

As authors, do you please have an opinion on how to solve this situation? We'd be happy to help with optimizations, but if any of the Craft CMS architects have already had some thoughts and ideas on these improvements, we'd like to know so we don't do double work. This will also increase the chances that even if we design and implement some specific optimizations ourselves, they will be incorporated into future versions faster and without major comments.

You may also be self-aware that, based on some historical architectural decisions, some forms of efficient caching are now not applicable at all. For example, I imagine that serializing some complexly loaded objects into the cache won't work because the architecture requires that lifecycle of their initialization throws a number of events on which other functionality depends.

I would be very grateful for any thoughts and ideas on how to grasp and implement these necessary optimizations.

If needed, I can also provide a complete configuration of our CMS, including a database dump to your e-mail address.

Thank you very much for your time and support.

Patch getRootFolderByVolumeId()

--- /dev/null
+++ ../src/services/Assets.php
@@ -549,10 +549,18 @@
      */
     public function getRootFolderByVolumeId(int $volumeId): ?VolumeFolder
     {
-        return $this->findFolder([
-            'volumeId' => $volumeId,
-            'parentId' => ':empty:',
-        ]);
+		static $cache = [];
+		if (isset($cache[$volumeId])) {
+			return $cache[$volumeId];
+		}
+
+		$folder = $this->findFolder([
+			'volumeId' => $volumeId,
+			'parentId' => ':empty:',
+		]);
+		$cache[$volumeId] = $folder;
+
+		return $folder;
     }

     /**

Craft CMS version

4.4.13

PHP version

8.1.18

Operating system and version

Debian 11 Bullseye

Database type and version

MariaDB 10.11.2

Image driver and version

GD 8.1.18

Installed plugins and versions

• Neo 3.7.9
• Supertable 3.0.9
• Redactor 3.0.4
• Typed linked field 2.1.5
• Other plugins (not much relevant to these performance issues - Blitz, Field Manager, Retour, Formie, Feed Me, Navigation, User Activity, SEOmatic, Icon Picker)

[brandonkelly]

Thanks for the getRootFolderByVolumeId() suggestion! I’ve made an optimization to that for Craft 4.5 via 8305fbe.

Would you mind sharing your config/project/ folder and Composer files with us? With those in place we could recreate your setup and run it through Blackfire to look for other optimization opportunities.

The Neo and SuperTable plugins, and perhaps even the Craft CMS level fields, lack some form of internal instance cache to ensure that even if a field is used within a form dozens or hundreds of times within a single run of a PHP script (e.g. in Neo/SuperTable blocks), that they are only initialized once. That is, even if an instance of a field/element is instantiated hundreds of times in a run of a single instance of a PHP script, that initialization and retrieval of information from the DB or filesystem is done only once.

Worth noting that this is generally already the case.

Unfortunately, the Neo plugin is not tailored to higher tens or hundreds of block types. Editing such a Neo field (prescription of block types) freezes JavaScript in the browser for tens of seconds after loading.

For the Neo or SuperTable plugins, a form of "lazy loading" would be useful in some critical places, both at the block type configuration level (in the Settings section) and within the page content forms. Even though the user only uses 0 to 10% of the content types within the management of these blocks or content management on a given page, a complete initialization of all of them is performed with each request.

Both of those field types are heavily inspired by Matrix. We are working to improve Matrix scalability in Craft 5, and would expect that those plugins will end up following our lead to some extent.

The Craft CMS as a whole lacks a sophisticated cache that would be built on, for example, tagging the cache and then also have selective cache invalidation based on the tags.

Yii does in fact have robust cache invalidation, including tags. Craft uses tag dependencies for various things, such as front-end {% cache %} tags and GraphQL queries.

Because for most projects that use Craft CMS, the CMS administration part (Settings, fields, sections, etc.) is completely disabled on production projects (CMS configuration happens more on DEV environments and is versioned). Therefore, it is completely unnecessary to have a series of DB queries, event/log throws related to loading something that is practically unchangeable in the production CMS, while doing normal content management on most forms in the CMS. The invalidation of this cache could occur by default in cache-flush commands within deployments. However, for this cache to be truly meaningful, it is not just about caching DB queries, but rather entire serialized instances of some objects that must be created over and over again for most requests in the CMS during normal operation and take tens or hundreds of milliseconds.

Caching entire objects can be error-prone, so we’ve generally avoided it, but you may be right. It’s worth looking into.

[janreges]

@brandonkelly - thank you very much for your reply and for quick optimization in getrootFolderByVolumeId().

I have already sent you an e-mail with composer.json and the complete contents of the project/config folder.

Considering that you have a great insight into the internal architecture of Craft CMS, I believe that you will be able to quickly find a couple of essential places to optimize.

In my opinion, the biggest quick-win can be the implementation of "field instance cache" for a place in the code with initialization/factory of fields, which are called very intensively and repeatedly even from Matrix/Neo/SuperTable.

Thank you also for referring to better options for working with the cache inside Yii. It's great that tagging support is also there and we will use it in our applications/websites. Due to scaling, it would be great if the tagging cache and ideally also the cache of entire PHP class instances were also used within the CMS. For really large projects, we use replicated databases and containerization, where the database does not run locally, so every saved DB query or CPU-cycle within PHP can represent a significant saving. Due to scaling, replicated databases or Redis clusters are often used, which also do not run locally with PHP, but on an another server. Therefore, using Redis only for DB query cache may not bring significant improvement (network latency is still there). This is the reason why, in some cases, serialized instances of ready-made classes can be stored in the cache, which had to perform a number of DB queries and other CPU/memory intensive operations.

If you import our project configuration into the CMS and see how slow the "Content editor" type page management forms are, I believe you will understand. I believe that, as an architect, you will realize very quickly that the high tens of percent of this overhead, which slows down most requests, can be efficiently cached and will not necessarily be performed for all pages/forms within the CMS movement.

Thank you very much for putting your efforts to this optimization.

[domstubbs]

I was asked to take a look at a site exhibiting similar issues this week. The developers have implemented a Neo content builder field with around 40 block types and performance in the control panel is very poor. They’ve implemented sufficient caching on the frontend that it’s fine from an end user perspective, but content managers are understandably frustrated with 5-15s load times when editing entries.

I’ve tried profiling some requests to identify obvious bottlenecks/repetition but there doesn’t seem to be any single thing that’s holding things up – more general inefficiencies.

In our case, a typical entry edit page load results in around 1k queries (of which nearly half are duplicates) and nearly 15k events logged.

If an additional test case would be useful please let me know and I can share a composer.json and project config as well.

One detail that I’m a little vague on – is Craft keeping an internal query cache per-request and returning results from that when duplicate queries are received? I assume not?

I’d also love to see support for extended caching at deploy time, as allowAdminChanges will always be disabled in prod and staging envs, so you’d hope that a lot of the queries relating to field structure could be compiled once and then safely eliminated from individual entry edit requests.

[brandonkelly]

@domstubbs Not sure why so much time is being spent in Guzzle. CP requests should generally not be making any HTTP requests directly. So it’s worth looking into why that’s happening.

The excessive time spent in Composer\Autoload\IncludeFile seems to indicate that the classmap isn’t optimized. You can fix that by running composer dump-autoload -o.

Beyond that, yes there’s definitely room for improvement on the caching front as @janreges pointed out, as well as generally finding ways to reduce the per-screen content complexity – which is one of our main areas of focus for Craft 5.

[domstubbs]

Thanks Brandon. I’ve just been debugging in a dev environment, hence the non-optimised classmap. I did start to dig into the Guzzle requests but once I saw they were tied to AWS Assets I assumed it was typical. I’ll take another look to see if I can see anything there.

If I disable the AWS plugin the Guzzle activity disappears but the response times don’t significantly change, so whatever it is it’s not a silver bullet.

That sounds great about Craft 5 – I look forward to giving it a try.

[domstubbs]

Just to draw a line under the Guzzle mystery, the site has some Neo fields with icons and their Twig svg() call is what’s generating the S3 requests.

Removing the svg() call is turning a 4.5s request into a 3s request, so I’ll see there might be an easy <img> replacement to be found in Neo.

[brandonkelly]

Huh, so the SVGs must have remote image references within them.

[domstubbs]

Nothing that obscure – the block icon SVGs are themselves on an S3 volume, so they’re inadvertently adding 40 HTTP requests to every entry edit page (since there are ~40 block types). I had a look at refactoring Neo to use cacheable <img> tags but it wouldn’t have been a quick change. Fortunately moving the icons volume from S3 to a local folder is an easy alternative and seems to deliver the same performance boost.

[adrianjean]

I have experienced this same behaviour on the CP side. On the front end I can optimize through twig and eager-loading, so no issue there. It's on the Admin / CP side that's slowest.

• I used to use Matrix -> SuperTable -> Matrix — but found that really slow on pages with several rows/cols/blocks.
• Then I converted to Neo -> SuperTable — found it slightly faster because Neo was handling a layer of structure, but still quite slow on anything but a simple page.
• I'm in the process of converting now to Neo -> Individual Fields — I notice that the Debug Toolbar reports half the DB calls but maybe only a slight (10%?) reduction in time spent both loading the page and saving an entry — with saving an entry being still quite slow.

I did notice that while saving an entry takes a while, when I edit an entry I see the temporary draft is saved rather quickly. Not sure if any of this helps.

Looking forward to what CraftCMS 5 has to offer!

[brandonkelly]

I did notice that while saving an entry takes a while, when I edit an entry I see the temporary draft is saved rather quickly. Not sure if any of this helps.

If that happens without any form edits, are you seeing a field get the blue “edited” status indicator?

[adrianjean]

(Late follow-up)
I do see the indicator, but this is with form edits.

(Update)

• Converting to Neo -> Individual Fields did improve page save time a little.
• I'm now building all new sites with CraftCMS 5.x and a new content builder with Neo -> CKEditor -> Entry Blocks and I find it better.

[brandonkelly]

@adrianjean Are you still getting the automatic draft creation issue?