TL; DR: Config overrides seem very handy, but can be pretty problematic when used for config entities. If you are using them in this way, you should make very sure that there aren't any unintended side effects. And if you maintain a contrib module that defines a config entity type, you should make sure they're safe to use with config overrides.

Fair warning: This blog post is mainly written for developers. While, ideally, site builders who want to use configuration overrides should also understand the inherent problems they come with, I don't feel it's possible to really explain those problems without getting pretty technical.

What are config overrides?

If you're not familiar with Drupal 8's concept of configuration overrides, I suggest you quickly read up on it in the documentation on drupal.org. In short, it allows you to define overrides for configuration (who would have thought?) in your settings.php, making it easy to have different settings for different environments – very common, of course, if you have separate development or testing environments.

The way those overrides work should be properly understood, though: the overrides exist solely in the code, are read into a global variable at the start of the page request and then, when a configuration item is accessed during a page request that global variable is checked for overrides to that specific configuration item. In other words, the config values in the database (or the config storage, if that's not the database in your case) should never contain the overridden values, those should only ever be present in settings.php.

To facilitate this, config items when normally loaded are always read-only – that way, the overrides loaded with them won't be written back to the database. If you want to edit some config values, you need to explicitly load the config item with \Drupal::configFactory()->getEditable('your_config.name'); to obtain a writable copy – which then will come without any overrides. This ensures that, as said before, no overrides will be written back to the storage.

Then what's the problem?

While this getEditable() requirement for editing configuration works pretty well, there is one big problem: this requirement, and whole "read-only" system, is just not there for config entities! When you load a config entity, even though it's basically just a config item with a class, it will always be writable – there isn't even a way to explicitly get a read-only version. There is loadOverrideFree() on ConfigEntityStorage, which you should absolutely use when making any chances to a config entity – but who really knows that? (I didn't, and I hope for reasons of vanity that I'm not the only one.)

The smart people working on Drupal Core of course quickly realized this. Their solution: on admin pages (which are already categorized as such to allow having an "admin theme"), where entities are usually edited, always load the override-free version of the entity for the route controller. That way, the edit will really only affect the values in the storage, and the overrides won't be touched at all. This leads to a slight UX WTF, but is generally a good solution, which works in a lot of cases. The problem, though, is that it's far from working for all of them. Also, since it's working in many cases, we might feel it's less urgent to point this potential problem out to contrib module developers and site builders.

So, what problems remain?

Nice of you to ask! In my opinion, there are still several remaining problems. While I'm not sure that all of them could even be fixed in Core, and it might certainly take a while (as it nearly always does – especially for such complicated issues), I think the main thing is to at least educate developers and site builders alike that these problems exist, and maybe how to spot them. Here is an overview of all problems I'm aware of:

Config entities aren't always edited on admin pages

While applying this fix for all admin pages is certainly a good heuristic, I'm pretty sure there are contrib modules out there that save config entities on non-admin pages. When that happens, the user will (unlike anywhere else in Drupal 8) actually see the overridden values while editing, and they will be saved back to the database/storage, which we don't want.

So, if you maintain a contrib module, be sure to always load the override-free version of a config entity before editing and saving it! This especially applies to Drush commands, where the generic fix for admin pages of course doesn't apply – unless you explicitly request it otherwise, you'll always get the overridden entity there.

Admin pages don't always load config entities just for editing

The reverse is also true: sometimes, admin pages will just show information about a certain config entity, not allow any editing. In this case, currently, the overridden values will not be displayed, potentially confusing users. In some cases, they'll not be able to see anywhere that the overridden values are actually applied correctly.

More of a problem, though, are other non-editing uses of config entities, which was one of the two main problems we faced in the Search API module (see Issue #2682369: Fix problems with overridden config entities): in our case, nearly all of the module's UI (except most searches) is actually on admin pages, and will therefore (as long as the config entity is loaded directly by the route controller – also a peculiar source of potential confusion, see next heading) use the override-free version. While this is great for the edit pages, it's disastrous for the other functionality available there – "Index now", "Clear indexed data", etc., would all use the entities without overrides. If you used config overrides to make the index read-only, or to attach them to a different search server, this would mean your production site's server could be corrupted or cleared by actions on your development site – not very good, in short.

Luckily, once you're aware of this problem, there's actually a pretty easy solution: you can just use the with_config_overrides parameter option in your MODULE.routing.yml file to get the overridden version of a config entity even on an admin page. That way, on pages where you want to do something else than edit/save the config entity, you'll get the correct version of the entity, with the effective values.

It's slightly more complicated on pages where you can both edit an entity and use it in some other way. In that case, simply choose one of the two to request via the route controller and then manually re-load the other version, with/without overrides, when appropriate.

Entities aren't always loaded by the route controller

It's important to note that the mentioned fix for admin pages is installed (please correct me here if I'm wrong) in the route controller. So, when the config entity in question is part of the URL (for instance, admin/structure/types/manage/<var>{node_type}</var>), it will (by default) be loaded without overrides, but loading it normally within a page callback or form builder method will yield the normal entity, with overrides. This is mostly not even a bug, but a feature: for most entity edit pages, the entity is in the URL, while (for instance) overview pages will do a bulk-load and get the overridden values.

However, in combination with the previous item, about displaying config entities on admin pages, it could lead to confusing situations for users: an overview page for the entities would use the overridden values, while clicking through to a "View" page for one of them would suddenly show the override-free version of that entity. And since (see the UX WTF above) we never actually show any indication in the UI about which values are overridden, the user could actually wonder which of them are the effective ones.

The really complicated problem

Up to now, the main problem was just being aware of the various problems that could occur. Once you're aware that they exist, they are all actually pretty easy to resolve – just see whether or not you save a config entity, and in either case make sure you load the appropriate version of it.

However, there is one last problem with overridden config entities, and this one is a real doozy – much too complex to put in a heading, for sure. I'd say that it will luckily only affect a minority of modules dealing with config entities, but where it strikes it's hard to spot, and might be even harder to fix. It's also what caused most of the work in the Search API issue mentioned above.

Let's take the Search API as the example and suppose you have an index with an overridden server: "server1" in the database, "server2" as the overridden value. Now, you edit the index and change something else – the description, for example. For editing, of course, the override-free entity version is used, all correct. Upon saving, though, the storage controller loads the previous entity version as the "original" – but this time with overrides. So, even though only the description was changed, from within hook_entity_update() and the entity's postSave() method it will look like the server was changed from "server2" to "server1". In this case, the Search API will remove all of the index's data from "server2" (where it should have remained) and initialize data structures for it on "server1" (where they won't be used), probably breaking all searches for that index to at least some extent (depending on backend).

Similar problems exist for "read-only" flags and other properties, and editing the actual property that is overridden of course leads to yet another set of problems. I'm happy to say that I'm confident we resolved all of these problems for the Search API (though, as mentioned before, I'd appreciate it if site owners who might be affected could thoroughly test this for their site, with the newest Beta release), but it's unfortunately very likely that there are other contrib modules that still have such problems, and might not even be aware of them.

The Core issue I created for this problem is this one: Issue #2744057: Inconsistencies when updating overridden config entities. However, I think it has actually centered more around the problem of reliably fixing the other problems listed above, by bringing the "immutable/editable" solution that's in place for normal config objects to config entities, too. (I might be mistaken, though – I frankly admit that the complexity of this issue gives me headaches.) Also important, sure, but it doesn't touch this last, actually hard problem, as far as I'm aware. (While the others are arguably also pretty hard to solve generically in Drupal Core, they are easy enough to work around in contrib modules using config entities, as long as you are aware of them.) I'm currently not even sure that a generic solution for this last problem is really feasible.

Therefore, it's all the more important for module developers, I think, to be aware that this problem exists and to carefully search their own code for places where something like this might happen. To quote swentel:

Damn, overrides are really not a safe thing at all.

Coda

I hope I could help at least some of you understand the complex problems with config entity overrides, and maybe in this way help discover and fix some bugs. In any case, many thanks to Alumei for first bringing this problem to my attention, patiently explaining it numerous times until I finally understood it, and even fearlessly tackling the Core issue to try and fix this for everyone else.

Image credit: showbiz kids