Navigation breaks with Strapi export/import

[unsalted]

We do a semi regular export from our production environment, and then import in our staging environment with the new Strapi import/export feature.

The navigation structure copies, but the content-type and menu relations are missing.
The impact is we can't use our staging environment for testing because navigation is broken.

strapi-plugin-navigation: 2.4
strapi: 4.8

[unsalted]

Per a recommendation from Strapi:
"the relational linking tables that holds relational information possibly was not imported but it would depend on how the plugin was developed and how it injects the relations and if they are polymorphic or not." I haven't found time to dive deep in how the relations are managed in this plugin, but we had this issue on imports to new databases, and copies alike.

[cyp3rius]

Hello @unsalted ,

Strapi is right here, we've got custom relations made to handle entities properly in multiple navigations. Let me dig into the import - export functionality of Strapi and see how we can fix this issue on our end.

[cyp3rius]

@unsalted I've did couple of tests and seems the issue been found.

I’ve checked in deep and exported data are correct, as well the imported relations & links are proper except one thing. After import the navigations_items_related.related_id is broken because the related content type id does not match. For example in export file I've got api::page.page with id:1 but after import the same entity got id:2 and in navigations_items_related the related_id stays 1 and is not updated to 2.

What I did is the import on a same database without cleaning up any indexing so counting is continued on every table. That many imports I did it always increase the id of my api::page.page entity but relation_id stays unchanged.

With a fresh database import works perfectly. I've notified Strapi about that case, because seems it's a non handled corner case for import / export. Please try to import on the clear database schema and you'll see it works.

[derrickmehaffy]

@cyp3rius do you think it's due to the sequence not being restarted on the target table before we reimport the data?

[cyp3rius]

Might be. I've reproduced the same thing couple of times with exact same result.

[derrickmehaffy]

Odd that the actual data tables work but the relational joining tables don't. When testing with normal content-types and relations I'm not able to reproduce the issue 🤔

[derrickmehaffy]

DEITS is designed to map the entries for the new IDs automatically and handle the linking

[cyp3rius]

Indeed, I've tested with relations done from the UI interface and typical content types - works. Possibly, but I'm not sure - the plugin content types and relations are injected to DB earlier than the content type from Content Manager and are not re-iterated once ID change.

Clear & new DB - No issues, as re-iteration is not necessary.

[derrickmehaffy]

I haven't looked at the code for this plugin in a -really- long time, let me see how you are are injecting them.

[derrickmehaffy]

I'm guessing that's designed for custom polymorphic relationships?

[cyp3rius]

I think the case is with the related content type as we moved from the poly relation to a mid state content type with id and type to allow linking same entities in different navigations. Poly relations were overwritting each others when there was a multiple usage.

@derrickmehaffy any idea how to provide custom import logic for plugin and update those fields properly, when content type id changes?

[derrickmehaffy]

Ah yeah I see:

strapi-plugin-navigation/server/content-types/navigations-items-related/schema.ts

Lines 25 to 46 in bdda349

	attributes: {
	related_id: {
	type: "string",
	required: true
	},
	related_type: {
	type: "string",
	required: true
	},
	field: {
	type: "string",
	required: true
	},
	order: {
	type: "integer",
	required: true
	},
	master: {
	type: "string",
	required: true
	}
	}

These aren't setup as real relations so the relation handler isn't treating them as such, they are just integers strings.

[derrickmehaffy]

any idea how to provide custom import logic for plugin and update those fields properly, when content type id changes?

I'm not sure, I'll need to talk to our DX Squad about it.

[cyp3rius]

Some kind of:

• change log / mapping with type, newID, oldID
• event emit
• lifecycle method afterImport with changes as input

Or similar, whats going to allow such entities been updated.

[derrickmehaffy]

The way DEITS handles links is defined in here: https://github.com/strapi/strapi/blob/main/packages/core/data-transfer/src/strapi/queries/link.ts

[derrickmehaffy]

The issue I'm guessing is we are directly looking for relation types and since those fields are just strings nothing is done to them

https://github.com/strapi/strapi/blob/155bb69ef68a3bc7c23d4ad2a0656a4a8c7e5ecd/packages/core/data-transfer/src/strapi/queries/link.ts#L285-L295

[unsalted]

Thanks for jumping on this issue so quickly. I appreciate it.

[cyp3rius]

@derrickmehaffy I've checked the transfer package and indeed as for now there is no possibility for us to "run" custom logic after import and get those changes applied. Do you think the DX team might consider some kind of routines for plugin creators like afterImport, beforeImport ?

[derrickmehaffy]

@derrickmehaffy I've checked the transfer package and indeed as for now there is no possibility for us to "run" custom logic after import and get those changes applied. Do you think the DX team might consider some kind of routines for plugin creators like afterImport, beforeImport ?

Probably not in the short term, we could in the future possibly but it'll be a while before they will have time to implement something.

[unsalted]

Any movement on this? We may need to drop the plugin and implement a custom solution because it has become a blocker on several initiatives. @derrickmehaffy it sounds like there isn't an existing hook for afterImport, has there been any movement on implementing one?

[cyp3rius]

Hello @unsalted,
Unfortunately without such hook or similar solution we won't be able to make the fix on the end of a plugin. Getting back to polymorphic relations is not possible because of their limitations in terms of linking same entities in multiple navigations and problems user faced in the past.

[derrickmehaffy]

Any movement on this? We may need to drop the plugin and implement a custom solution because it has become a blocker on several initiatives. @derrickmehaffy it sounds like there isn't an existing hook for afterImport, has there been any movement on implementing one?

For the moment no we won't have the time for the next few months to add something like this to DTS.

[derrickmehaffy]

Hello @unsalted,
Unfortunately without such hook or similar solution we won't be able to make the fix on the end of a plugin. Getting back to polymorphic relations is not possible because of their limitations in terms of linking same entities in multiple navigations and problems user faced in the past.

I don't believe polymorphic relations should be used here and probably should be using oneWay relationships. I don't have a full understanding of how that is stored in the plugin though.

[cyp3rius]

I don't believe polymorphic relations should be used here and probably should be using oneWay relationships. I don't have a full understanding of how that is stored in the plugin though.

Navigation Items got relations to unspecified content types so if relations, then only poly might be used as I understand that correctly. OneWay relations must have specified target which represent the Collection / Single Type. As we've been using pyli in a past but it wasn't perfect and need a lot of control.

[EsGeh]

Correct me if I am wrong:

The problem lies in referencing related content in menu items via ids, which are resolved manually, instead of by explicit relations which would automatically be handled by strapi, as pointed out above by @derrickmehaffy.
This seems to be necessary, because the referenced type is not known beforehand and cannot be hardcoded, because it is instead configurable by the user.
On the other hand depending on the ids never to change seems a bad design decision to me and results in exactly the problem under discussion.

If I remember correctly, such a situation in other ORM-Systems is modeled via something called "polymorphic relations".
The decisive question is, whether such a feature is available in strapi...?
It doesn't seem so!
However, there should definitely be such a feature or an equivalent concept. How are we supposed to handle such scenarios in strapi?

[EsGeh]

As a practical intermediate solution one might consider a feature that enables referencing related user defined content via some other unique field other than it's ID...

[derrickmehaffy]

Yes we do have polymorphic relationships but generally advise against them due to their complexity. For example the media field is actually a polymorphic relationship.

Polymorphic relationships I don't believe are documented but it is possible.

[EsGeh]

I simply made a new navigation plugin that supports strapi import/export, since I don't see much progress in solving major issues such as emport/export.

sghp-nav

It's quite stable, but the feature set might differ. Happy to take feature requests and pull requests. Any feedback is most welcome.

Pros:

• i18n, Nationalization: seperate navigation for every locale
• Configurable via config file
• Strapi conformant REST API to fetch hierarchical menu data
• Typescript types for REST responses included
• Navigation items may be associated with a custom content type (ie. that represents a page or section on your website)
• Import / Export of navigation structure via strapis native command line tools

Another alternative is:

Strapi Menus

[certainlyakey]

@unsalted I've did couple of tests and seems the issue been found.

I’ve checked in deep and exported data are correct, as well the imported relations & links are proper except one thing. After import the navigations_items_related.related_id is broken because the related content type id does not match. For example in export file I've got api::page.page with id:1 but after import the same entity got id:2 and in navigations_items_related the related_id stays 1 and is not updated to 2.

What I did is the import on a same database without cleaning up any indexing so counting is continued on every table. That many imports I did it always increase the id of my api::page.page entity but relation_id stays unchanged.

With a fresh database import works perfectly. I've notified Strapi about that case, because seems it's a non handled corner case for import / export. Please try to import on the clear database schema and you'll see it works.

I understand that one solution to this problem is to "clear database schema" and import again. How exactly do i clear a database schema? Is there a CLI command or should I just delete the DB file (I use SQLite)?

[derrickmehaffy]

This will be somewhat solved in Strapi v5 as we are moving to string-based CUIDs instead (basically short form UUIDs) as primary keys instead of integers.