2.3 - "incompatible embedded data stream feature with encrypted receive" when no encryption is in-play

[sempervictus]

System information

Type	Version/Name
Distribution Name	Arch
Distribution Version	Current
Kernel Version	6.6.75
Architecture	x64
OpenZFS Version	2.3.0

Describe the problem you're observing

Attempting to migrate a dataset to another pool via send/recv (zfs send -Rve sourcepool/dataset@mig_to_targetpool|zfs recv -vFu targetpool/dataset) fails with

warning: cannot send 'sourcepool/dataset@mysnap-20221029': Input/output error
TIME        SENT   SNAPSHOT sourcepool/dataset@znap-20230813-185251
00:37:02   60.8K   sourcepool/dataset@znap-20230813-185251
cannot receive new filesystem stream: incompatible embedded data stream feature with encrypted receive.

Neither pool has ever had encryption used due to the instability we've seen over the years with ICP code. At no point has ZFS encryption been used for any block of either pool. Both pools are still @ zfs2.2 feature sets:

$ zpool get all | grep -E 'disabled|crypt'
sourcepool  feature@encryption             enabled                        local
sourcepool  feature@redaction_list_spill   disabled                       local
sourcepool  feature@raidz_expansion        disabled                       local
sourcepool  feature@fast_dedup             disabled                       local
sourcepool  feature@longname               disabled                       local
sourcepool  feature@large_microzap         disabled                       local
targetpool  feature@encryption             enabled                        local
targetpool  feature@redaction_list_spill   disabled                       local
targetpool  feature@raidz_expansion        disabled                       local
targetpool  feature@fast_dedup             disabled                       local
targetpool  feature@longname               disabled                       local
targetpool  feature@large_microzap         disabled                       local

dmesg is clean and SMART counters on both NVMEs are at zeroes. Source sits atop dm-crypt using OPAL HW encryption with a LUKS header while the target pool is over a conventional dm-crypt software target. Neither should matter.

Describe how to reproduce the problem

Include any warning/errors/backtraces from the system logs

[sempervictus]

Dropping the e from the send invocation still fails, this time with

cannot receive new filesystem stream: invalid backup stream

Source pool scrubs without error, destination pool too 😕

[whoschek]

I've encountered this error msg before and the workaround was to run 'zfs receive -A' on the destination dataset, as this was caused by a prior interrupted 'zfs receive -s', despite the error message seemingly pointing into another direction...

[sempervictus]

I've encountered this error msg before and the workaround was to run 'zfs receive -A' on the destination dataset, as this was caused by a prior interrupted 'zfs receive -s', despite the error message seemingly pointing into another direction...

Interesting, thank you. Will take a shot at that.

[whoschek]

Seems related to, or a dupe of, #12480