Continuation Token is invalid when we upgrade the sdk.

[SamGuoMsft]

We are continuously addressing and improving the SDK, if possible, make sure the problem persist in the latest SDK version.

Describe the bug
When we upgraded our sdk from 3.11.0-preview to 3.20.1-preview, we found the old continuation token which came from 3.11.0 -preview is invalid when we pass to the new sdk. The error we got is Failed to parse start of range: {"token":""1731886"","range":{"min":"","max":"FF"}}, and the continuation token from 3.11.0-preview is "{"V":0,"Rid":"71clAN7W-7o=","Continuation":[{"token":"\"1731886\"","range":{"min":"","max":"FF"}}],"Range":{"min":"","max":"FF"}}"

By looking at latest version sdk code when we convert v1 token to v2, we treat token, min, max at same level, but min and max are under range property. It seems like a bug
if (!cosmosObject.TryGetValue("min", out CosmosString min))
{
return TryCatch.FromException(
new MalformedChangeFeedContinuationTokenException(
message: $"Failed to parse start of range: {cosmosObject}."));
}

if (!cosmosObject.TryGetValue("max", out CosmosString max))
{
return TryCatch.FromException(
new MalformedChangeFeedContinuationTokenException(
message: $"Failed to parse end of range: {cosmosObject}."));
}

if (!cosmosObject.TryGetValue("token", out CosmosElement token))
{
return TryCatch.FromException(
new MalformedChangeFeedContinuationTokenException(
message: $"Failed to parse token: {cosmosObject}."));
}

To Reproduce
Use the token which listed on top to pass to latest sdk.

Expected behavior
The continuation token should be also valid in the new sdk version also.

Actual behavior
It is failed in the new sdk version.

Environment summary
SDK Version: upgraded sdk from 3.11.0-preview to 3.20.1-preview
OS Version: Windows

Additional context
Add any other context about the problem here (for example, complete stack traces or logs).

[j82w]

The API is now GA, but the breaking change was caused by the following change while the API was in preview: #1933

@ealsur have you seen this before?

[ealsur]

@SamGuoMsft Change Feed iterator was GA in 3.20 (no need to keep using previews). As @j82w mentioned, during the preview process, there were changes made to the continuation format (that's the reason these go through preview first and the implications from the consumption side is that things can change between preview and GA).

[ealsur]

Having said that, there was indeed a migration attempt added in that PR when the format was changed but it had no tests or anything verifying it actually worked 😞

You are correct in the sense that if that code was indeed working as intended, it would have migrated the old continuation. I am creating a PR to fix it.

But just as a note, this breaking changes are expected during feature preview, that is the whole point of the preview.

[SamGuoMsft]

Thanks @ealsur, I totally understand and agree with you about the preview thing, but the issue is that our team currently rely on a preview API which least below. Would you mind to let me know whether there is equivalent API in GA version so that we can get rid of preview version?
public abstract FeedIterator GetItemQueryIterator(FeedRange feedRange, QueryDefinition queryDefinition, string continuationToken = null, QueryRequestOptions requestOptions = null);

[ealsur]

That API has not GAed, the one that was (and where the gap between versions is) is GetChangeFeedIterator (which is the one that had the continuation format change).

GetChangeFeedIterator being GA is present in the GA package (and also on preview, preview contains all GA + extra APIs in preview). In your case, since you are using the Query with FeedRanges API, you need to remain in preview package, as GetItemQueryIterator with FeedRange has not GAed.