Error: Failed to deserialize Geometry object because 'type' property is either absent or has invalid value.

[septst]

I am trying to store geospatial data in my cosmos db container named Trips and retrieve the items from my .net core web api (.net 5). I am using Microsoft.Azure.Cosmos (3.19.0) and have not configured anything explicitly for serializing. The save works but when I try to retrieve I am getting this error:

Newtonsoft.Json.JsonSerializationException: Failed to deserialize Geometry object because 'type' property is either absent or has invalid value.

Cosmos Client Instance:

 var options = new CosmosClientOptions()
   {
      AllowBulkExecution = true,
      SerializerOptions = new CosmosSerializationOptions()
      {
         PropertyNamingPolicy = CosmosPropertyNamingPolicy.CamelCase
      }
   };
   var cosmosClient = new CosmosClient(account, key, options);

Model:

public class Trip
    {
        [JsonPropertyName("id"), Required]
        public string Id { get; set; }

        [JsonPropertyName("vehicleId"), Required]
        public string VehicleId { get; set; }

        [JsonPropertyName("startDateTime"), Required]
        public DateTime StartDateTime { get; set; }

        [JsonPropertyName("endDateTime"), Required]
        public DateTime EndDateTime { get; set; }

        [JsonPropertyName("wayPoints"), Required]
        public List<WayPoint> WayPoints { get; set; }
    }

    public class WayPoint
    {
        [JsonPropertyName("timeStamp"), Required]
        public DateTime TimeStamp { get; set; }

        [JsonPropertyName("point"), Required]
        public Point Point { get; set; }
    }

Service:

public async Task<IEnumerable<Trip>> GetMultipleAsync(string vehicleId = "")
{
   var queryDefinition = new QueryDefinition($"Select * from c where c.vehicleId = \"{vehicleId}\"");
   var queryIterator = _container.GetItemQueryIterator<Trip>(queryDefinition);
    
   var trips = new List<Trip>();
   while(queryIterator.HasMoreResults)
   {
      var response = await queryIterator.ReadNextAsync();
      trips.AddRange(response.ToList());
   }
   return trips;
}

Json sample:

{
    "id": "a9153ca0-e171-4fe8-bcfe-733ac75f6b85",
    "vehicleId": "599abc63-eafb-4015-ac65-fc6aed48d9aa",
    "startDateTime": "2021-06-17T00:00:00Z",
    "endDateTime": "2021-06-17T23:55:00Z",
    "wayPoints": [
        {
            "timeStamp": "2021-06-17T00:00:00Z",
            "point": {
                "Position": {
                    "Coordinates": [
                        51.23156579100001,
                        -0.603818000999999
                    ],
                    "Longitude": 77.23156579100001,
                    "Latitude": 12.016038180009999,
                    "Altitude": null
                },
                "Crs": {
                    "Type": 0
                },
                "Type": 0,
                "BoundingBox": null,
                "AdditionalProperties": {}
            }
        },
        {
            "timeStamp": "2021-06-17T00:05:00Z",
            "point": {
                "Position": {
                    "Coordinates": [
                        51.23159449100001,
                        -0.01703846700999
                    ],
                    "Longitude": 77.23159449100001,
                    "Latitude": 12.603846700999998,
                    "Altitude": null
                },
                "Crs": {
                    "Type": 0
                },
                "Type": 0,
                "BoundingBox": null,
                "AdditionalProperties": {}
            }
        },
        ////////////
        {
            "timeStamp": "2021-06-17T23:55:00Z",
            "point": {
                "Position": {
                    "Coordinates": [
                        51.23980269100042,
                        -0.01961205490099
                    ],
                    "Longitude": 77.23980269100042,
                    "Latitude": 12.612054900999901,
                    "Altitude": null
                },
                "Crs": {
                    "Type": 0
                },
                "Type": 0,
                "BoundingBox": null,
                "AdditionalProperties": {}
            }
        }
    ],
}

Any help would be appreciated. Thanks!

Environment summary
SDK Version: 3.19.0
OS Version: Windows 10

I have also posted this issue in stack overflow but no luck so far.

[septst]

I guess I missed to add a crucial part. In my controller, I used the below to write to a json file to bulk upload to cosmos.
using FileStream createStream = System.IO.File.Create(@"C:\repos\TripGenerator\trips.json"); await JsonSerializer.SerializeAsync( createStream, trips);

This didn't serialize the properties of Geometry type "Point", as pointed out by Matias Quaranta in my stackoverflow post. So, I modified the above code as below:

 using FileStream createStream = System.IO.File.Create(@"C:\repos\TripGenerator\trips.json");
            await JsonSerializer.SerializeAsync(
                createStream,
                trips,
                new JsonSerializerOptions
                {
                    PropertyNamingPolicy = JsonNamingPolicy.CamelCase
                });

Doing so, the difference I see between the data I save directly with CreateItemAsync and that I generated using the above json serializer is the way my geometry data is serialized. Cosoms DB rejects my json file.

Direct Save:

{
    "id": "a9f8aee8-d8d5-4fbf-bccc-fb5aac031ab1",
    "vehicleId": "39afa3f8-c9e8-4d86-916f-8972552526f0",
    "startDateTime": "2021-06-19T00:00:00Z",
    "endDateTime": "2021-06-19T15:50:00Z",
    "wayPoints": [
        {
            "timeStamp": "2021-06-19T00:00:00Z",
            "point": {
                "type": "Point",
                "coordinates": [
                    77.33932400799999,
                    12.540575257
                ]
            }
        },
        {
            "timeStamp": "2021-06-19T00:05:00Z",
            "point": {
                "type": "Point",
                "coordinates": [
                    77.33934300799999,
                    12.540594257
                ]
            }
        }
}

Upload Item from Portal:

{
    "id": "b1e0d6bf-d1de-4e5f-81c9-81b333542ca0",
    "vehicleId": "d3749180-2de8-4d85-8c77-10d21204f9cd",
    "startDateTime": "2021-06-19T16:00:00Z",
    "endDateTime": "2021-06-19T23:55:00Z",
    "wayPoints": [
        {
            "timeStamp": "2021-06-19T16:00:00Z",
            "point": {
                "position": {
                    "coordinates": [
                        77.281050633,
                        12.535945981000001
                    ],
                    "longitude": 77.281050633,
                    "latitude": 12.535945981000001,
                    "altitude": null
                },
                "crs": {
                    "type": 0
                },
                "type": 0,
                "boundingBox": null,
                "additionalProperties": {}
            }
        },
        {
            "timeStamp": "2021-06-19T16:05:00Z",
            "point": {
                "position": {
                    "coordinates": [
                        77.28106013300001,
                        12.535955481000002
                    ],
                    "longitude": 77.28106013300001,
                    "latitude": 12.535955481000002,
                    "altitude": null
                },
                "crs": {
                    "type": 0
                },
                "type": 0,
                "boundingBox": null,
                "additionalProperties": {}
            }
        }
}

I understand that Cosmos serializer has some special converter for geometry types. Can you please advise how I can write to my local json file in the way the cosmos db accepts. My main goal is to generate a good amount of data to do some testing. Thanks in advance.

[ealsur]

Can you share what you mean by "Cosoms DB rejects my json file."? What error are you getting?

I answered the linked Stackoverflow post, that means your original issue is resolved?

[septst]

Can you share what you mean by "Cosoms DB rejects my json file."? What error are you getting?

I answered the linked Stackoverflow post, that means your original issue is resolved?

Yes, my original issue is resolved. Apologies I am unable to replicate the file rejection. Everything works when I directly save the data in cosmos db. But, I am looking for a way to write the data to a json before uploading it in cosmos db. Thanks much!

[ealsur]

I am looking for a way to write the data to a json before uploading it in cosmos db.

I'm very sorry but I'm not able to understand clearly.

Are you asking how to save data to a Json file or are you asking how to generate a Stream from your Json file? The problem I see is your Json file can have multiple documents I guess? You cannot load a Json file that might contain multiple documents and upload them in one go.

CreateItem can be called for 1 document each time it is invoked, so if your file has an array of documents, you need to first read the file, generate 1 item per document, and then use that list of items or Streams to call CreateItem.

If you are using Bulk mode, just remember to execute the CreateItems concurrently: https://devblogs.microsoft.com/cosmosdb/introducing-bulk-support-in-the-net-sdk/

[septst]

Apologies for not making it clear.

Controller Action:

var trips = GenerateTrips(vehicles, startDateTime);

using FileStream createStream = System.IO.File.Create(@"C:\repos\TripGenerator\trips.json");
await JsonSerializer.SerializeAsync(
    createStream,
    trips,
    new JsonSerializerOptions
    {
        PropertyNamingPolicy = JsonNamingPolicy.CamelCase
    });

return Ok(new { Message = "The trip data is successfully written to Trips.Json" });
////cosmos db call
//await _tripService.AddMultipleAsync(trips);
//return Ok(new { Message = "The trip data is successfully added to Trips collection." });

In the above piece of code, I generate a list of objects (test data) of type Trip in a large number say 10,000 to begin with but can go up to few millions. I want to save these in cosmos db faster. This is mostly an one-time activity. I found using the portal to upload a json file is faster than the below code that uses concurrent tasks as you suggested.

public async Task AddMultipleAsync(List<Trip> trips)
{
    var concurrentTasks = new List<Task>();

    foreach (var trip in trips)
    {
        concurrentTasks.Add(_container.CreateItemAsync(trip, new PartitionKey(trip.VehicleId)));
    }

    await Task.WhenAll(concurrentTasks);
}

If I use the Json serializer to write the data to a file and upload it, I am unable to fetch it due to the type discrepancy (as in the samples given above) in my geometry property "point" (which was my original problem). Hence, I need to write data into my json file in the way cosmos db serializes and stores the "point" data.

Many thanks for your help again! Much appreciated!

[ealsur]

I see, so there are 2 points.

One is, how do you generate a Json file that you can upload on the Azure Portal with the right casing. I think the problem is you are mixing serialization technologies.

If you are generating a Json file and uploading, you are in full control of what that content is and the SDK is not related at all. The key point there though is that you seem to be using System.Text.Json to generate the File content. The JsonSerializer.SerializeAsync seems to be coming from System.Text.Json, right? The V3 SDK uses Newtonsoft.Json as serialization technology and the Geometry classes all have decorators for Newtonsoft.Json. So if you are generating the raw Json file for Portal upload, use Newtonsoft.Json instead of System.Text.Json.

The second aspect is Bulk. If you are using Bulk (this also means turning on the Bulk mode) then the performance will always be much faster than uploading it through the Portal. You can see some benchmark results at #2440. When the client is on Direct mode, it is directly reaching the backend replica. Uploading a file through the Portal and having the Portal save the file contents as documents as requests being sent from the browser cannot be faster than accessing the Replica through a connected TCP channel. Make sure your Controller is async and that the client is on Bulk mode. You can monitor the RU consumption also and see if you are getting any 429/Throttles (Bulk mode might be even pushing data faster than your provisioned throughput can process it and causing throttling).

[septst]

Thanks for taking time to give this detailed explanation and responding to my queries @ealsur. This helps.