unexpected results when adding optional attributes to a projection expression

[anatolzak]

Describe the bug

If your entity has an optional attribute and you specify this attribute in the attributes selection, ElectroDB behaves as follows: If the item in DynamoDB does not include the optional attribute:

1. It returns null for the item in a get operation ({ data: null }).
2. It excludes the item from the results of a query operation ({ data: [], cursor: null }).
3. It returns an array containing null for a batch get operation, although the TypeScript type does not indicate that the array can contain nulls ({ data: [null], unprocessed: [] }).

Here is an example illustrating the differences in the query operation when specifying optional fields in the attributes (the name attribute is defined as optional at the Entity level):

const queryWithAttributes = await db.entities.User.query
  .byId({ id: '1' })
  .go({ attributes: ['name'] });

const queryWithoutAttributes = await db.entities.User.query.byId({ id: '1' }).go();

console.log({ queryWithAttributes, queryWithoutAttributes });

/*
{
	"queryWithAttributes": {
		"data": [],
		"cursor": null
	},
	"queryWithoutAttributes": {
		"data": [{
			"id": "1"
		}],
		"cursor": null
	}
}
*/

Here is an example illustrating the differences in the get operation when specifying optional fields in the attributes:

const getWithAttributes = await db.entities.User.get({ id: '1' }).go({ attributes: ['name'] });

const getWithoutAttributes = await db.entities.User.get({ id: '1' }).go();

console.log({ getWithAttributes, getWithoutAttributes });

/*
{
	"getWithAttributes": {
		"data": null
	},
	"getWithoutAttributes": {
		"data": {
			"id": "1"
		}
	}
}
*/

Here is an example illustrating the differences in the batchGet operation when specifying optional fields in the attributes:

const batchGetWithAttributes = await db.entities.User.get([{ id: '1' }]).go({ attributes: ['name'] });

const batchGetWithoutAttributes = await db.entities.User.get([{ id: '1' }]).go();

console.log({ batchGetWithAttributes, batchGetWithoutAttributes });

/*
{
	"batchGetWithAttributes": {
		"data": [
			null
		],
		"unprocessed": []
	},
	"batchGetWithoutAttributes": {
		"data": [{
			"id": "1"
		}],
		"unprocessed": []
	}
}
*/

ElectroDB Version
3.0.1

ElectroDB Playground Link
Not relevant, as the issue occurs after ElectroDB receives the response from DynamoDB, and ElectroDB doesn't return all the expected items.

I created a minimal reproducible repository: https://github.com/anatolzak/electrodb-optional-attributes-bug. Just run npm install and ./run.sh.

Entity/Service Definitions

{
    model: {
      entity: 'user',
      version: '1',
      service: 'app',
    },
    attributes: {
      id: { type: 'string', required: true, readOnly: true },
      name: { type: 'string' },
    },
    indexes: {
      byId: {
        pk: { field: 'pk', composite: ['id'] },
        sk: { field: 'sk', composite: [] },
      },
    },
  }

Expected behavior
Items should not be excluded from the results of read operations if they don't contain optional attributes.

[tywalch]

Thanks @anatolzak! I will take a look at this to better understand what's going on under the hood, thanks for sending it over, hope to reply back soon!

[tywalch]

@anatolzak Firstly, thank you very much for the example repo; it helped me quickly jump into this ticket! Is it safe to assume the behavior you're expecting (in the specific case of your example) is to have an empty object returned?

[anatolzak]

@tywalch glad the repo was helpful! And yes, I would expect an empty object