Update _key generation

[eXtremeX]

Summary

Added

• Made user, group and incident keys more detailed by including the
  name/email property.

Updated

• Updated SDK dependencies to 8.29.2.
• Changed ENV USERNAME property name to USERID.

• According to the documentation, it is possible that the Unique record identifier (sys_id), the parameter used to generate the key, can have duplicate values under certain circumstances.
• Another possible cause is that the sys_id may be unique only to the table it is on, meaning different tables can have entries with the same sys_id.
• As a fix for these, the key generation was updated in such a way that it takes the entity type, the sys_id and the name of the entity to generate the key.
• An issue was also found in the .env file, wherein the USERNAME parameter was not being properly detected. To fix this, the parameter was renamed to USERID instead.

Type of change

Please leave any irrelevant options unchecked.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to
  not work as expected)
• This change requires a documentation update

Checklist

General Development Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[zemberdotnet]

Do you have examples of where this fixed the duplicate key issue? I'm surprised the id wasn't unique.

[zemberdotnet]

I just want to confirm that this fixes the issue, because changing the _key value will cause a large amount of churn in users' graphs. If we can avoid that it's better, but does fix the issue I'm good to go with it.

[eXtremeX]

Hey @zemberdotnet, we aren't 100% sure if this will fix the issue but we thought it was worth a try - weren't aware changing _key has negative impact as a whole on the other hand.

The code surrounding user ingestion is very simple:

Client method part:

return this.iterateTableResources({
      table: ServiceNowTable.USER,
      callback,
});

Step function part:

await client.iterateUsers(async (user) => {
    const userEntity = await jobState.addEntity(createUserEntity(user));

    await jobState.addRelationship(
      createDirectRelationship({
        _class: RelationshipClass.HAS,
        from: accountEntity,
        to: userEntity,
      }),
    );
});

Some ideas that came to mind were either to increase the _key complexity or to maybe skip (truly) duplicate users but it's not like we're expecting them by default and that alone might hide some other bug that's causing this.

Do you have some ideas what you might try doing here?

[zemberdotnet]

Hey @eXtremeX 👋 Thank you for the detailed explanation.

weren't aware changing _key has negative impact as a whole on the other hand.

In this case, it's probably okay. It will cause all entities to be fully deleted and recreated, but this probably doesn't produce enough data for that to be a huge concern.

I'm good to try this approach if we can address the other comment I have below.

[zemberdotnet]

Hey, last problem before this can be approved. Do we have to change this field name? It will cause breaking changes and require migration for all integrations that use this field.