Support SendGrid's new marketing API

[marius-stanescu]

I am trying to create a contact using this library and add it to a recipient list, but I am getting an "Access forbidden" exception. The SendGrid API key I use has Full Access. What could be the issue?

Here is my code:

var client = new Client(apiKey);
var contactId = await client.Contacts.CreateAsync("[email protected]");
await client.Lists.AddRecipientAsync(1, contactId);

[marius-stanescu]

After reading some more, I found out there are two APIs for Marketing Campaigns, a legacy one and a new one. I saw this library uses the legacy Marketing Campaigns API, which is not supported any more for new accounts, so, this is the reason for the exception. Are there any plans for implementing the new API?

[Jericho]

This is news to me!

I try to stay up to date with SendGrid's release by reading their release notes but I don't see any mention that they have introduced a new API for marketing campaigns and that they no longer support the original API. I also just checked their documentation for the 'Contacts' resource and I don't see any mention that's it's been deprecated and replaced with something new.

Can you help me understand what they changed?

[marius-stanescu]

If you look at the API documentation page, on the left sidebar there are two sections regarding marketing campaigns:

• LEGACY MARKETING CAMPAIGNS (eg: Contacts) which are the endpoints this library implement
• NEW MARKETING CAMPAIGNS (eg: Contacts)

There's also this section I stumbled upon, that talks about different pricing plans for the legacy API and the new API: https://sendgrid.com/docs/ui/sending-email/migrating-from-legacy-marketing-campaigns/#choosing-a-new-marketing-campaigns-plan.

I guess that if you have a new account (with the new pricing plan) as we do, then you don't have access to the legacy API anymore. You are right, the documentation is not very clear about this, but that's what I think is happening. Even on their 'try it out' pages, on the legacy API endpoints I get "Access forbidden", but the new API endpoints work.

[Jericho]

Now that you point it out, I do see the new section in the left sidebar. I will be away from my desk all day today but I’ll look at it tomorrow.

[Jericho]

I will need to spend some time thinking about how I will be able to provide access to the old and the new API in the StrongGrid library. My understanding is that new SendGrid accounts are forced to use the new API but existing accounts can continue to use the old API until they decide to migrate to the new API which means that I can't simply get rid of the old and replace with the new.

I'm open to suggestions.

[marius-stanescu]

I don't think you can just add a version parameter, as not only the endpoints URLs have changed, but also the objects being sent. So, I think you need to provide 'NewClient' and 'LegacyClient' classes. Or, at least, on the existing Client class provide NewContacts, NewLists, etc. properties.

[nevridge]

I also am working on a project for saving clients into SendGrid using the StrongGrid library. We would like to continue to access the old system while we figure out how to migrate to the new system. So thanks for not just getting rid of the old for the new APIs 😄 We very recently discussed just waiting for StrongGrid to add the new APIs before migrating.

[Jericho]

@nevridge I assume most SendGrid customers will also want to continue using the 'old' api for some time so I definitely don't want to remove this functionality (at least not in the short term). What I am struggling with though is how introduce the new functionality while minimizing the number of breaking changes for those who still use the old api.

I have been considering ideas very similar to what @marius-stanescu suggested. Currently my thinking is to rename the current Client class to LegacyClient and create a new Client class. This would make it easy to mark the legacy client as 'obsolete in a future version and eventually to completely remove this functionality if we decide that it's appropriate to do so (again, as I said earlier, I have no intention of removing this functionality any time soon). The down side of this idea, in my opinion, is that current developers who want to continue using the legacy api would have to modify their source code to replace var myClient = new Client(apikey); with var myClient = new LegacyClient(apikey); which is trivial for those of us who are aware of the changes we are introducing but not obvious to the casual developer who is not following this discussion.

What do you guys think? Is that too big of a breaking change?

[nevridge]

@Jericho I think that is a pretty big breaking change, however, it's really SendGrid that has made this change so that it is necessary for us to integrate.

I have a question regarding your LegacyClient suggestion; would only the legacy functionality be in the the LegacyClient and Client would also not contain any of this legacy functionality? My concern here is that the legacy functionality exists in both "types" of client. That would be more confusing.

Also, have you considered renaming the resource (i.e. Client.Contacts vs. Client.LegacyContacts)? I don't know if that is better or not, but it would avoid two separate clients. This is the other solution @marius-stanescu mentions as well. There may be other negatives to this approach I'm not seeing though.

Either way I see it you have a major breaking change coming up.

[Jericho]

would only the legacy functionality be in the the LegacyClient and Client would also not contain any of this legacy functionality?

The legacy client would contain the legacy functionality such as legacy contacts, legacy lists, legacy segmentation, legacy campaigns, etc. in addition to common functionality such as alerts, users, subusers, teammates, global settings, designs, etc.

The legacy functionality would be removed from the new client and replaced with new contacts, new lists, new custom fields, etc. in addition the common functionality.

have you considered renaming the resource (i.e. Client.Contacts vs. Client.LegacyContacts)? I don't know if that is better or not, but it would avoid two separate clients

Yes I did consider it and I agree with you that it would avoid having two clients but the drawback is that the client would have resources such as: LegacyContacts and Contacts, LegacyLists and Lists, LegacySenderIdentities and SenderIdentities, etc. The reason why I didn't like this option is because a developer who wants to continue using the legacy api would have to update every line of code that invokes a 'Contacts' method or a 'Lists' method or a 'Campaigns' method.

For example, if you have C# code to create a new contact and add this contact to a list similar to the following:

var contactId = await client.Contacts.CreateAsync(email, firstName, lastName, customFields, null, cancellationToken);
await client.Lists.AddRecipientAsync(list.Id, contactId, null, cancellationToken).ConfigureAwait(false);

you would have to modify these two lines like so:

var contactId = await client.LegacyContacts.CreateAsync(email, firstName, lastName, customFields, null, cancellationToken);
await client.LegacyLists.AddRecipientAsync(list.Id, contactId, null, cancellationToken).ConfigureAwait(false);

I think this breaking change is even more disruptive that the option we previously discussed.

To be honest with you, I don't really like either one of these options. I can already see the avalanche of support requests from developers who don't understand why their code used to work fine with a previous version of StrongGrid but it no longer works. I just thought that the first option was the least disruptive one.

On a separate but related topic: has anybody noticed that the new api is using the word "Send" to refer to a bulk email as opposed to a "Campaign" in the old api. I personally have always used the word campaign and I think that StrongGrid should continue using it but I was wondering if you guys think that we should change our terminology?

[nevridge]

I think your argument for the LegacyClient vs Legacy[Resource] is sound. I agree that it seems like it would be less disruptive. I don't envy the support requests you mention, but it really falls on SendGrid for making the big change.

In regards to the separate topic you raise, I have not yet worked with those resources. After looking over the SendGrid documentation it seems like they may have combined functionality in both the Campaigns API and the Cancel Scheduled Sends in the legacy section into the new Single Sends. However, the Cancel Scheduled Sends references a batchID and I don't see that anywhere in the Single Sends. I'm going to have to reserve my right to an opinion on this topic. I don't feel like I know enough about it currently to make one.

[filoe]

Any progress on this?

[Jericho]

Working on it. It's not as trivial as it may seem. I plan to have an early alpha version published soon on my MyGet feed and solicit feedback.

The most recent problem I faced had to do with a bizarre invalid content-type exception thrown when a request to the new API contains the usual Content-Type: application/json; charset=utf-8 header. This header is necessary to invoke the legacy API but their new API rejects the "charset" part. I have no idea why this requirement changed, and of course it's completely undocumented!

@marius-stanescu already reported this problem to SendGrid and even submitted a PR to resolve the problem. Unfortunately, his solution works with the new API but breaks all calls to the legacy API. I drew inspiration from his solution and improved it to ensure that StrongGrid can be configured to include or omit the charset on a per request basis.

[Jericho]

For alpha version is available on my MyGet feed.

What's included:

• A client called LegacyClient which allows access to SendGrid's legacy API
• A client called 'Client' which allows access to the new API. This client currently only contains the Contacts resource (lists, categories, custom fields, segments, etc will come later)
  • you can Upsert (i.e. update or insert) one or multiple contacts
  • you can delete one, multiple or all contacts
  • you can retrieve a specific contact
  • you can retrieve all contacts. Please note the SendGrid has deprecated paging and the API returns only the first 50 contacts.

I'm currently working on implementing the functionality to import/export contacts and also searching.

[Jericho]

Made some progress in the last few days: added the ability to import and export contacts and also manage custom fields (create, update, delete, retrieve, etc.) so I should be able to post another version to my MyGet feed soon.

However, I can't figure out how to pass the data for custom fields when creating a new contact. This has changed in the new API and the documentation is particularly vague: it simply says that you need to send an 'object' but it doesn't describe the shape of this object:

Does anybody have additional information about this?

[hostr]

I got you @Jericho. The "custom_fields" is an object with the keys being your custom field names. No spaces are allowed in custom fields so it maps directly to the variable name.

Here's an example:

            "custom_fields": {
                "age": 50,
                "business_name": "Test",
                "create_date": "2020-02-01T18:00:00Z"
            },

An array/dictionary would have been a nicer API design choice, but I guess this works too...

Btw, any idea about that new sendgrid contact search endpoint? https://sendgrid.api-docs.io/v3.0/contacts/search-contacts I have no idea what "valid SGQL" means or what it looks like. SendGrid Query Language? Something Graph Query Language?

[Jericho]

Thanks for the info regarding custom fields. I'll try your suggestion and report back.

Regarding SGQL: I have the exact same question! I have never heard of this query language. Probably the query language for whatever database they use (mysql, Oracle or something like that). I found a project on GitHub that seem relevant: https://github.com/profusion/sgqlc

UPDATE (2020-07-02): SGQL stands for Structured Graph Query Language. Several clients for .NET are available.

[filoe]

Where to retrieve the contact lists?
The new upsert method for contacts taks IEnumerable as list ids. The lists of the legacy client have ids of type long.

[Jericho]

Where to retrieve the contact lists?

I added the Contacts.GetAll method in the new client to retrieve multiple contacts but unfortunately, SendGrid is limiting the result to 50 contacts and they no longer allow paging through the resultset. If you need to retrieve more than 50 contacts, you might want to investigate Search. I don't know yet if they also limit the result to 50 records.

The new upsert method for contacts taks IEnumerable as list ids. The lists of the legacy client have ids of type long.

Yes indeed, the list id was a long in the legacy API but it is now a string. In fact I have noticed other identifier that were changed from long to string.

[Jericho]

@hostr Thanks again for the suggestion but unfortunately when I upsert a contact with the following:

"custom_fields": {
    "stronggrid_nickname":"Joe",
    "stronggrid_age":25
}

I get this error message:

invalid custom field ids supplied - stronggrid_age,stronggrid_nickname

If I use the field's identifier instead of the name, like this:

"custom_fields": {
    "e45_T":"Joe",
    "e46_N":25
}

it goes through without any problem. Thanks again for putting me on the right track.

[Jericho]

Updated package was just pushed to my MyGet feed. It includes custom fields management and importing/exporting contacts.

[Jericho]

Posted another update to my MyGet feed. This new version includes the new Lists resource.

[filoe]

Updated package was just pushed to my MyGet feed. It includes custom fields management and importing/exporting contacts.

Still having that problem runing 0.64.0-new-marketing-ap0069 using this code:

var contactId = await _sendGridClient.Contacts.UpsertAsync(contactModel.Email,
    listIds: new string[] {list.Id},
    customFields: new Parameter<IEnumerable<Field>>(new Field[]
    {
        new Field<string>("user_id", "user_id", contactModel.UserId),
        new Field<string>("stage", "stage", _hostingEnvrionment.EnvironmentName)
    }));

Tells me invalid custom field ids supplied.
Without using customFields it works fine.

[Jericho]

SendGrid changed the way they handle custom fields in this new API. Their legacy API allows you to conveniently specify the name of the fields such as user_id and stage but this no longer works in their new API. They expect you to provide the Id of the field which should look something like this: e45_T (this is the id of one of my custom fields, it will be different in your environment of course). So your code sample should look something like this:

var jobId = await _sendGridClient.Contacts.UpsertAsync(contactModel.Email,
    listIds: new string[] {list.Id},
    customFields: new Field[]
    {
        new Field<string>("e45_T", "user_id", "user_id", contactModel.UserId),
        new Field<string>("e46_T", "stage", "stage", _hostingEnvrionment.EnvironmentName)
    });

Also, please note that SendGrid no longer returns the unique id of the created record like they used to in their legacy API. Their new API processes your "upsert" request asynchronously (it may take a few seconds for your contact to be created or updated) therefore their API endpoint returns the job id.

[filoe]

Thanks for the detailed answer!
Where can I find the id of the custom field?

[Jericho]

You can get the id when you create the field:

var nicknameField = await client.CustomFields.CreateAsync("stronggrid_nickname", FieldType.Text, cancellationToken).ConfigureAwait(false);
await log.WriteLineAsync($"Field {nicknameField.Name} created. The Id of this new field is {nicknameField.Id}").ConfigureAwait(false);

If the field has already been created, you can retrieve all custom fields and filter the result like so:

var fields = await client.CustomFields.GetAllAsync(cancellationToken).ConfigureAwait(false);
await log.WriteLineAsync($"All custom fields retrieved. There are {fields.Length} fields").ConfigureAwait(false);

var userIdField = fields.Single(f => f.Name == "user_id");
await log.WriteLineAsync($"The Id of the user_id field is {userIdField.Id}").ConfigureAwait(false);

[Jericho]

I just completed adding the SingleSends resource to StrongGrid and just noticed something odd: in order to create a new single send you must specify the unique identifier of the sender identity but the new API doesn't allow you to retrieve the list of existing identities... (see: https://sendgrid.api-docs.io/v3.0/senders) Very odd?!?!?! It feels unfinished...

[Jericho]

Pushed another update to the StrongGrid package on my MyGet feed with the new SingleSends resource which replaces Campaigns.

[Jericho]

Regarding my comment from a few days ago: I was able to figure out the Get and GetAll methods on the SenderIdentities resource. It's just the documentation that is not up-to-date.

[Jericho]

Pushed another update. It feels pretty much feature complete at this point with two major exceptions: searching for contacts and segmenting contacts. The documentation simply says that you need to send a "query" but does not describe how a query should be structured.

At this point, I am looking forward to getting some feedback.

[Jericho]

SendGrid published some documentation about their query language yesterday which is now available on their web site.

I will review it and try to figure out if I can provide a strongly typed way of searching and segmenting contacts.

[Jericho]

Published a new version to MyGet with the ability to search for contacts like this example:

var firstNameCriteria = new SearchCriteriaEqual<ContactsFilterField>(ContactsFilterField.FirstName, "John");
var LastNameCriteria = new SearchCriteriaEqual<ContactsFilterField>(ContactsFilterField.LastName, "Doe");
var searchResult = await client.Contacts.SearchAsync(new[] { firstNameCriteria, LastNameCriteria }, cancellationToken).ConfigureAwait(false);
await log.WriteLineAsync($"Found {searchResult.Length} contacts named John Doe").ConfigureAwait(false);

Please note that currently only 10 fields (such as first name, last name, city, postal code, etc.) can be used to filter the result because these are the ten fields that SendGrid has documented. I suspect there are more fields but they have been omitted from documentation. I raised an issue and will update the list of Contacts filter fields when SenGrid updates their documentation.

[Jericho]

Pushed another version to MyGet. Allows you to specify strongly typed search criteria when creating a new segment (or updating an exiting one).

[Jericho]

One step closer to releasing this new feature: I merged the feature branch into develop and published what should be the final alpha package to MyGet.

@marius-stanescu @nevridge @filoe Any of you had an opportunity to try the alpha package? I would be grateful for feedback before I publish a final version to Nuget.

[marius-stanescu]

Thanks for your efforts. I am sorry, but I don't have the time now to try it out.

[Jericho]

🎉 This issue has been resolved in version 0.66.0 🎉

The release is available on:

• GitHub Release
• NuGet Package

Your GitReleaseManager bot 📦🚀