Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛 Source LinkedIn Ads: hands 429 response status code #8382

Merged
merged 5 commits into from
Dec 2, 2021
Merged

🐛 Source LinkedIn Ads: hands 429 response status code #8382

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
1772915
Add log message for 429 status_code
annalvova05 Dec 1, 2021
9daa660
fix
annalvova05 Dec 1, 2021
782c05a
Merge remote-tracking branch 'origin/master' into alvova/7206-fix-lin…
annalvova05 Dec 2, 2021
935c8e6
bump version
annalvova05 Dec 2, 2021
51c0013
bump version and format
annalvova05 Dec 2, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions airbyte-integrations/connectors/source-linkedin-ads/source_linkedin_ads/source.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,18 @@ def parse_response(self, response: requests.Response, **kwargs) -> Iterable[Mapp
"""
yield from transform_data(response.json().get("elements"))

def should_retry(self, response: requests.Response) -> bool:
if response.status_code == 429:
error_message = (
f"Stream {self.name}: LinkedIn API requests are rate limited. "
f"Rate limits specify the maximum number of API calls that can be made in a 24 hour period. "
f"These limits reset at midnight UTC every day. "
f"You can find more information here https://docs.airbyte.io/integrations/sources/linkedin-ads. "
f"Also quotas and usage are here: https://www.linkedin.com/developers/apps."
)
self.logger.error(error_message)
return super().should_retry(response)


class Accounts(LinkedinAdsStream):
"""
Expand Down
4 changes: 2 additions & 2 deletions docs/integrations/sources/linkedin-ads.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ This Source is capable of syncing the following data as streams:

### Performance considerations

There are official Rate Limits for LinkedIn Ads API Usage, [more information here](https://docs.microsoft.com/en-us/linkedin/shared/api-guide/concepts/rate-limits?context=linkedin/marketing/context). Rate limited requests will receive a 429 response. In rare cases, LinkedIn may also return a 429 response as part of infrastructure protection. API service will return to normal automatically. In such cases you will receive the next error message:
There are official Rate Limits for LinkedIn Ads API Usage, [more information here](https://docs.microsoft.com/en-us/linkedin/shared/api-guide/concepts/rate-limits?context=linkedin/marketing/context). Rate limited requests will receive a 429 response. Rate limits specify the maximum number of API calls that can be made in a 24 hour period. These limits reset at midnight UTC every day. In rare cases, LinkedIn may also return a 429 response as part of infrastructure protection. API service will return to normal automatically. In such cases you will receive the next error message:

```text
"Caught retryable error '<some_error> or null' after <some_number> tries. Waiting <some_number> seconds then retrying..."
Expand All @@ -59,7 +59,7 @@ This is expected when the connector hits the 429 - Rate Limit Exceeded HTTP Erro
"Max try rate limit exceded..."
```

After 5 unsuccessful attempts - the connector will stop the sync operation. In such cases check your Rate Limits [on this page](https://www.linkedin.com/developers/apps) &gt; Choose you app &gt; Analytics
After 5 unsuccessful attempts - the connector will stop the sync operation. In such cases check your Rate Limits [on this page](https://www.linkedin.com/developers/apps) &gt; Choose you app &gt; Analytics.

## Getting started
The API user account should be assigned the following permissions for the API endpoints:
Expand Down