Source: Amazon Seller Partner - Updated Report API version + New reports #16629

krisjan-oldekamp · 2022-09-13T08:34:07Z

What

Updated Amazon Seller Reports API verison

Updated version to version 2020-09-04 to 2021-06-30.

Changes: No decryption necesary anymore + the data return field changed (response is not in the "payload" property anymore, so removed this).
See https://developer-docs.amazon.com/sp-api/docs/reports-api-v2021-06-30-reference

Updated connector test

The connector assumed that the "Orders" report is available for every account. So it used the "Orders" endpoint to check if authenticated successfully. However this is not the case for Vendor-only accounts within Amazon Seller API. So did an alternative check.
When you enter invalid credentials, this is the error message:

Exception('Error while refreshing access token: 401 Client Error: Unauthorized for url: https://api.amazon.com/auth/o2/token')"}}

However, when you do have valid credentials, but do not have access to the "Orders" endpoint (since thats not available for Vendor only accounts):

HTTPError('403 Client Error: Forbidden for url: https://sellingpartnerapi-eu.amazon.com/orders/v0/orders?LastUpdatedAfter=xxxx

So I've added an additional check, that checks if "403 Client Error" is in the the error response, and if so, let the connection test pass.

Added new reports

GET_VENDOR_SALES_REPORT > Incremental, it has to make a request for every day, otherwise the data is will be aggregated for the requested datarange. Also make sure you have configured the required params in reportOptions and set Period In Days setting to 1 > https://developer-docs.amazon.com/sp-api/docs/report-type-values#vendor-retail-analytics-reports
GET_VENDOR_INVENTORY_REPORT > Full-refresh. Also added a check / data delay param, because data availability SLA is in place (data available after 72 hours) > https://developer-docs.amazon.com/sp-api/docs/report-type-values#vendor-retail-analytics-reports
GET_AFN_INVENTORY_DATA > FBA (Fullfillment by Amazon) Inventory data. Full refresh > https://developer-docs.amazon.com/sp-api/docs/report-type-values#fulfillment-by-amazon-fba-reports
GET_AFN_INVENTORY_DATA_BY_COUNTRY > FBA (Fullfillment by Amazon) Inventory data. Full refresh. Only available in Europe > https://developer-docs.amazon.com/sp-api/docs/report-type-values#fulfillment-by-amazon-fba-reports
GET_SALES_AND_TRAFFIC_REPORT > Incremental. Its has to make a request for every day, otherwise the data is will be aggregated for the requested datarange. The reponse returns two iterables, by date level and by ASIN (= amazon product-id) level. I've selected the ASIN level, so there is no data param in the response, so I've added it manually. >https://developer-docs.amazon.com/sp-api/docs/report-type-values#seller-retail-analytics-reports

🚨 User Impact 🚨

No. API version update does not alter fieldnames etc.

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

CLAassistant · 2022-09-13T08:34:13Z

All committers have signed the CLA.

sajarin · 2022-09-13T15:30:52Z

Hi @krisjan-oldekamp thanks for the PR. This PR is part of Airbyte's Community Maintainer program and someone from our community will be assigned to review your PR and help it get merged. Thanks for being patient and thanks for improving this connector.

sajarin · 2022-09-14T17:12:30Z

assigned to @tuanchris

tuanchris · 2022-09-15T21:39:13Z

Hi @krisjan-oldekamp, can you post the logs of passing integration tests and unit test? I tried running your code locally with Airbyte's credentials and was faced with this error:
403 Client Error: Forbidden for url: https://sellingpartnerapi-na.amazon.com/reports/2021-06-30/reports

@sajarin can you help check with the team to see if they should update the permissions for the new endpionts?

krisjan-oldekamp · 2022-09-19T12:34:26Z

@tuanchris what report are you trying to fetch? The multi-country inventory report is only available in Europe (and I see you're trying to request a report in North America -> https://sellingpartnerapi-**na**.amazon.com/reports/2021-06-30/reports).

amazon-seller-partner-unit-tests.txt

tuanchris · 2022-09-19T14:00:04Z

@krisjan-oldekamp Oh that can be the case. I'm testing the new streams that you implemented. Can you share the integration tests passing also?

krisjan-oldekamp · 2022-09-20T08:45:01Z

@tuanchris see logs attached (deleted some of the result data).

Good to know, you need to configure these report options, since some of the parameters are mandatory:

{
"GET_VENDOR_INVENTORY_REPORT": {"reportPeriod": "WEEK", "distributorView": "MANUFACTURING", "sellingProgram": "RETAIL"},
"GET_VENDOR_SALES_REPORT": {"reportPeriod": "DAY", "distributorView": "MANUFACTURING", "sellingProgram": "RETAIL"},
"GET_SALES_AND_TRAFFIC_REPORT": {"dateGranularity": "DAY", "asinGranularity": "CHILD"}
}
amazon-seller-partner-integration-tests.txt

I also commited some small changes, as I encountered these issues today.

tuanchris

Hi @krisjan-oldekamp, I left a minor comment below, please have a look. Can you also:

run: ./gradlew :airbyte-integrations:connectors:source-amazon-seller-partner:airbytePythonFormat and fix some errors?
clean up the codes that you commented out
I'm not too familiar with the Amazon seller partner API, can you confirm if bumping API version to 2021-06-30 would require users to reconfigure permissions?

tuanchris · 2022-09-21T00:53:54Z

...-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/source.py

@@ -96,7 +101,7 @@ def get_sts_credentials(config: AmazonSellerPartnerConfig) -> dict:

    def check_connection(self, logger: AirbyteLogger, config: Mapping[str, Any]) -> Tuple[bool, Any]:
        """
-        Check connection to Amazon SP API by requesting the list of reports as this endpoint should be available for any config.
+        Check connection to Amazon SP API by requesting the list of reports as this endpoint should be available for any config. > NOT THE CASE, for VENDOR-ONLY accounts, the Orders endpoint is not available.


This is good to know. Please update the doc to describe the new behavior.

krisjan-oldekamp · 2022-09-21T07:52:39Z

@tuanchris

I've removed commented out code
Updated the connection test doc
I've spend a couple of hours to get the Gradle tests to work(....), but I keep getting errors unrelated to the Amazon Seller Partner source, so could you or somebody else run the format test and pass me the things to fix?

tuanchris · 2022-09-21T15:51:13Z

@krisjan-oldekamp I was able to run it and created a pull request on your branch here https://github.com/stacktonic-com/airbyte/pull/1

minor fix for python format

krisjan-oldekamp · 2022-09-22T06:40:09Z

@tuanchris Thanks a lot, merged the pull request. And I can confirm users don't have to reconfigure permissions when bumping the API version (I was able to fetch all the reports from the previous and the current version with the same credentials that were created months ago / withouth reconfiguring).

tuanchris · 2022-09-22T14:31:25Z

Thank you @krisjan-oldekamp
@sajarin good to merge from my end.

sajarin · 2022-09-23T15:47:19Z

/test connector=connectors/source-amazon-seller-partner

🕑 connectors/source-amazon-seller-partner https://github.com/airbytehq/airbyte/actions/runs/3113967202
✅ connectors/source-amazon-seller-partner https://github.com/airbytehq/airbyte/actions/runs/3113967202
Python tests coverage:

Name                                        Stmts   Miss  Cover
---------------------------------------------------------------
source_amazon_seller_partner/spec.py           23      0   100%
source_amazon_seller_partner/__init__.py        2      0   100%
source_amazon_seller_partner/constants.py      37      1    97%
source_amazon_seller_partner/source.py         51     12    76%
source_amazon_seller_partner/streams.py       524    245    53%
source_amazon_seller_partner/auth.py           61     36    41%
---------------------------------------------------------------
TOTAL                                         698    294    58%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       121     25    79%   21-23, 29-31, 36-43, 48-61, 208-216
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       110     48    56%   23-26, 32, 36, 39-64, 67-69, 72-74, 77-79, 82-84, 87-89, 92-110, 144-146
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1322    463    65%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestBasicRead.test_read because not found in the config
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestFullRefresh.test_sequential_reads because not found in the config
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestIncremental.test_two_sequential_reads because not found in the config
======================== 22 passed, 3 skipped in 18.47s ========================

sajarin · 2022-09-23T18:20:34Z

/publish connector=connectors/source-amazon-seller-partner

🕑 Publishing the following connectors:
connectors/source-amazon-seller-partner
https://github.com/airbytehq/airbyte/actions/runs/3114788952

Connector	Did it publish?	Were definitions generated?
connectors/source-amazon-seller-partner	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

sajarin · 2022-09-23T18:40:56Z

@krisjan-oldekamp, not sure I have permission to push to your branch.

Can you bump the version in the Dockerfile and add a line to the changelog in docs/integrations/sources/amazon-seller-partner.md?

krisjan-oldekamp · 2022-09-24T08:39:50Z

@sajarin Done!

sajarin · 2022-09-24T08:45:00Z

/publish connector=connectors/source-amazon-seller-partner

🕑 Publishing the following connectors:
connectors/source-amazon-seller-partner
https://github.com/airbytehq/airbyte/actions/runs/3117696728

Connector	Did it publish?	Were definitions generated?
connectors/source-amazon-seller-partner	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

krisjan-oldekamp · 2022-09-24T09:23:31Z

@sajarin hmm, access seems to be related to this issue: community/community#5634 I;ve added you and octavia-squidington-iii as a maintainer manually, but not sure if this would work (since status in "pending invite")

krisjan-oldekamp · 2022-09-26T08:01:08Z

@sajarin Is it possible for octavia-squidington-iii to accept the invitation or do you see other ways to fix the access issue?

sajarin · 2022-09-26T17:19:03Z

@krisjan-oldekamp, I think it should be good to merge.

sajarin · 2022-09-26T17:19:23Z

Thanks @krisjan-oldekamp for the PR and @tuanchris for the review!

…rts (airbytehq#16629) * Updated Report API version + New reports * Fixed typo and added ability to set fetch period to a fixed range * Removed old code + updated connection test doc * minor fix for python format * Bump version to 0.2.26 and updated docs * Docs update, fix typo in PR number Co-authored-by: Tuan Nguyen <[email protected]> Co-authored-by: Sajarin <[email protected]>

Updated Report API version + New reports

906e3f1

octavia-squidington-iii added bounty community connectors/source/amazon-seller-partner labels Sep 13, 2022

github-actions bot added the area/connectors Connector related issues label Sep 13, 2022

sajarin changed the title ~~Updated Report API version + New reports~~ Source: Amazon Seller Partner - Updated Report API version + New reports Sep 13, 2022

sajarin added the bounty-XL Maintainer program: claimable extra large bounty PR label Sep 13, 2022

Fixed typo and added ability to set fetch period to a fixed range

7a1bed1

tuanchris suggested changes Sep 21, 2022

View reviewed changes

Removed old code + updated connection test doc

4e66038

minor fix for python format

ebcc07c

Merge pull request #1 from tuanchris/stacktonic-com/fix_python_format

ee23067

minor fix for python format

krisjan-oldekamp added 2 commits September 24, 2022 10:37

Bump version to 0.2.26 and updated docs

6bb7ddf

Merge remote-tracking branch 'origin/master'

c0fd513

github-actions bot added the area/documentation Improvements or additions to documentation label Sep 24, 2022

Docs update, fix typo in PR number

11a20bc

Merge branch 'master' into master

c75b02d

sajarin approved these changes Sep 26, 2022

View reviewed changes

sajarin merged commit 3939ff3 into airbytehq:master Sep 26, 2022

This was referenced Sep 27, 2022

Bump Airbyte version from 0.40.9 to 0.40.10 #17285

Closed

Bump Airbyte version from 0.40.9 to 0.40.10 #17359

Merged

sajarin assigned tuanchris Oct 6, 2022

bleonard added the reward-200 label Oct 31, 2022

roman-yermilov-gl mentioned this pull request Jun 6, 2023

🐛 Source Amazon Seller Partner: Fix non vendor accounts connector create/check issue #27050

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source: Amazon Seller Partner - Updated Report API version + New reports #16629

Source: Amazon Seller Partner - Updated Report API version + New reports #16629

krisjan-oldekamp commented Sep 13, 2022 •

edited by lazebnyi

Loading

CLAassistant commented Sep 13, 2022 •

edited

Loading

sajarin commented Sep 13, 2022

sajarin commented Sep 14, 2022

tuanchris commented Sep 15, 2022

krisjan-oldekamp commented Sep 19, 2022

tuanchris commented Sep 19, 2022

krisjan-oldekamp commented Sep 20, 2022

tuanchris left a comment •

edited

Loading

tuanchris Sep 21, 2022

krisjan-oldekamp commented Sep 21, 2022

tuanchris commented Sep 21, 2022

krisjan-oldekamp commented Sep 22, 2022

tuanchris commented Sep 22, 2022

sajarin commented Sep 23, 2022 •

edited by github-actions bot

Loading

sajarin commented Sep 23, 2022 •

edited by github-actions bot

Loading

sajarin commented Sep 23, 2022

krisjan-oldekamp commented Sep 24, 2022

sajarin commented Sep 24, 2022 •

edited by github-actions bot

Loading

krisjan-oldekamp commented Sep 24, 2022

krisjan-oldekamp commented Sep 26, 2022

sajarin commented Sep 26, 2022

sajarin commented Sep 26, 2022

Source: Amazon Seller Partner - Updated Report API version + New reports #16629

Source: Amazon Seller Partner - Updated Report API version + New reports #16629

Conversation

krisjan-oldekamp commented Sep 13, 2022 • edited by lazebnyi Loading

What

Updated Amazon Seller Reports API verison

Updated connector test

Added new reports

🚨 User Impact 🚨

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

CLAassistant commented Sep 13, 2022 • edited Loading

sajarin commented Sep 13, 2022

sajarin commented Sep 14, 2022

tuanchris commented Sep 15, 2022

krisjan-oldekamp commented Sep 19, 2022

tuanchris commented Sep 19, 2022

krisjan-oldekamp commented Sep 20, 2022

tuanchris left a comment • edited Loading

Choose a reason for hiding this comment

tuanchris Sep 21, 2022

Choose a reason for hiding this comment

krisjan-oldekamp commented Sep 21, 2022

tuanchris commented Sep 21, 2022

krisjan-oldekamp commented Sep 22, 2022

tuanchris commented Sep 22, 2022

sajarin commented Sep 23, 2022 • edited by github-actions bot Loading

Build Passed

sajarin commented Sep 23, 2022 • edited by github-actions bot Loading

sajarin commented Sep 23, 2022

krisjan-oldekamp commented Sep 24, 2022

sajarin commented Sep 24, 2022 • edited by github-actions bot Loading

krisjan-oldekamp commented Sep 24, 2022

krisjan-oldekamp commented Sep 26, 2022

sajarin commented Sep 26, 2022

sajarin commented Sep 26, 2022

krisjan-oldekamp commented Sep 13, 2022 •

edited by lazebnyi

Loading

CLAassistant commented Sep 13, 2022 •

edited

Loading

tuanchris left a comment •

edited

Loading

sajarin commented Sep 23, 2022 •

edited by github-actions bot

Loading

sajarin commented Sep 23, 2022 •

edited by github-actions bot

Loading

sajarin commented Sep 24, 2022 •

edited by github-actions bot

Loading