🐛Source Google search console: Fix incremental sync: keep all urls in state object #9194

grubberr · 2021-12-29T17:24:24Z

Signed-off-by: Sergey Chvalyuk [email protected]

What

There is a bug in SearchAnalytics.get_updated_state, it keeps state only for one (not multiple) url
We need to fix it.

How

Describe the solution

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr · 2021-12-31T12:27:58Z

/test connector=connectors/source-google-search-console

🕑 connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/1640729725
✅ connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/1640729725
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      6    92%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              242     96    60%
	 source_acceptance_test/tests/test_full_refresh.py       38      0   100%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     17    69%
	 source_acceptance_test/utils/compare.py                 62     23    63%
	 source_acceptance_test/utils/connector_runner.py       110     48    56%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  979    404    59%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                            Stmts   Miss  Cover
	 -----------------------------------------------------------------------------------
	 source_google_search_console/__init__.py                            2      0   100%
	 source_google_search_console/service_account_authenticator.py      14      6    57%
	 source_google_search_console/source.py                             37     22    41%
	 source_google_search_console/streams.py                           118     27    77%
	 -----------------------------------------------------------------------------------
	 TOTAL                                                             171     55    68%

sherifnada · 2022-01-05T00:00:51Z

@grubberr thanks for creating this! what's needed to wrap it up?

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr · 2022-01-05T15:01:16Z

/test connector=connectors/source-google-search-console

🕑 connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/1658936725
✅ connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/1658936725
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      6    92%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              242     96    60%
	 source_acceptance_test/tests/test_full_refresh.py       38      0   100%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     17    69%
	 source_acceptance_test/utils/compare.py                 62     23    63%
	 source_acceptance_test/utils/connector_runner.py       110     48    56%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  979    404    59%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                            Stmts   Miss  Cover
	 -----------------------------------------------------------------------------------
	 source_google_search_console/__init__.py                            2      0   100%
	 source_google_search_console/service_account_authenticator.py      14      6    57%
	 source_google_search_console/source.py                             37     22    41%
	 source_google_search_console/streams.py                           118     27    77%
	 -----------------------------------------------------------------------------------
	 TOTAL                                                             171     55    68%

antixar

@grubberr , please publish your changes

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr · 2022-01-05T16:03:56Z

/test connector=connectors/source-google-search-console

🕑 connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/1659206607
✅ connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/1659206607
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      6    92%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              242     96    60%
	 source_acceptance_test/tests/test_full_refresh.py       38      0   100%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     17    69%
	 source_acceptance_test/utils/compare.py                 62     23    63%
	 source_acceptance_test/utils/connector_runner.py       110     48    56%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  979    404    59%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                            Stmts   Miss  Cover
	 -----------------------------------------------------------------------------------
	 source_google_search_console/__init__.py                            2      0   100%
	 source_google_search_console/service_account_authenticator.py      14      6    57%
	 source_google_search_console/source.py                             37     22    41%
	 source_google_search_console/streams.py                           118     27    77%
	 -----------------------------------------------------------------------------------
	 TOTAL                                                             171     55    68%

grubberr · 2022-01-05T16:19:30Z

/publish connector=connectors/source-google-search-console

🕑 connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/1659271481
✅ connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/1659271481

Signed-off-by: Sergey Chvalyuk <[email protected]>

bugfix: keep all urls in state object

cdde769

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr self-assigned this Dec 29, 2021

github-actions bot added the area/connectors Connector related issues label Dec 29, 2021

grubberr linked an issue Dec 29, 2021 that may be closed by this pull request

🐛 Source Google Search Console: incremental stream not working incrementally #8798

Closed

grubberr temporarily deployed to more-secrets December 29, 2021 17:24 Inactive

google-search-console.md updated

d76d4e1

Signed-off-by: Sergey Chvalyuk <[email protected]>

github-actions bot added the area/documentation Improvements or additions to documentation label Dec 29, 2021

grubberr changed the title ~~🐛Source Google search console: keep all urls in state object~~ 🐛Source Google search console: Fix incremental sync: keep all urls in state object Dec 29, 2021

grubberr temporarily deployed to more-secrets December 29, 2021 17:29 Inactive

grubberr requested a review from midavadim December 29, 2021 17:33

midavadim approved these changes Dec 30, 2021

View reviewed changes

grubberr requested a review from midavadim December 30, 2021 15:20

grubberr added 2 commits December 31, 2021 13:39

Merge branch 'master' into grubberr/8798-source-google-search-console

ca2704f

Signed-off-by: Sergey Chvalyuk <[email protected]>

fix 0.1.10 -> 0.1.11

a2f6532

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr temporarily deployed to more-secrets December 31, 2021 11:42 Inactive

0.1.10 -> 0.1.11 (fix)

cbdacae

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr temporarily deployed to more-secrets December 31, 2021 11:44 Inactive

test_updated_state added

aa3e349

grubberr temporarily deployed to more-secrets December 31, 2021 12:26 Inactive

jrhizor temporarily deployed to more-secrets December 31, 2021 12:30 Inactive

grubberr requested a review from antixar December 31, 2021 13:18

grubberr added 2 commits January 5, 2022 16:47

Merge branch 'master' into grubberr/8798-source-google-search-console

c150461

add example of state structure

b9b30e0

Signed-off-by: Sergey Chvalyuk <[email protected]>

antixar approved these changes Jan 5, 2022

View reviewed changes

grubberr temporarily deployed to more-secrets January 5, 2022 15:02 Inactive

jrhizor temporarily deployed to more-secrets January 5, 2022 15:03 Inactive

Merge branch 'master' into grubberr/8798-source-google-search-console

3184eb7

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr temporarily deployed to more-secrets January 5, 2022 16:05 Inactive

jrhizor temporarily deployed to more-secrets January 5, 2022 16:06 Inactive

jrhizor temporarily deployed to more-secrets January 5, 2022 16:22 Inactive

source_specs.yaml updated

614d311

Signed-off-by: Sergey Chvalyuk <[email protected]>

grubberr merged commit 8ace9ea into master Jan 5, 2022

grubberr deleted the grubberr/8798-source-google-search-console branch January 5, 2022 16:40

grubberr temporarily deployed to more-secrets January 5, 2022 16:41 Inactive

jrhizor mentioned this pull request Jan 7, 2022

Bump Airbyte version from 0.35.3-alpha to 0.35.4-alpha #9353

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛Source Google search console: Fix incremental sync: keep all urls in state object #9194

🐛Source Google search console: Fix incremental sync: keep all urls in state object #9194

grubberr commented Dec 29, 2021 •

edited

Loading

grubberr commented Dec 31, 2021 •

edited by github-actions bot

Loading

sherifnada commented Jan 5, 2022

grubberr commented Jan 5, 2022 •

edited by github-actions bot

Loading

antixar left a comment

grubberr commented Jan 5, 2022 •

edited by github-actions bot

Loading

grubberr commented Jan 5, 2022 •

edited by github-actions bot

Loading

🐛Source Google search console: Fix incremental sync: keep all urls in state object #9194

🐛Source Google search console: Fix incremental sync: keep all urls in state object #9194

Conversation

grubberr commented Dec 29, 2021 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

grubberr commented Dec 31, 2021 • edited by github-actions bot Loading

sherifnada commented Jan 5, 2022

grubberr commented Jan 5, 2022 • edited by github-actions bot Loading

antixar left a comment

Choose a reason for hiding this comment

grubberr commented Jan 5, 2022 • edited by github-actions bot Loading

grubberr commented Jan 5, 2022 • edited by github-actions bot Loading

grubberr commented Dec 29, 2021 •

edited

Loading

grubberr commented Dec 31, 2021 •

edited by github-actions bot

Loading

grubberr commented Jan 5, 2022 •

edited by github-actions bot

Loading

grubberr commented Jan 5, 2022 •

edited by github-actions bot

Loading

grubberr commented Jan 5, 2022 •

edited by github-actions bot

Loading