[ANT-745] Complete Lever Integration #2

marxlow · 2023-10-09T14:11:27Z

Requires setting up connectors with this initial state. 631152000000 = 1990-01-01, we are essentially initializing state for the first time which is necessary for our current implementation of incremental sub streams.

[
  {
    "streamDescriptor": {
      "name": "opportunities"
    },
    "streamState": {
      "updatedAt": 631152000000
    }
  }
]

github-actions · 2023-10-09T14:11:51Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

marxlow · 2023-10-11T08:58:03Z

airbyte-integrations/connectors/source-lever/source_lever/source.py

+    def parse_response(self, response: requests.Response, stream_slice:[Mapping[str, Any]], **kwargs) -> Iterable[Mapping]:
+        records = response.json()["data"]
+        if not records:
+            records = [{}]
+
+        for record in records:
+            record["opportunity"] = stream_slice["opportunity_id"]
+        yield from records


The fix for causing heartbeat issues.

Can you add a comment explaining the fix?

junquanlim

LGTM

NatElkins · 2023-10-11T14:04:11Z

I'm going to take a longer look at this later, hopefully today. But can you explain why we have fully forked the repo? I am not against it, just want to understand the reasoning (as opposed to simply using the SDK in a separate repo, if such a thing is possible).

NatElkins · 2023-10-11T14:04:32Z

I just worry about it being a maintenance burden.

NatElkins

I just gave this a quick first pass. Would love to touch base on this later this evening (your morning) for some synchronous review.

NatElkins · 2023-10-11T14:06:58Z

airbyte-integrations/connectors/source-lever/source_lever/source.py

+    # 1797 opportunities
+    # 1 record read
+    # stream_params = {"confidentiality": "all", "expand": "contact", "stage_id": "e54475bb-d3ad-43ff-b8b9-76c4fc38e78c" }
+
+    # 8311 opportunities
+    # stream_params = {"confidentiality": "all", "expand": "contact", "stage_id": "3a255cc8-0732-4bee-92bd-62acfec3572c" }


NatElkins · 2023-10-11T14:08:17Z

airbyte-integrations/connectors/source-lever/source_lever/source.py

+# TODO: Basic incremental stream
+class IncrementalLeverStream(LeverStream, ABC):
+    """
+    TODO fill in details of this class to implement functionality related to incremental syncs for your connector.
+         if you do not need to implement incremental sync for any streams, remove this class.
+    """
+
+    # TODO: Fill in to checkpoint stream reads after N records. This prevents re-reading of data if the stream fails for any reason.
+    state_checkpoint_interval = None
+
+    @property
+    def cursor_field(self) -> str:
+        """
+        TODO
+        Override to return the cursor field used by this stream e.g: an API entity might always use created_at as the cursor field. This is
+        usually id or date based. This field's presence tells the framework this in an incremental stream. Required for incremental.
+
+        :return str: The name of the cursor field.
+        """
+        return []
+
+    def get_updated_state(self, current_stream_state: MutableMapping[str, Any], latest_record: Mapping[str, Any]) -> Mapping[str, Any]:
+        """
+        Override to determine the latest state after reading the latest record. This typically compared the cursor_field from the latest record and
+        the current state and picks the 'most' recent cursor. This is how a stream's state is determined. Required for incremental.
+        """
+        return {}


If we are not using this, should we delete it? If we want to keep it around in case we plan on implementing it later, can we improve comment at the top?

Good catch but I will keep it for the next upcoming PR to add incremental extracts

NatElkins · 2023-10-11T14:08:56Z

airbyte-integrations/connectors/source-lever/source_lever/source.py

+    def path(self, stream_slice: Mapping[str, any] = None, **kwargs) -> str:
+        return f"opportunities/{stream_slice['opportunity_id']}/{self.name}"
+
+    def stream_slices(self, **kwargs) -> Iterable[Optional[Mapping[str, Any]]]:


Is it permitted to type Any as a more specific type? Or must it match Any?

Resolved this. It's permitted to type Any. Not being loose here, this is generated from the Airbyte script

NatElkins · 2023-10-11T14:09:17Z

airbyte-integrations/connectors/source-lever/source_lever/source.py

+    def parse_response(self, response: requests.Response, stream_slice:[Mapping[str, Any]], **kwargs) -> Iterable[Mapping]:
+        records = response.json()["data"]
+        if not records:
+            records = [{}]
+
+        for record in records:
+            record["opportunity"] = stream_slice["opportunity_id"]
+        yield from records


Can you add a comment explaining the fix?

NatElkins · 2023-10-11T14:13:43Z

airbyte-integrations/connectors/source-lever/source_lever/source.py

+    def parse_response(self, response: requests.Response, stream_slice:Mapping[str, Any],  **kwargs) -> Iterable[Mapping]:
+        yield from response.json()["data"]
+
+class Opportunities(LeverStream):


Should we add a link to the Opportunities API docs?

https://hire.lever.co/developer/documentation#list-all-opportunities

Or

https://hire.lever.co/developer/documentation#retrieve-a-single-opportunity

NatElkins · 2023-10-11T14:14:27Z

airbyte-integrations/connectors/source-lever/source_lever/source.py

+            return {"offset": response_data["next"]}
+
+    def request_params(
+        self, stream_state: Mapping[str, Any], stream_slice: Mapping[str, any] = None, next_page_token: Mapping[str, Any] = None


Should any be Any here?

Yup, resolved

NatElkins · 2023-10-11T14:14:51Z

airbyte-integrations/connectors/source-lever/source_lever/source.py

+    #     super().__init__(**kwargs)
+    #     self._start_date = start_date
+
+    def path(self, stream_slice: Mapping[str, any] = None, **kwargs) -> str:


Should this be Any?

Yeap, resolved

NatElkins · 2023-10-11T14:15:57Z

airbyte-integrations/connectors/source-lever/unit_tests/test_incremental_streams.py

+#
+# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
+#
+


Is this file actually being used?

Nope, but i'll leave it to write tests

NatElkins · 2023-10-11T14:16:14Z

airbyte-integrations/connectors/source-lever/unit_tests/test_source.py

+    source = SourceLever()
+    config_mock = MagicMock()
+    streams = source.streams(config_mock)
+    # TODO: replace this with your streams number


NatElkins · 2023-10-11T14:16:49Z

airbyte-integrations/connectors/source-lever/unit_tests/test_source.py

+    config_mock = MagicMock()
+    streams = source.streams(config_mock)
+    # TODO: replace this with your streams number
+    expected_streams_number = 2


Is this assertion correct?

marxlow · 2023-10-19T13:22:17Z

airbyte-integrations/connectors/source-lever/source_lever/source.py

+    def parse_response(self, response: requests.Response, stream_slice:Mapping[str, Any],  **kwargs) -> Iterable[Mapping]:
+        yield from response.json()["data"]
+
+"""


Important - need to pay special attention here @junquanlim

ANTs-745-complete-lever-integration

8e237b3

Fixed postings schema

dd00e53

marxlow commented Oct 11, 2023

View reviewed changes

junquanlim approved these changes Oct 11, 2023

View reviewed changes

NatElkins reviewed Oct 11, 2023

View reviewed changes

Marx added 4 commits October 16, 2023 14:20

Update offers schema to include approved data

ff2bc87

Remove applications sub stream & Expand applications in Opportunities

bbfce94

Complete incremental stream

3b7bfbe

Remove offer stage_id param used for testing

407da6c

marxlow commented Oct 19, 2023

View reviewed changes

Fix issue where cursor_field can be null

a08e00c

marxlow merged commit d003f19 into main Jan 2, 2024

marxlow deleted the ANTS-745 branch January 2, 2024 06:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ANT-745] Complete Lever Integration #2

[ANT-745] Complete Lever Integration #2

marxlow commented Oct 9, 2023 •

edited

Loading

github-actions bot commented Oct 9, 2023

marxlow Oct 11, 2023

NatElkins Oct 11, 2023

junquanlim left a comment

NatElkins commented Oct 11, 2023

NatElkins commented Oct 11, 2023

NatElkins left a comment

NatElkins Oct 11, 2023

marxlow Oct 19, 2023

NatElkins Oct 11, 2023

marxlow Oct 13, 2023

NatElkins Oct 11, 2023

marxlow Oct 19, 2023

NatElkins Oct 11, 2023

NatElkins Oct 11, 2023

marxlow Oct 19, 2023

NatElkins Oct 11, 2023

marxlow Oct 19, 2023

NatElkins Oct 11, 2023

marxlow Oct 19, 2023

NatElkins Oct 11, 2023 •

edited

Loading

marxlow Oct 19, 2023

NatElkins Oct 11, 2023

marxlow Oct 19, 2023

NatElkins Oct 11, 2023

marxlow Oct 19, 2023

marxlow Oct 19, 2023

[ANT-745] Complete Lever Integration #2

[ANT-745] Complete Lever Integration #2

Conversation

marxlow commented Oct 9, 2023 • edited Loading

github-actions bot commented Oct 9, 2023

Before Merging a Connector Pull Request

Choose a reason for hiding this comment

Choose a reason for hiding this comment

junquanlim left a comment

Choose a reason for hiding this comment

NatElkins commented Oct 11, 2023

NatElkins commented Oct 11, 2023

NatElkins left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NatElkins Oct 11, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marxlow commented Oct 9, 2023 •

edited

Loading

NatElkins Oct 11, 2023 •

edited

Loading