Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: added api to jobs handler to get required debug info #9703

Merged
merged 1 commit into from
Jan 25, 2022

Conversation

harshithmullapudi
Copy link
Contributor

@harshithmullapudi harshithmullapudi commented Jan 21, 2022

What

Adding API to get the required information to debug any job (sync | check connection).

How

Added API api/v1/jobs/get_debug_info this returns

{
    "job": {
        "id": 1,
        "configType": "sync",
        "configId": "ef296385-6796-413f-ac1b-49c4caba3f2b",
        "status": "succeeded",
        "airbyteVersion": "0.35.7-alpha",
        "sourceDefinition": {
            "sourceDefinitionId": "253487c0-2246-43ba-a21f-5116b20a2c50",
            "name": "Google Ads",
            "dockerRepository": "airbyte/source-google-ads",
            "dockerImageTag": "0.1.21",
            "documentationUrl": "https://docs.airbyte.io/integrations/sources/google-ads",
            "icon": "<?xml version=\"1.0\" encoding=\"utf-8\"?>\r\n<!-- Generator: Adobe Illustrator 19.0.0, SVG Export Plug-In . SVG Version: 6.00 Build 0)  -->\r\n<svg version=\"1.1\" id=\"Layer_1\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" x=\"0px\" y=\"0px\"\r\n\t viewBox=\"-393 31 500 500\" style=\"enable-background:new -393 31 500 500;\" xml:space=\"preserve\">\r\n<style type=\"text/css\">\r\n\t.st0{fill:#3C8BD9;}\r\n\t.st1{fill:#FABC04;}\r\n\t.st2{fill:#34A852;}\r\n\t.st3{fill:#E1C025;}\r\n</style>\r\n<g>\r\n\t<path class=\"st0\" d=\"M-201.3,156.2c3.5-9.3,8.4-17.8,15.6-24.7c28.8-28.1,76.4-21,96,14.3c14.7,26.8,30.3,52.9,45.4,79.4\r\n\t\tc25.3,44,50.9,87.9,75.9,132c21,36.9-1.8,83.5-43.5,89.8c-25.6,3.8-49.5-7.9-62.8-30.9c-22.2-38.7-44.5-77.3-66.7-115.8\r\n\t\tc-0.4-0.9-1-1.6-1.6-2.4c-2.4-1.9-3.4-4.7-4.9-7.2c-9.8-17.3-20-34.5-29.8-51.7c-6.3-11.2-12.9-22.2-19.3-33.4\r\n\t\tc-5.7-10-8.4-20.9-8.1-32.3C-204.7,167.4-203.9,161.5-201.3,156.2\"/>\r\n\t<path class=\"st1\" d=\"M-201.3,156.2c-1.3,5.3-2.5,10.6-2.8,16.2c-0.4,12.3,2.6,23.8,8.8,34.5c16.2,27.8,32.3,55.7,48.4,83.6\r\n\t\tc1.5,2.5,2.6,5,4.1,7.3c-8.8,15.3-17.6,30.4-26.6,45.7c-12.3,21.3-24.7,42.8-37.2,64.1c-0.6,0-0.7-0.3-0.9-0.7\r\n\t\tc-0.1-1.2,0.3-2.2,0.6-3.4c6-22,1-41.6-14.1-58.4c-9.3-10.1-21-15.9-34.5-17.8c-17.6-2.5-33.2,2.1-47.2,13.1\r\n\t\tc-2.5,1.9-4.1,4.7-7.1,6.2c-0.6,0-0.9-0.3-1-0.7c7.1-12.2,14-24.4,21-36.6c29.1-50.6,58.2-101.1,87.5-151.6\r\n\t\tC-202,157.3-201.6,156.8-201.3,156.2\"/>\r\n\t<path class=\"st2\" d=\"M-310.2,346.5c2.8-2.5,5.4-5.1,8.4-7.5c35.7-28.2,89.4-7.8,97.2,36.9c1.9,10.7,0.9,21-2.4,31.3\r\n\t\tc-0.1,0.9-0.3,1.6-0.6,2.5c-1.3,2.4-2.5,4.9-4,7.2c-13.1,21.6-32.3,32.3-57.6,30.7c-29-2.1-51.7-23.8-55.7-52.6\r\n\t\tc-1.9-14,0.9-27,8.1-39.1c1.5-2.6,3.2-5,4.9-7.6C-311.2,347.6-311.5,346.5-310.2,346.5\"/>\r\n\t<path class=\"st1\" d=\"M-310.2,346.5c-0.6,0.6-0.6,1.6-1.6,1.8c-0.1-1,0.4-1.6,1-2.4L-310.2,346.5\"/>\r\n\t<path class=\"st3\" d=\"M-207.6,409.7c-0.6-1,0-1.8,0.6-2.5c0.1,0.1,0.4,0.4,0.6,0.6L-207.6,409.7\"/>\r\n</g>\r\n</svg>\r\n"
        },
        "destinationDefinition": {
            "destinationDefinitionId": "a625d593-bba5-4a1c-a53d-2d246268a816",
            "name": "Local JSON",
            "dockerRepository": "airbyte/destination-local-json",
            "dockerImageTag": "0.2.9",
            "documentationUrl": "https://docs.airbyte.io/integrations/destinations/local-json",
            "icon": "<svg xmlns=\"http://www.w3.org/2000/svg\" enable-background=\"new 0 0 24 24\" viewBox=\"0 0 24 24\" fill=\"black\" width=\"24px\" height=\"24px\"><g><rect fill=\"none\" height=\"24\" width=\"24\"/><path d=\"M20.41,8.41l-4.83-4.83C15.21,3.21,14.7,3,14.17,3H5C3.9,3,3,3.9,3,5v14c0,1.1,0.9,2,2,2h14c1.1,0,2-0.9,2-2V9.83 C21,9.3,20.79,8.79,20.41,8.41z M7,7h7v2H7V7z M17,17H7v-2h10V17z M17,13H7v-2h10V13z\"/></g></svg>"
        }
    },
    "attempts": [
        {
            "attempt": {
                "id": 0,
                "status": "succeeded",
                "createdAt": 1643021711,
                "updatedAt": 1643021716,
                "endedAt": 1643021716,
                "bytesSynced": 46143,
                "recordsSynced": 13,
                "totalStats": {
                    "recordsEmitted": 13,
                    "bytesEmitted": 46143,
                    "stateMessagesEmitted": 0,
                    "recordsCommitted": 13
                },
                "streamStats": [
                    {
                        "streamName": "campaigns",
                        "stats": {
                            "recordsEmitted": 13,
                            "bytesEmitted": 46143,
                            "stateMessagesEmitted": null,
                            "recordsCommitted": 13
                        }
                    }
                ]
            },
            "logs": {
                "logLines": []"
                ]
            }
        }
    ]
}

Above is the sample output for the API

We have added information like

  1. Airbyte version
  2. Source
  3. Destination
  4. Attempts information including logs

@github-actions github-actions bot added area/api Related to the api area/platform issues related to the platform area/server labels Jan 21, 2022
@harshithmullapudi harshithmullapudi temporarily deployed to more-secrets January 21, 2022 19:58 Inactive
@harshithmullapudi harshithmullapudi temporarily deployed to more-secrets January 24, 2022 09:11 Inactive
@github-actions github-actions bot added the area/documentation Improvements or additions to documentation label Jan 24, 2022
@harshithmullapudi harshithmullapudi temporarily deployed to more-secrets January 24, 2022 13:06 Inactive
Copy link
Contributor

@cgardens cgardens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! I think this will be very helpful. Good place to start.

@harshithmullapudi harshithmullapudi changed the title first cut: added api to jobs handler feat: added api to jobs handler to get required debug info Jan 25, 2022
@harshithmullapudi harshithmullapudi merged commit 0dfbfdc into master Jan 25, 2022
@harshithmullapudi harshithmullapudi deleted the harshith/job-debug-info branch January 25, 2022 05:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/api Related to the api area/documentation Improvements or additions to documentation area/platform issues related to the platform area/server
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants