Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lower scan interval for OpenSky #92593

Merged
merged 4 commits into from
May 5, 2023
Merged

Conversation

joostlek
Copy link
Member

@joostlek joostlek commented May 5, 2023

Proposed change

Lower the rate limit for OpenSky. Currently it polls every 4 minutes, but that triggers a 429 giving a not valid JSON back. According to the docs, anonymous users have up to 400 credits per day to give. 400/24 = ~16,7 requests per hour. To be safe the rate limit is put to every 4 minutes.

Type of change

  • Dependency upgrade
  • Bugfix (non-breaking change which fixes an issue)
  • New integration (thank you!)
  • New feature (which adds functionality to an existing integration)
  • Deprecation (breaking change to happen in the future)
  • Breaking change (fix/feature causing existing functionality to break)
  • Code quality improvements to existing code or addition of tests

Additional information

  • This PR fixes or closes issue: fixes OpenSky #92540
  • This PR is related to issue:
  • Link to documentation pull request:

Checklist

  • The code change is tested and works locally.
  • Local tests pass. Your PR cannot be merged unless tests pass
  • There is no commented out code in this PR.
  • I have followed the development checklist
  • I have followed the perfect PR recommendations
  • The code has been formatted using Black (black --fast homeassistant tests)
  • Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

If the code communicates with devices, web services, or third-party tools:

  • The manifest file has all fields filled out correctly.
    Updated and included derived files by running: python3 -m script.hassfest.
  • New or updated dependencies have been added to requirements_all.txt.
    Updated by running python3 -m script.gen_requirements_all.
  • For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.
  • Untested files have been added to .coveragerc.

To help with the load of incoming pull requests:

@joostlek
Copy link
Member Author

joostlek commented May 5, 2023

I am also thinking of adding a check if the status of the api call is not 200, make the sensor unavailable. Would that be a good practice?

@epenet
Copy link
Contributor

epenet commented May 5, 2023

I am also thinking of adding a check if the status of the api call is not 200, make the sensor unavailable. Would that be a good practice?

Yes, that would be good practise (in a separate PR).

A couple of other good practises that should also be considered:

  • move the logic to an external library, also moving away from request (sync) and moving to httpx or aiohttp (async)
  • move to config flow
  • ...

If you are planning to do more work on OpenSky you should consider adding yourself as a code owner.

@joostlek
Copy link
Member Author

joostlek commented May 5, 2023

Maybe later, just wanted to fix this bug haha. First going to work on LastFM and Twitch :)

@epenet epenet added this to the 2023.5.2 milestone May 5, 2023
Copy link
Contributor

@epenet epenet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

@joostlek
Copy link
Member Author

joostlek commented May 5, 2023

Hold up, I think I am making a mistake here. Just happened to have the opensky page open, and it says that requesting /states/all (without any location details) costs 4 credits, so that would mean 100 requests per day. So I think for the integration to actually make sense it should change functionality (as it now just fetches all flights and looks up the distance and then checks if that distance is under the _radius)

(Also note, I did test this by just keeping it running locally and open debugger when the response wasn't 200, but I did not test it for a full day)

@balloob balloob marked this pull request as draft May 5, 2023 12:41
@balloob
Copy link
Member

balloob commented May 5, 2023

@joostlek converted it to draft. Mark it ready for review when it's ready. An initial step for a hot fix would be to just hit the /all endpoint with an interval that is within limits, and upgrade what is fetched for the next major release in a follow-up PR.

@balloob balloob modified the milestones: 2023.5.2, 2023.5.3 May 5, 2023
@joostlek joostlek marked this pull request as ready for review May 5, 2023 12:56
Copy link
Contributor

@epenet epenet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

@balloob balloob merged commit 6d1e607 into home-assistant:dev May 5, 2023
@balloob balloob modified the milestones: 2023.5.3, 2023.5.2 May 5, 2023
balloob pushed a commit that referenced this pull request May 5, 2023
* Lower scan interval for opensky to avoid hitting rate limit

* Lower scan interval for opensky to avoid hitting rate limit

* Update homeassistant/components/opensky/sensor.py

Co-authored-by: epenet <[email protected]>

* Update homeassistant/components/opensky/sensor.py

Co-authored-by: epenet <[email protected]>

---------

Co-authored-by: epenet <[email protected]>
@balloob balloob mentioned this pull request May 5, 2023
@OzGav
Copy link
Contributor

OzGav commented May 5, 2023

Whilst this fixes the issue of the component exceeding the api limit it makes it less than useful as only checking for aircraft every 15mins isn’t very useful. As this PR was approved it would be great if mine, that was closed, could be reconsidered as it fixes this issue properly (although I do note it doesn’t meet the new component architecture requirements but I hope some latitude could be given for existing components) #91162

edit: I have actually improved the code further since that PR and it is working well and capturing such things as server unavailable as was mentioned above as well as only pulling the necessary data from the server (instead of every flight in the world)

@milandzuris
Copy link

Maybe later, just wanted to fix this bug haha. First going to work on LastFM and Twitch :)

Twitch have more better integration on HACS

@github-actions github-actions bot locked and limited conversation to collaborators May 7, 2023
@joostlek joostlek deleted the opensky_ratelimit branch May 27, 2023 13:56
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

OpenSky
5 participants