-
Notifications
You must be signed in to change notification settings - Fork 11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🤟 Fix user-agent banning by ESIOS server #35
Conversation
and rotate it if banning is detected, using common User-Agent browser identifiers (server returns 403 status for a PUBLIC url 😤, and it's just banning us based on that 😢)
so in the end, is only 1 request/day 🤩
🚀 For those who cannot wait until the fix is integrated and published in HA Core, just unpack the following ZIP file in your Or, if you already have an ESIOS token, [maybe this is more interesting for you at this moment], and want to become a beta-tester for a new (or just evolved) integration 🧪, try this one instead 😜: esios.zip |
The custom pvpc is running, thanks. I'm also trying esios, and I got a token today, but I can't see the Esios in the integration list... (yes the folder is inside custom_components) HA over raspi, core-2021.11.5. Any directions appreciated! |
Yes! it works again with custom pvpc. Tomorrow I'll probably get the esios token and test the other one. |
Testing... Logger: custom_components.esios.esios_data.api 2021-12-01 23:00:00+00:00 not found in {} -> list index out of range |
I have now an ESIOS token, do you have instructions about its configuration with this esios.zip version? |
Go to integrations, add new one, look up for esios api, and then enter your token and save. EDIT: working |
Applying the quick fix using token ends up in REE forbidding the API call again. Not sure how the code works, but if it triggers many updates while the price curves don't change, i.e. we are pulling the same data over and over again, they end up banning the access via token as well. |
I'm now using the new ESIOS integration. It works!! 😃🥳. Congratulations for your work and will inform you in case of any issues. |
Hi @fernandodpr, I'm trying to install ESIOS as a custom component but the integration do not appear on the list, Is there any special procedure I'm missing? Thanks! |
Hi @deltazerorsan,
|
Hola, Logger: aiopvpc.pvpc_data Forbidden error with 'https://api.esios.ree.es/archives/70/download_json?locale=es&date=2021-03-28' -> Headers: <CIMultiDictProxy('Content-Type': 'text/html', 'Cache-Control': 'no-cache, no-store', 'Connection': 'close', 'Content-Length': '859', 'X-Iinfo': '7-36927098-0 2NNN RT(1638458838950 41) q(0 -1 -1 1) r(0 -1) B15(11,1720275,0)', 'Strict-Transport-Security': 'max-age=31536000', 'Set-Cookie': 'visid_incap_1885724=KidT3bTWTI2TGT+AnDKTHtblqGEAAAAAQUIPAAAAAABu0gwW9yWPEzn0GXj7gbnN; expires=Thu, 01 Dec 2022 17:05:24 GMT; HttpOnly; path=/; Domain=.esios.ree.es; Secure; SameSite=None', 'Set-Cookie': 'incap_ses_510_1885724=EKq/CL/EZERT51dhzOITB9blqGEAAAAAouTHo/Y/3dMijGy+Jb4gqg==; path=/; Domain=.esios.ree.es; Secure; SameSite=None')> Logger: aiopvpc.pvpc_data BAD Download of PVPC prices from 2021-11-29 09:00:00 to 2021-12-02 13:27:15.397859 in 0.13 sec |
I've followed this steps with my personal token and is working!. |
SOLVED, IT WAS A PROBLEM IN HA WITH ANY MODULE ESIOS WORKING PROPERLY:
Yeah! My problem is between step 2 and 3... no ESIOS API Data... Are you using a development version of HA? Thanks |
No, I'm using standard version of HA. Could you look into /config/info. To check if HA It's detecting the integration? |
Nope, it's not there... I have other three custom components, including pvpc, working correctly... mmm, so estrange... I got the zip from here: https://github.com/azogue/aiopvpc/files/7531844/esios.zip |
Is there any official repository for the API version of pvpc_hourly_pricing integration (ESIOS API connector) The one referenced in the integration (esios.zip) doesn't seem to work: |
gracias :), llevaba varios días usando velas por no saber el precio! xD. |
@josecarre el sensor nuevo por defecto tiene otra id. Lo que puedes hacer es modificar la id una vez instalado y colocar la misma que tenías anteriormente (recuerda eliminar la otra integración). De esa forma en los gráficos tendrás histórico, en grafana también en caso de que lo uses y tampoco tendrás que modificar tus automatizaciones. |
I've tried the two options and both are working OK for me. Thanks! |
I have tried |
I tried also esios.zip with the personal token and it works. But it only shows the price on the day, not for the next day. Am I alone? |
You need to wait until 20:00 ... 20:30 and then you have next day. Untill
00:00.
It works fine for me.
El mar., 21 dic. 2021 12:31, samolo75 ***@***.***> escribió:
… I tried also esios.zip with the personal token and it works. But it only
shows the price on the day, not for the next day. Am I alone?
—
Reply to this email directly, view it on GitHub
<#35 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AM33FISVUM6M7DDNMHDGW2DUSBQQ5ANCNFSM5JFLYPSQ>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
I am pretty sure I did check late on the day, for a few days and I never got the next day prices. I will keep checking. Thanks |
Yeah it's a little trickier cause the attributes for next day are present only between the time REE offers them and 00:00. Right now 22:50 CET I can see the "next day" attributes, like:
but later they disappear and you get an Unknown wherever you are using it... But tomorrow they reappear whit the values of the next day, and so on. |
Ok. Sorry. It actually works fine when I checked yesterday. I guess I didn't do it right before. |
closes #34
related to home-assistant/core#60556
Quick-Fix motivated by the last change in the ESIOS server (on 2021-11-30 😱), which is now apparently banning HomeAssistant requests, filtering us out because of the 'User-Agent' headers data 😤, as the server is returning a 403 status code error for a PUBLIC url 🤷.
So, until future moves to authenticated endpoints, or alternative data sources, let's continue playing 🐈 and 🐁 with the ESIOS webadmins 🤣🤣
[Serious note for those webadmins]: I'm just a volunteer, and I'm an email away for contact, so please, if you are reading this, give me a call before filtering us all out 🤝🥺
Changes:
⚡ Substantially decrease the number of API requests to ESIOS, avoiding unnecessary calls to refresh data for the same electricity prices.
Before, when used from the
pvpc_hourly_pricing
HA Core integration, the ESIOS API was called 2 times/hour from 0h to 20h, and 4 times/hour in the evening, from 20h to 0h, retrieving today + tomorrow prices.This makes a total of ~56 requests/day, which is not a lot 😅, but it seems the aggregated total for the HA user base (🔥 >30k requests/day just counting users pushing HA analytics) is being some kind of a problem for ESIOS,
as it looks like they're trying to bane us 🥺😭
Now, the API handler avoids calls to retrieve already available prices, cutting down the number of requests to just 1-2 requests/day 🤩
🐛 Set standard
User-Agent
header info, to try to avoid server-side banning 🙈, and rotate it if banning is detected, using common User-Agent browser identifiers.♻️ Minor code refactor to prepare for future library changes, in order to move to authenticated API endpoints in future versions.