Offline copy of data #2530
Comments
|
Thank you for opening your first issue here 👍. Be sure to follow the issue template if you chose one. |
|
I'm actually working on something like that 😸 |
|
Hi @anthonyharrison, thank you for the idea. endoflife.date is using the static site generator Jekyll. Given the static nature of endoflife.date that may be difficult to implement: JSON and HTML file are only generated when there is an update on the master branch. |
|
Would a dataset published via a NPM package be good enough? Or a separate git repository that could fulfill the "update whenever needed" requirement easily? I've been wanting to do this for a while, by means of uploading the generated JSON files (preferably in the v1 API format) to a release on GitHub.
As an aside, we have an endpoint that solves this: https://endoflife.date/api/all.json. @adriens Could you detail your plan to solve for this? |
|
The API endpoint is a good start. Just getting a download of all of the
data in JSON would be very useful. To find out what has changed since the
last download could be done a number of ways. Simplest is to say if there
has been any changes since a particular date in which case just download
all the data again. The more elegant but slightly more complex would be to
download all the changes since a particular date rather than all the data.
But given the current amount of data isn't huge I imagine the first
solution would be a good start.
I would rather not force the introduction of a new ecosystem (npm).
…On Sat, 18 Feb 2023, 14:18 Nemo, ***@***.***> wrote:
Would a dataset published via a NPM package be good enough? Or a separate
git repository that could fulfill the "update whenever needed" requirement
easily?
I've been wanting to do this for a while, by means of uploading the
generated JSON files (preferably in the v1 API format) to a release on
GitHub.
But this would also require that I know all of the products in the first
place
As an aside, we have an endpoint that solves this:
https://endoflife.date/api/all.json.
@adriens <https://github.com/adriens> Could you detail your plan to solve
for this?
—
Reply to this email directly, view it on GitHub
<#2530 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACAID24Q4TJCDSD74QGYSXTWYDK4ZANCNFSM6AAAAAAU7ZXDKQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Maybe would you appreciate this repo : https://github.com/adriens/endoflife.date-nested |
|
@adriens I can certainly use this as a starting point. However the https://endoflife.date/api/all.json already provides the data in JSON - if this was enhanced to include some more metadata e..g the date of the data dump, this would be the start of something very useful. |
@captn3m0 Would it not be possible to maintain a history of changes to the information contained within the _data directory and then return details of the products which have changed via an API? The API endpoints will allow me to get all of the data but they will require that I get all of the data and not just the updated? |
A timestamp containing the date of the json file would be easy to add, but it requires the v1 API format (under development, see #2080 and https://deploy-preview-2080--endoflife-date.netlify.app/docs/api/v1/ for a preview). Unfortunately the current format (v0) cannot be updated without introducing a breaking change, and we did not planned to add new endpoints. I do not mind adding a new |
|
Our current bandwidth usage is around ~50GB out of our 1TB limit, so I don't see any issue there. If this ever gets problematic due to this endpoint, we can set a redirect to another host/implement caching etc easily. However, I don't think we should be abusing our API to essentially serve a dataset. I can suggest few alternative approaches:
@anthonyharrison I'd be curious about the usecase here, to see if we can improve the API/documentation/roadmap further to account for this. |
|
We can also add a new json endpoint called NOTE : adding just revision_date to our current endpoints wont fix the main problem that users still need to redownload same big file if we wont implement this idea |
@usta, is XYZ the product name ? If yes the product files are not that big (2 to 20 KB each I would say), so I think sending two requests separately could take longer than retrieving all the data in one shot. Note that v1 product API endpoint already includes a |
@captn3m0 I am trying to develop an automated audit function which will identify whether a product is under support, under extended support or EOL and trigger some workflows For products which are nearing end of supptort, I want to be able to trigger a workflow to look at the upgrade path; for those which are EOL (or nearing EOL), I would want to trigger a different workflow. |
@marcwrobel Nope i mean all , upcomingEOL , ... endpoints |
|
@captn3m0 , here is a first proof of concept : https://www.kaggle.com/datasets/adriensales/endoflifedate/ Please notice that :
|
select category,
count(*)
from product_categories
group by category
having count(*) > 10;
|
|
There are some cool surprises I'm working on too, on the same topic. |
|
|
|
☝️ Other files will be added : does anyone want to give a try to a ; |
|
🤩 |
|
I opened a PR to implement the idea in #2530 (comment), since I liked that idea and would make use of it myself. @adriens I see this as orthogonal to your efforts. Your work seems much more full-featured as compared to the simple GitHub Action I wrote, but I still think having a GitHub Release with a simple file is useful. |
This is a major rework of the API with a lot of breaking changes. See CHANGELOG_API.md for more information. Note that we thought of disabling API generation in development (using JEKYLL_ENV like the Jekyll Feed plugin - see https://github.com/jekyll/jekyll-feed/blob/master/lib/jekyll-feed/generator.rb#L145), but it was finally reverted. It does not work well with Netlify preview, and generate production URL (i.e. https://endoflife.date URLs) in development which makes it difficult to use.
This is a major rework of the API with a lot of breaking changes. See CHANGELOG_API.md for more information. Note that we thought of disabling API generation in development (using JEKYLL_ENV like the Jekyll Feed plugin - see https://github.com/jekyll/jekyll-feed/blob/master/lib/jekyll-feed/generator.rb#L145), but it was finally reverted. It does not work well with Netlify preview, and generate production URL (i.e. https://endoflife.date URLs) in development which makes it difficult to use.
This is a major rework of the API with a lot of breaking changes. See CHANGELOG_API.md for more information. Note that we thought of disabling API generation in development (using JEKYLL_ENV like the Jekyll Feed plugin - see https://github.com/jekyll/jekyll-feed/blob/master/lib/jekyll-feed/generator.rb#L145), but it was finally reverted. It does not work well with Netlify preview, and generate production URL (i.e. https://endoflife.date URLs) in development which makes it difficult to use.
This is a major rework of the API with a lot of breaking changes. See CHANGELOG_API.md for more information. Note that we thought of disabling API generation in development (using JEKYLL_ENV like the Jekyll Feed plugin - see https://github.com/jekyll/jekyll-feed/blob/master/lib/jekyll-feed/generator.rb#L145), but it was finally reverted. It does not work well with Netlify preview, and generate production URL (i.e. https://endoflife.date URLs) in development which makes it difficult to use.
This is a major rework of the API with a lot of breaking changes. See CHANGELOG_API.md for more information. Note that we thought of disabling API generation in development (using JEKYLL_ENV like the Jekyll Feed plugin - see https://github.com/jekyll/jekyll-feed/blob/master/lib/jekyll-feed/generator.rb#L145), but it was finally reverted. It does not work well with Netlify preview, and generate production URL (i.e. https://endoflife.date URLs) in development which makes it difficult to use.
This is a major rework of the API with a lot of breaking changes. See CHANGELOG_API.md for more information. Note that we thought of disabling API generation in development (using JEKYLL_ENV like the Jekyll Feed plugin - see https://github.com/jekyll/jekyll-feed/blob/master/lib/jekyll-feed/generator.rb#L145), but it was finally reverted. It does not work well with Netlify preview, and generate production URL (i.e. https://endoflife.date URLs) in development which makes it difficult to use.
This is a major rework of the API with a lot of breaking changes. See CHANGELOG_API.md for more information. Note that we thought of disabling API generation in development (using JEKYLL_ENV like the Jekyll Feed plugin - see https://github.com/jekyll/jekyll-feed/blob/master/lib/jekyll-feed/generator.rb#L145), but it was finally reverted. It does not work well with Netlify preview, and generate production URL (i.e. https://endoflife.date URLs) in development which makes it difficult to use.
This is a major rework of the API with a lot of breaking changes. See CHANGELOG_API.md for more information. Note that we thought of disabling API generation in development (using JEKYLL_ENV like the Jekyll Feed plugin - see https://github.com/jekyll/jekyll-feed/blob/master/lib/jekyll-feed/generator.rb#L145), but it was finally reverted. It does not work well with Netlify preview, and generate production URL (i.e. https://endoflife.date URLs) in development which makes it difficult to use.
This is a major rework of the API with a lot of breaking changes. See CHANGELOG_API.md for more information. Note that we thought of disabling API generation in development (using JEKYLL_ENV like the Jekyll Feed plugin - see https://github.com/jekyll/jekyll-feed/blob/master/lib/jekyll-feed/generator.rb#L145), but it was finally reverted. It does not work well with Netlify preview, and generate production URL (i.e. https://endoflife.date URLs) in development which makes it difficult to use.
This is a major rework of the API with a lot of breaking changes. See CHANGELOG_API.md for more information. Note that we thought of disabling API generation in development (using JEKYLL_ENV like the Jekyll Feed plugin - see https://github.com/jekyll/jekyll-feed/blob/master/lib/jekyll-feed/generator.rb#L145), but it was finally reverted. It does not work well with Netlify preview, and generate production URL (i.e. https://endoflife.date URLs) in development which makes it difficult to use.
This is a major rework of the API with a lot of breaking changes. See CHANGELOG_API.md for more information. Note that we thought of disabling API generation in development (using JEKYLL_ENV like the Jekyll Feed plugin - see https://github.com/jekyll/jekyll-feed/blob/master/lib/jekyll-feed/generator.rb#L145), but it was finally reverted. It does not work well with Netlify preview, and generate production URL (i.e. https://endoflife.date URLs) in development which makes it difficult to use.
This is a major rework of the API with a lot of breaking changes. See CHANGELOG_API.md for more information. Note that we thought of disabling API generation in development (using JEKYLL_ENV like the Jekyll Feed plugin - see https://github.com/jekyll/jekyll-feed/blob/master/lib/jekyll-feed/generator.rb#L145), but it was finally reverted. It does not work well with Netlify preview, and generate production URL (i.e. https://endoflife.date URLs) in development which makes it difficult to use.
This is a major rework of the API with a lot of breaking changes. See CHANGELOG_API.md for more information. Note that we thought of disabling API generation in development (using JEKYLL_ENV like the Jekyll Feed plugin - see https://github.com/jekyll/jekyll-feed/blob/master/lib/jekyll-feed/generator.rb#L145), but it was finally reverted. It does not work well with Netlify preview, and generate production URL (i.e. https://endoflife.date URLs) in development which makes it difficult to use.
This is a major rework of the API with a lot of breaking changes. See CHANGELOG_API.md for more information. Note that we thought of disabling API generation in development (using JEKYLL_ENV like the Jekyll Feed plugin - see https://github.com/jekyll/jekyll-feed/blob/master/lib/jekyll-feed/generator.rb#L145), but it was finally reverted. It does not work well with Netlify preview, and generate production URL (i.e. https://endoflife.date URLs) in development which makes it difficult to use.
This is a major rework of the API with a lot of breaking changes. See CHANGELOG_API.md for more information. Note that we thought of disabling API generation in development (using JEKYLL_ENV like the Jekyll Feed plugin - see https://github.com/jekyll/jekyll-feed/blob/master/lib/jekyll-feed/generator.rb#L145), but it was finally reverted. It does not work well with Netlify preview, and generate production URL (i.e. https://endoflife.date URLs) in development which makes it difficult to use.
|
any ETA of deploying it? |
|
@TimBrown1611 I use the API endpoint https://endoflife.date/api/all.json to get the list of products and then request the data for each product. I store the set of data in a local file so I can then operate offline. |
You can also get a ready-to-use snapshot here on Kaggle : ⏳ endoflife.date database (duckdb) |
This is a major rework of the API with a lot of breaking changes. See CHANGELOG_API.md for more information. Note that we thought of disabling API generation in development (using JEKYLL_ENV like the Jekyll Feed plugin - see https://github.com/jekyll/jekyll-feed/blob/master/lib/jekyll-feed/generator.rb#L145), but it was finally reverted. It does not work well with Netlify preview, and generate production URL (i.e. https://endoflife.date URLs) in development which makes it difficult to use.
This is a major rework of the API with a lot of breaking changes. See CHANGELOG_API.md for more information. Note that we thought of disabling API generation in development (using JEKYLL_ENV like the Jekyll Feed plugin - see https://github.com/jekyll/jekyll-feed/blob/master/lib/jekyll-feed/generator.rb#L145), but it was finally reverted. It does not work well with Netlify preview, and generate production URL (i.e. https://endoflife.date URLs) in development which makes it difficult to use.
This is a major rework of the API with a lot of breaking changes. See CHANGELOG_API.md for more information. Note that we thought of disabling API generation in development (using JEKYLL_ENV like the Jekyll Feed plugin - see https://github.com/jekyll/jekyll-feed/blob/master/lib/jekyll-feed/generator.rb#L145), but it was finally reverted. It does not work well with Netlify preview, and generate production URL (i.e. https://endoflife.date URLs) in development which makes it difficult to use.
This is a major rework of the API with a lot of breaking changes. See CHANGELOG_API.md for more information. Note that we thought of disabling API generation in development (using JEKYLL_ENV like the Jekyll Feed plugin - see https://github.com/jekyll/jekyll-feed/blob/master/lib/jekyll-feed/generator.rb#L145), but it was finally reverted. It does not work well with Netlify preview, and generate production URL (i.e. https://endoflife.date URLs) in development which makes it difficult to use.
hello @adriens, |
|
Sure @tomersein : Daily 😆 |
|
Also the one relying on API v1 |
This is a major rework of the API with a lot of breaking changes. See CHANGELOG_API.md for more information. Note that we thought of disabling API generation in development (using JEKYLL_ENV like the Jekyll Feed plugin - see https://github.com/jekyll/jekyll-feed/blob/master/lib/jekyll-feed/generator.rb#L145), but it was finally reverted. It does not work well with Netlify preview, and generate production URL (i.e. https://endoflife.date URLs) in development which makes it difficult to use.
This is a major rework of the API with a lot of breaking changes. See CHANGELOG_API.md for more information. Note that we thought of disabling API generation in development (using JEKYLL_ENV like the Jekyll Feed plugin - see https://github.com/jekyll/jekyll-feed/blob/master/lib/jekyll-feed/generator.rb#L145), but it was finally reverted. It does not work well with Netlify preview, and generate production URL (i.e. https://endoflife.date URLs) in development which makes it difficult to use.
This is a major rework of the API with a lot of breaking changes. See CHANGELOG_API.md for more information. Note that we thought of disabling API generation in development (using JEKYLL_ENV like the Jekyll Feed plugin - see https://github.com/jekyll/jekyll-feed/blob/master/lib/jekyll-feed/generator.rb#L145), but it was finally reverted. It does not work well with Netlify preview, and generate production URL (i.e. https://endoflife.date URLs) in development which makes it difficult to use.
I really like the idea but to avoid repeated calls of the API for every product I would like data on, I would like to be maintain a local copy of the data and then only download updates each time I start my application (or after a particular time period e.g. only request updates once every 24 hours)
Ideally, I would be able to get the data in JSON format which I can then manage locally.
Alternative would be to call the API for every product to get the product data for each product. But this would also require that I know all of the products in the first place which given the dynamic nature of the data isn't very attractive.
The text was updated successfully, but these errors were encountered: