Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

how to read several collection more quickly #1480

Open
adryen31 opened this issue Dec 5, 2024 · 1 comment
Open

how to read several collection more quickly #1480

adryen31 opened this issue Dec 5, 2024 · 1 comment
Labels

Comments

@adryen31
Copy link

adryen31 commented Dec 5, 2024

Dear,

I have a catalog with approximately 61000 link as rel 'child'

I search a mean to parse and compute it more quickly,

image

Here my code:
satellites = catalog.get_collections() for satellite in satellites: elapsed_time = 0 satellite_to_add.append(convert_to_satellite(satellite, None)) elapsed_time += time.time() - start_time print("elapsed_time to fetch: " + str(elapsed_time))

But I saw that for process 65K, it took 49 minutes. I search a mean to increase the speed of process, I think is the time that the 'for' fetch the next data of satellite (because my process in convert_to_satellite is not reprensentative, only 10 lines on mapping json attribute).

Do you know a mean to process quickly, for example the api with get_collection() fetch more than one data satellite ?
I doubt that the api will make a new HTTP request every iteration. Does it work like this? I’m in doubt. If not, do you know a good method?

have a nice day
Adrien

@gadomski
Copy link
Member

It's a bit tricky to tell what's going on here, but in general if you're searching a STAC API you should be using pystac-client. Can you provide a code snippet showing all of what you're doing? You can format your code snippit like this:

```python
from pystac_client import Client

client = Client.open("http://my.great.stac.api/v1")
client.search(...)
```

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants