DCAT JSON pagination

[mhogeweg]

Is the expectation that all of an agency's public datasets are listed in a single data.json (xml, html) file? this might result in huge files that are not easy to navigate for developers who are looking for some specific dataset.

has it been considered to implement a filter and pagination approach using opensearch (http://www.opensearch.org) or breaking big catalogs into chunks similar to what is done in the sitemap protocol (http://www.sitemaps.org/).

[rufuspollock]

Perhaps worth connecting this with http://spec.datacatalogs.org/ and specifically this discussion issue: okfn/data-catalog-spec#7

[konklone]

I don't think these are going to be navigated directly, like browsers scrolling through it in Chrome with JSONView turned on. It's there so it can get funneled into other systems that do implement things like pagination. Having one JSON file means agencies can meet the requirements through an abundance of mechanisms, even just publishing a static file and not having to run a hosted service at all.

[kachok]

Rationale for this is what @konklone said. Assuming that data.gov will be run on CKAN, developers can easily access same metadata via CKAN API which provides pagination and accomodate similar need for majority of developers.

[jpmckinney]

+1 single file for source document
+1 pagination for systems consuming that source document

[benbalter]

Sounds like the general consensus is that pagination is best handled upstream.

@mhogeweg thanks for taking the time to raise the issue. If you have a chance, would be awesome to see a pull request to document how to paginate, available resources, etc.