Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose an API for the selective reset of store caches #339

Closed
giohappy opened this issue Jul 27, 2022 · 7 comments
Closed

Expose an API for the selective reset of store caches #339

giohappy opened this issue Jul 27, 2022 · 7 comments

Comments

@giohappy
Copy link

giohappy commented Jul 27, 2022

At the moment the only available API for resetting a store's cache is the global /reset endpoint, which is clearly sub-optimal, to say the least:

  • if a GetCapabilities request arrives after the global reset the store will recollect the schemas for every layer
  • if a GetFeature request arrives, the internal XML cache will have to be recalculated, which requires recollecting the schemas for each layer

In the case of a DB store (as it is the case, for example, for any vector layer created in GeoNode) it means that in both the above cases we will have a spike of connections and requests to the DB.

Having a selective reset we can mitigate both situations. In particular, in case of a GetFeature request, the XML will still be re-created but it won't need to hit the DB, since most part of the schemas will still be inside the store cache.

For a selective reset, we envisage the following operations to be implemented and exposed by the API:

  • reset of a specific store, entirely
  • reset of a specific layer from a store (e.g. single table for a DB store)

the first could be exposed by the DataStores and CoverageStoress APIs:

  • [POST] /workspaces/<ws>/datastores/<cs>/reset
  • [POST] /workspaces/<ws>/coveragestores/<cs>/reset

The latter could be exposed by the FeatureTypes and the Coverages APIs:

  • [POST] /workspaces/<ws>/datastores/<ds>/featuretypes/<ft>/reset
  • [POST] /workspaces/<ws>/coveragestores/<cs>/coverages/<c>/reset
@giohappy giohappy changed the title Expose an API for the selective removal of store cache Expose an API for the selective removal of store caches Jul 27, 2022
@nmco
Copy link

nmco commented Jul 27, 2022

@fernando let's follow the usual workflow.

@aaime
Copy link
Member

aaime commented Jul 27, 2022

For raster layers, it's just a matter of re-creating the reader, there is no sensible way to do it otherwise (and most of the raster stores are single layer anyways, netcdf being modified need to have their header re-parsed anyways). This can be fully controlled on the GeoServer side.

For vector layers, we'll need to open an extension point in GeoTools DataStore so that stores that do any sort of caching, can be cleaned up internally. For example, ContentDataStore, the base class for most stores (but not all, mind!) has an internal cache, the "entries" field, that holds information about each feature type, including its schema.

I'm guessing a CachingStore interface could be created, that has a "clear(String typeName)" and a "clear(Name name)" methods, which then various stores can decide to implement (GeoNode wise, they care about the PostGIS store only AFAIK,
which implies making the changes in its ContentDataStore superclass, I believe).

@giohappy giohappy changed the title Expose an API for the selective removal of store caches Expose an API for the selective reset of store caches Jul 27, 2022
@nmco nmco assigned taba90 and unassigned nmco and fernandor777 Jul 29, 2022
@simboss simboss assigned aaime and unassigned taba90 and simboss Aug 2, 2022
@aaime
Copy link
Member

aaime commented Aug 3, 2022

I'm going to write a quick proposal, I'll make the estimate based on the reactions I get. Should be well received IMHO.

@aaime
Copy link
Member

aaime commented Aug 4, 2022

Seems be getting well received, I estimate a couple of days of work, tops. I'd urge to get this done sooner rather than later, as the 2.22 RC official deadline is in a few days (18th of the month), missing it would cause backports friction.

@simboss
Copy link
Member

simboss commented Aug 4, 2022

note, we need this on 2.21.x and 2.20.x if possible.

@aaime
Copy link
Member

aaime commented Aug 10, 2022

Proposal and PR have been out for a few days now.

The proposal hasn't yet received all votes, in case of missing votes we'll be able to declare it passed Aug 13th (10 days rule).

@giohappy
Copy link
Author

giohappy commented Sep 6, 2022

@simboss simboss closed this as completed Oct 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants