Merge pull request #531 from crim-ca/docker-quoting-estimator

.github/labeler-files.yml

@@ -92,8 +92,10 @@ feature/providers:
   - tests/functional/test_wps_provider.py
   - tests/wps_restapi/test_providers.py
 
-feature/quoting:
-  - weaver/wps_restapi/quotation
+feature/quotation:
+  - weaver/quotation/**/*
+  - weaver/wps_restapi/quotation/**
+  - docs/source/quotation.rst
 
 feature/vault:
   - weaver/vault/**/*

.github/labeler-words.yml

@@ -3,19 +3,34 @@
 #   - https://github.com/github/issue-labeler
 
 
+feature/AWS:
+  - '(Bucket|bucket)'
+  - Amazon
+  - AWS
+  - S3
+
 feature/CWL:
   - '(CWL|cwl)'
   - 'workflow'
 
 feature/oas:
   - '(OpenAPI|openapi)'
   - '(Swagger|swagger)'
+  - OAS
   - cornice
   - colander
 
 feature/docker:
   - '(Docker|docker)'
 
+feature/quotation:
+  - '(Quote|quote)'
+  - '(Quotation|quotation)'
+  - '(Bill|bill)'
+  - '(Billing|billing)'
+  - '(Estimator|estimator)'
+  - '(Estimation|estimation)'
+
 feature/vault:
   - '(Vault|vault)'
 

CHANGES.rst

@@ -12,11 +12,63 @@ Changes
 
 Changes:
 --------
-- No change.
+- Add ``weaver.quotation = true|false`` setting that allows control over the activation of all endpoints and operations
+  related to the `OGC API - Processes` |ogc-proc-ext-billing-short|_ and |ogc-proc-ext-quotation-short|_ extensions.
+- Add support to configure a quotation estimation algorithm for each respective `Process` with new requests
+  using ``GET``, ``PUT``, ``DELETE`` methods on ``/processes/{processID}/estimator`` endpoint. The configured
+  algorithm is provided by a reference `Docker` image defined by ``weaver.quotation_docker_[...]`` settings.
+  The algorithm itself expects a highly customizable configuration to estimate quotation parameters based on
+  conceptual categories, as defined by the |quote-estimator-config|_ schema optionally using versatile `ONNX`_
+  definitions. The `Docker` operation should return a JSON matching the |quote-estimation-result|_ schema, which is
+  parsed and included in the produced `Quote` based on provided `Process` execution parameters.
+- Add `Process` execution I/O pre-validation against the `Process` description before submitting the `Job` to avoid
+  unnecessary allocation of computing resources for erroneous cases that can easily be detected in advance.
+- Add ``$schema`` references to source `OGC API - Processes` or other schema registries for applicable content
+  definitions in responses.
+- Add missing `OGC API - Processes` schema references with published definitions
+  under ``https://schemas.opengis.net/ogcapi/processes/part1/1.0/`` when applicable.
+- Add ``links`` request query parameter to ``/processes`` and ``/providers/{providerID}/processes`` listing to
+  provide control over reporting of ``links`` for each `Process` summary item. By default, ``link=true`` and
+  automatically disable it when ``detail=false`` is specified.
+- Add missing ``405`` response schema for all `OpenAPI` endpoints as handled by the API when the requested HTTP method
+  is not applicable for the given path.
+- Renamed ``weaver.quote_sync_max_wait`` to ``weaver.quotation_sync_max_wait`` to better align with new configuration
+  settings for the |ogc-proc-ext-quotation-short| extension. Old value will still be checked for backward compatibility.
+- Renamed ``weaver.exec_sync_max_wait`` to ``weaver.execute_sync_max_wait`` to better align with the corresponding
+  parameter for quotation. Old value will still be checked for backward compatibility.
+- Add ``Lazify`` utility class for holding a string with delayed computation and caching that returns its representation
+  on-demand during formatting or other string operations to reduce the impact of its long generation. This can be used
+  with a callable returning a string representation that can be discarded without invocation on inactive logging levels.
+- Add ``count`` field to `JSON` output of endpoints that support paging to provide the number of items returned within
+  the paged result. Adjust the ``/quotations`` endpoint that was using it instead of ``total`` like it was done on other
+  listing endpoints.
+- Add ``detail`` query parameter for the ``/quotations`` endpoint to allow listing of `Quote` summary details instead
+  of only IDs by default, similarly to the ``/jobs`` endpoint.
+
+.. |ogc-proc-ext-billing-short| replace:: Billing
+.. _ogc-proc-ext-billing-short: https://github.com/opengeospatial/ogcapi-processes/tree/master/extensions/billing
+.. |ogc-proc-ext-quotation-short| replace:: Quotation
+.. _ogc-proc-ext-quotation-short: https://github.com/opengeospatial/ogcapi-processes/tree/master/extensions/quotation
+.. |quote-estimator-config| replace:: *Quote Estimator Configuration*
+.. _quote-estimator-config: weaver/schemas/quotation/quote-estimator.yaml
+.. |quote-estimation-result| replace:: *Quote Estimation Result*
+.. _quote-estimation-result: weaver/schemas/quotation/quote-estimation-result.yaml
+.. _ONNX: https://onnx.ai/
 
 Fixes:
 ------
-- No change.
+- Fix schema meta fields (``title``, ``summary``, ``description``, etc.) not being rendered in `OpenAPI` output for
+  keyword schemas (``allOf``, ``anyOf``, ``oneOf``, ``not``).
+- Fix schema definitions not being rendered in `OpenAPI` into the requested order
+  by ``_sort_first`` and ``_sort_after`` control attributes.
+- Fix request cache always invalidated when no explicit ``allowed_codes`` where provided in ``request_extra``, although
+  the request succeeded, causing caching optimization to never actually be used on following requests in this case.
+- Fix cached requests misbehaving when combined with ``stream=True`` argument due to contents not being stored in the
+  object for following requests, causing them to raise ``StreamConsumedError`` when calling the chunk iterator again.
+- Fix execution payloads for functional tests using ``WorkflowRESTScatterCopyNetCDF``, ``WorkflowRESTSelectCopyNetCDF``,
+  ``WorkflowWPS1ScatterCopyNetCDF`` and``WorkflowWPS1SelectCopyNetCDF`` processes, which requested invalid output
+  identifiers. Those erroneous definitions were detected using the new `Process` execution I/O pre-validation against
+  the corresponding `Process` descriptions on `Job` submission.
 
 .. _changes_4.29.0:
 

Makefile

@@ -719,16 +719,16 @@ fix-md: install-npm-remarklint fix-md-only	## fix Markdown linting problems afte
 ## -- Documentation targets ----------------------------------------------------------------------------------------- ##
 
 .PHONY: docs-build
-docs-build: clean-docs	## generate HTML documentation with Sphinx
+docs-build:		## generate HTML documentation with Sphinx
 	@echo "Generating docs with Sphinx..."
 	@bash -c '$(CONDA_CMD) $(MAKE) -C "$(APP_ROOT)/docs" html'
 	@-echo "Documentation available: file://$(APP_ROOT)/docs/build/html/index.html"
 
 .PHONY: docs-only
-docs-only: docs-build	  ## generate HTML documentation with Sphinx (alias)
+docs-only: docs-build	## generate HTML documentation with Sphinx (alias)
 
 .PHONY: docs
-docs: install-doc docs-only  ## generate HTML documentation with Sphinx after dependencies installation
+docs: install-doc clean-docs docs-only	## generate HTML documentation with Sphinx after dependencies installation
 
 ## -- Versioning targets -------------------------------------------------------------------------------------------- ##
 

README.rst

@@ -14,11 +14,11 @@ Weaver
 Weaver (the nest-builder)
   *Weaver birds build exquisite and elaborate nest structures that are a rival to any human feat of engineering.
   Some of these nests are the largest structures to be built by birds.*
-  (`Eden <https://eden.uktv.co.uk/animals/birds/article/weaver-birds/>`_).
+  [`Eden <https://eden.uktv.co.uk/animals/birds/article/weaver-birds/>`_].
 
   *Although weavers are named for their elaborately woven nests, some are notable for their selective parasitic
   nesting habits instead.*
-  (`Wikipedia <https://en.wikipedia.org/wiki/Ploceidae>`_)
+  [`Wikipedia <https://en.wikipedia.org/wiki/Ploceidae>`_]
 
 `Weaver` is an OGC-API flavored |ems| that allows the execution of workflows chaining various
 applications and |wps| inputs and outputs. Remote execution is deferred by the `EMS` to one or many

config/weaver.ini.example

@@ -20,14 +20,15 @@ mongodb.port = 27017
 mongodb.db_name = weaver
 
 # caching
-cache.regions = doc, request, result
 cache.type = memory
 cache.doc.expire = 86400
 cache.doc.enabled = true
 cache.request.expire = 60
 cache.request.enabled = false
 cache.result.expire = 3600
 cache.result.enabled = false
+cache.quotation.expire = 3600
+cache.quotation.enabled = true
 
 # NOTE:
 #   For all below parameters, settings suffixed by `_url` are automatically generated from their corresponding `_path`
@@ -68,11 +69,23 @@ weaver.ssl_verify = true
 weaver.request_options =
 
 # --- Weaver Execution settings ---
-
 # maximum wait time allowed for Prefer header to run Job/Quote synchronously
 # over this limit, they will automatically fallback to asynchronous execution/estimation
-weaver.exec_sync_max_wait = 20
-weaver.quote_sync_max_wait = 20
+weaver.execute_sync_max_wait = 20
+
+# --- Weaver Quotation settings ---
+# enable support of quotation extension
+# https://github.com/opengeospatial/ogcapi-processes/tree/master/extensions/quotation
+weaver.quotation = true
+weaver.quotation_sync_max_wait = 20
+# parameters to accomplish quote estimation
+weaver.quotation_docker_image = registry.gitlab.com/crim.ca/clients/cubewerx/quote-estimator:0.2.0
+weaver.quotation_docker_username =
+weaver.quotation_docker_password =
+# currency conversion
+weaver.quotation_currency_default = USD
+weaver.quotation_currency_converter =
+weaver.quotation_currency_token =
 
 # --- Weaver CWL settings ---
 # NOTE: [experimental]

docs/examples/vault_upload.py

@@ -14,4 +14,4 @@
         "application/json; charset=UTF-8"
     )
 }
-requests.post("https://weaver.example.com/vault", files=files)
+requests.post("https://weaver.example.com/vault", files=files, timeout=5)

docs/source/appendix.rst

@@ -18,6 +18,10 @@ Glossary
         | Corresponds to a region, often provided by :term:`OGC` :term:`WKT` definition, employed for :term:`OpenSearch`
           queries in the context of :term:`EOImage` inputs.
 
+    API
+        | Application Programming Interface
+        | Most typically, referring to the use of HTTP requests following an :term:`OpenAPI` specification.
+
     Application Package
         General term that refers to *"what and how to execute"* the :term:`Process`. Application Packages provide the
         core details about the execution methodology of the underlying operation that defines the :term:`Process`, and
@@ -26,7 +30,16 @@ Glossary
         defined by another similar approach. See the :ref:`Application Package` section for all relevant details.
 
     AWS
-        Amazon Web Services
+        | Amazon Web Services
+        | In the context of `Weaver`, most often referring specifically to the use of :term:`S3` buckets.
+
+    Bill
+    Billing
+        Result from :ref:`quotation_billing` following :ref:`quote_estimation` when enabled on the `Weaver` instance.
+
+        .. seealso::
+            - :ref:`quotation`
+            - :ref:`conf_quotation`
 
     CLI
         | Command Line Interface
@@ -124,18 +137,30 @@ Glossary
         and responses, as well as :term:`I/O` definitions for :term:`Process` description.
 
         .. seealso::
-            https://spec.openapis.org/oas/v3.1.0
+            |OpenAPI-spec|_
 
     OGC
         |ogc|_
 
     OGC API - Processes
-        The new API that defines JSON REST-binding representation of :term:`WPS` :term:`Process` collection.
+        The new :term:`API` that defines :term:`JSON` REST-binding representation
+        of :term:`WPS` :term:`Process` collection.
+
+    ONNX
+        The |ONNX-long|_ standard is an open format employed for sharing machine learning model representations
+        with an agnostic approach across frameworks, tools, runtimes, and compilers.
+
+        .. seealso::
+            :ref:`quotation_estimator_model`
 
     OpenSearch
         Protocol of lookup and retrieval of remotely stored files.
         Please refer to :ref:`OpenSearch Data Source` for details.
 
+    OWS
+        | :term:`OGC` Web Services
+        | Family of services including :term:`WPS`, defined prior to the family of :term:`OGC` :term:`API` standards.
+
     Process
         Entity that describes the required inputs, produced outputs, and any applicable metadata for the execution of
         the defined script, calculation, or operation.
@@ -144,6 +169,22 @@ Glossary
         Entity that offers an ensemble of :term:`Process` under it. It is typically a reference to a remote service,
         where any :term:`Process` it provides is fetched dynamically on demand.
 
+    Quote
+    Quotation
+        Result from :ref:`quote_estimation` when enabled on the `Weaver` instance.
+
+        .. seealso::
+            - :ref:`quotation`
+            - :ref:`conf_quotation`
+
+    Quote Estimator
+    Quotation Estimator
+        A model that can provide cost estimations regarding the execution of a :term:`Process` to form a term:`Quote`.
+
+        .. seealso::
+            - :ref:`quotation_estimator_model`
+            - :ref:`conf_quotation`
+
     Request Options
         Configuration settings that can be defined for `Weaver` in order to automatically insert additional
         HTTP request parameters, authentication or other any relevant rules when target URLs are matched.

docs/source/cli.rst

@@ -356,7 +356,7 @@ any stored results from a successful execution.
 
 Sample Output:
 
-.. literalinclude:: ../../weaver/wps_restapi/examples/job_dismiss_success.json
+.. literalinclude:: ../../weaver/wps_restapi/examples/job_dismissed_success.json
     :language: json
 
 .. _cli_example_status:
@@ -406,7 +406,7 @@ Accomplishes the :term:`Job` listing request to obtain known :term:`Job` definit
 
 Sample Output:
 
-.. literalinclude:: ../../weaver/wps_restapi/examples/job_listing.json
+.. literalinclude:: ../../weaver/wps_restapi/examples/jobs_listing.json
     :language: json
 
 .. _cli_example_monitor:

docs/source/conf.py

@@ -17,12 +17,16 @@
 # pylint: disable=C0103,invalid-name
 # pylint: disable=C0209,consider-using-f-string
 
-import json
 import logging
 import os
 import re
 import sys
 
+# for OpenAPI generation
+import simplejson as json
+from pyramid.config import Configurator
+from sphinx.domains.std import warn_missing_reference
+
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
@@ -35,11 +39,7 @@
 sys.path.insert(0, os.path.abspath(DOC_PRJ_ROOT))
 
 from weaver import __meta__  # isort:skip # noqa: E402 # pylint: disable=C0413
-
-# for api generation
 from weaver.wps_restapi.api import get_openapi_json  # isort:skip # noqa: E402
-from pyramid.config import Configurator  # isort:skip # noqa: E402
-from sphinx.domains.std import warn_missing_reference  # isort:skip # noqa: E402
 
 DOC_PKG_ROOT = os.path.join(DOC_PRJ_ROOT, __meta__.__name__)
 
@@ -58,7 +58,7 @@
     "sphinx.ext.autodoc",   # document code docstrings
     "sphinx.ext.autosectionlabel",  # help make cross-references to title/sections
     "cloud_sptheme.ext.autodoc_sections",   # allow sections in docstrings code
-    "sphinx.ext.githubpages",   # for publishing the doc to github pages
+    "sphinx.ext.githubpages",   # for publishing the doc to GitHub pages
     "sphinx.ext.todo",          # support directives
     "sphinx.ext.viewcode",      # add links to highlighted source code
     "sphinx.ext.intersphinx",   # add links to other projects documentation
@@ -104,7 +104,7 @@ def doc_redirect_include(file_path):
 if not os.path.isdir(DOC_BLD_ROOT):
     os.makedirs(DOC_BLD_ROOT)
 with open(api_spec_file, "w") as f:
-    json.dump(api_spec_json, f)
+    json.dump(api_spec_json, f, use_decimal=True)
 
 redoc = [{
     "name": __meta__.__title__,
@@ -418,6 +418,8 @@ def doc_redirect_include(file_path):
     r"./config.*",
     r"./docs.*",
     r"docs/source/.*",
+    r"weaver/schemas/quotation/quote-estimator.yaml",
+    r"weaver/schemas/quotation/quote-estimation-result.yaml",
     # inter-reference between document page and section headers
     # when link is itself a documentation reference, they are not resolved correctly, but this works with text replaces
     r":ref:`.*`",