#451 update documentation

docs/accounts.md

@@ -27,4 +27,92 @@ harness:
     - administrator
 ```
 
-See the [Gogatekeeper official documentation](https://github.com/gogatekeeper/gatekeeper/blob/master/docs/user-guide.md) for more.
+See the [Gogatekeeper official documentation](https://github.com/gogatekeeper/gatekeeper/blob/master/docs/user-guide.md) for more.
+
+
+## Backend development
+### Secure and enpoint with the Gatekeeper
+
+The simplest solution to give authorized access to some api endpoint is to configure the gatekeeper (see above).
+
+```yaml
+harness:
+  ...
+  secured: true
+  uri_role_mapping:
+  - uri: /*
+    methods:
+    - POST
+    - PUT
+    - DELETE
+    roles:
+    - administrator
+```
+
+### Secure an enpoint with OpenAPI
+
+In every api endpoint that you want to secure, add the bearerAuth security as in the example:
+
+```yaml
+paths:
+  /valid:
+    get:
+      summary: Check if the token is valid. Get a token by logging into the base url
+      security:
+        - bearerAuth: []
+```
+
+In the components section, add the following
+```yaml
+components:
+  securitySchemes:
+    bearerAuth:
+      type: http
+      scheme: bearer
+      bearerFormat: JWT
+      x-bearerInfoFunc: cloudharness.auth.decode_token
+```
+
+See the examples:
+
+* [Secured with openapi](/applications/samples/backend/samples/controllers/auth_controller.py) (actually a normal api, the openapi configuration does everything)
+* [Openapi configuration: add bearerAuth](/applications/samples/api/samples.yaml#L20)  
+* [Openapi configuration: configure bearer handler](/applications/samples/api/samples.yaml#L141)  
+
+
+### Use the AuthClient
+
+The Cloudharness AuthClient is a handy wrapper for the Keycloak REST API.
+This wrapper class can be used to retrieve the current user of the http(s) request
+or to retrieve the Keycloak groups with all users etc.
+
+All functions of the AuthClient class are wrapped by the `with_refreshtoken` decorator
+to auto refresh the token in case the token is expired. There is no need to manually
+refresh the token.
+
+`AuthClient` uses the `admin_api` account to log in into the Keycloak admin REST api
+the password is stored in the `accounts` secret and is retrieve using the Cloudharness
+`get_secret` function (imported from `cloudharness.utils.secrets`)
+
+For more information about the usage of the `AuthClient` see the Python doc strings
+
+
+**Important note:**
+
+it is mandatory that the application deployment has a hard dependency to the 
+`accounts` application. This dependency will mount the accounts secret to the pods.
+
+<br/>
+
+Examples:
+```python
+from cloudharness.auth.keycloak import AuthClient
+from cloudharness.models import User
+
+ac = AuthClient()
+
+current_user: User = ac.get_current_user()
+email = current_user.email
+
+all_groups = ac.get_groups(with_members=True)
+```

docs/applications/development/backend-development.md

@@ -1,115 +1,105 @@
 # Develop in the backend with CloudHarness
 
-## Base images and libraries
-TODO 
-
 ## Create a default Backend with Flask and Openapi
-TODO
 
-## Use the CloudHarness Python library
+Although there is no restriction on the technology used to build
+your application, starting from a semi auto-generated Python/Flask application is the
+recommended way to create a backend microservice within a 
+CloudHarness based solution.
 
-### Get applications references and configurations
-TODO
+To create the initial scaffolding, run:
 
-### Check authentication and Authorization
+```
+harness-application app-name
+```
 
-#### Secure a RESTful API
+The development then can go through the following steps:
+1. Edit the openapi file at `applications/app-name/api/openapi.yaml`. Can edit the yaml file directly or use more friendly interfaces like Apicurio Studio or SwaggerHub. It is recommended to use a different tag for every resource type: one controller is generated for each tag.
+1. Regenerate the application code stubs with `harness-generate . -i`
+1. *(optional)* edit the setup.py and requirements.txt according to your library requirements
+1. Implement your logic inside `backend/app_name/controllers`. It is recommended to implement the actual application logic on custom files implementing the business logic and the service logic.
+1. *(optional)*  add **/controllers to the .openapi-generator-ignore so that the controllers won't be overwritten by the next generation
+1. *(optional)* customize the Dockerfile
+1. *(optional)* add other custom endpoints
 
-Note: this document is not a tutorial on how to secure an application on Cloud Harness. The aim is giving the CH developer an insight on how it's implemented. To see an example of a secured api, see samples application:
+The application entry point is `backend/__main__.py`: it can be run as
+a simple Python script or module to debug locally.
 
-* [Secured backend api](/applications/samples/backend/samples/controllers/auth_controller.py) (actually a normal api, the openapi configuraition does everything)
-* [Openapi configuration: add bearerAuth](/applications/samples/api/samples.yaml#L20)  
-* [Openapi configuration: configure bearer handler](/applications/samples/api/samples.yaml#L141)  
 
+### Base libraries and images
 
-Following some insights to have a web application dashboard secured with username and password and then interact with a RESTful API using JWT 
+The simplest way to use the shared CloudHarness functionality in an 
+application is to inherit your application Docker image from one
+of the base images.
+The base images include preinstalled the CloufHarness common libraries.
 
-Using OpenAPI to generate the code, we introduce the following security
+Since the images are built together with the rest of the system,
+we use arguments for the reference, like:
 
-```yaml
-security:
-  - bearerAuth: []
-components:
-  securitySchemes: 
-    bearerAuth:
-      type: http
-      scheme: bearer
-      bearerFormat: JWT
+```dockerfile
+ARG $CLOUDHARNESS_BASE
+FROM $CLOUDHARNESS_BASE
+...
 ```
 
-Then we look for `security_controller.py` file:
+For the image dependency to be recognized by the build scripts,
+the dependencty must be declared in the `values.yaml` file of your application, e.g.:
 
-```python
-def info_from_bearerAuth(token):
-    SCHEMA = 'https://'
-    AUTH_DOMAIN = os.environ.get('AUTH_DOMAIN')
-    AUTH_REALM = os.environ.get('AUTH_REALM')
-    BASE_PATH = f"//{os.path.join(AUTH_DOMAIN, 'auth/realms', AUTH_REALM)}"
-    AUTH_PUBLIC_KEY_URL = urljoin(SCHEMA, BASE_PATH)
-
-    # We extract KC public key to validate the JWT we receive 
-    KEY = json.loads(requests.get(AUTH_PUBLIC_KEY_URL, verify=False).text)['public_key'] 
-    
-    # Create the key
-    KEY = f"-----BEGIN PUBLIC KEY-----\n{KEY}\n-----END PUBLIC KEY-----"
-    
-    try:
-        # Here we decode the JWT
-        decoded = jwt.decode(token, KEY, audience='account', algorithms='RS256')
-    except:
-        current_app.logger.debug(f"Error validating user: {sys.exc_info()}")
-        return None
-    
-    # Here we proceed to do all the validation we need to check if we grant access to the RESTful API 
-    valid = 'offline_access' in decoded['realm_access']['roles']
-    current_app.logger.debug(valid)
-    return {'uid': 'user_id' }
+```yaml
+harness:
+  dependencies:
+    build:
+    - cloudharness-base
 ```
 
-#### Using the AuthClient
+Every image defined as a base image or a common image can be used as a
+build dependency.
 
-The Cloudharness AuthClient is a handy wrapper for the Keycloak REST API.
-This wrapper class can be used to retrieve the current user of the http(s) request
-or to retrieve the Keycloak groups with all users etc.
+For more details about how to define your custom image and the available images, see [here](../../base-common-images.md)
 
-All functions of the AuthClient class are wrapped by the `with_refreshtoken` decorator
-to auto refresh the token in case the token is expired. There is no need to manually
-refresh the token.
+## Use the CloudHarness runtime Python library
 
-`AuthClient` uses the `admin_api` account to log in into the Keycloak admin REST api
-the password is stored in the `accounts` secret and is retrieve using the Cloudharness
-`get_secret` function (imported from `cloudharness.utils.secrets`)
+The CloudHarness runtime library shares some common functionality that
+helps the backend development in Python.
+The runtime library depends on the `cloudharness_models` library, which builds the common 
+ground to understand and use the relevant data types.
 
-<br/>
+The main functionality provided is:
+- Access to the solution configuration and secrets
+- Access and manipulate users, authentication and authorization
+- Create and listen to orchestration events
+- Create and monitor workflow operations
 
-For more information about the usage of the `AuthClient` see the Python doc strings
-
-<br/>
-
-**Important note:**
+### Get applications references and configurations
 
-it is mandatory that the application deployment has a hard dependency to the 
-`accounts` application. This dependency will mount the accounts secret to the pods.
+The applications configuration api gives access to a proxy object
+containing all the data from the values.yaml file at runtime.
 
-<br/>
+The object returned is of type `cloudharness.applications.ApplicationConfiguration`,
+a subtype of the wrapper [ApplicationConfig](../../model/ApplicationConfig.md).
 
-Examples:
 ```python
-from cloudharness.auth.keycloak import AuthClient
-
-ac = AuthClient()
+from cloudharness import applications
 
-def example1():
-  current_user = ac.get_current_user()
+uut: applications.ApplicationConfiguration = applications.get_configuration('app1')
 
-def exampl2():
-  all_groups = ac.get_groups(with_members=True)
+uut.is_auto_service() # has a service?
+uut.is_auto_deployment() # has a deployment?
+uut.is_sentry_enabled() # is sentry enabled?
+uut.image_name # get the image name
+uut.get_public_address() # get the public (external) address, as configured in Ingress
+uut.get_service_address() # internal address to make calls to this application
 ```
 
+### Check authentication and Authorization
+
+See [accounts specific documentation](../../accounts.md#Backend-development).
 
 ### Run workflows
 
-See [the workflows api](./workflows-api.md) dedicated document.
+See the [workflows api](./workflows-api.md) dedicated document.
+
+### Events and orchestration 
 
 ## Debug inside the cluster
 

docs/events.md

@@ -24,3 +24,58 @@ Also add to your hosts file
 127.0.0.1      kafka-0.broker.mnp.svc.cluster.local bootstrap.mnp.svc.cluster.local
 ```
 
+## Backend library
+
+### Data Create, Delete and Change Events
+
+Data change events are a special kind of event used to notify the system that some
+data is created/changed/deleted.
+
+The best way to send a CDC Event is by a decorator in a service function:
+
+```python
+from cloudharness.events.decorators import send_event
+
+@send_event(message_type="my_object", operation="create")
+def create_myobject(self, body):
+    created_object = ... # database logic
+    return created_object
+```
+
+
+The above event can be consumed as:
+
+```python
+from cloudharness.events.client import EventClient
+from cloudharness.models import CDCEvent
+
+def handler(app, event_client, message: CDCEvent):
+    ...
+
+event_client = EventClient("my_object")
+event_client.async_consume(handler=handler, group_id="ch-notifications")
+```
+
+For a concrete code example of the CDC events, see the [notification application](/applications/notifications/server/notifications/controllers/notifications_controller.py)
+
+### Consume and handle a generic event
+
+```python
+from cloudharness.events.client import EventClient
+
+def my_callback(event_client, message):
+    ...
+
+client = EventClient("my-topic")
+client.async_consume(group_id="my-group", handler=my_callback)
+```
+
+
+### Produce a generic event
+
+```python
+from cloudharness.workflows.utils import notify_queue
+
+my_message = {"a": "b"}
+notify_queue("my-topic", my_message)
+```

docs/model/ApplicationHarnessConfig.md

@@ -4,21 +4,24 @@
 ## Properties
 Name | Type | Description | Notes
 ------------ | ------------- | ------------- | -------------
-**deployment** | **bool, date, datetime, dict, float, int, list, str, none_type** | Defines reference deployment parameters. Values maps to k8s spec | [optional] 
-**service** | **bool, date, datetime, dict, float, int, list, str, none_type** | Defines automatic service parameters. | [optional] 
+**deployment** | [**DeploymentAutoArtifactConfig**](DeploymentAutoArtifactConfig.md) |  | [optional] 
+**service** | [**ServiceAutoArtifactConfig**](ServiceAutoArtifactConfig.md) |  | [optional] 
 **subdomain** | **str** | If specified, an ingress will be created at [subdomain].[.Values.domain] | [optional] 
 **aliases** | **[str]** | If specified, an ingress will be created at [alias].[.Values.domain] for each alias | [optional] 
 **domain** | **str** | If specified, an ingress will be created at [domain] | [optional] 
-**dependencies** | **bool, date, datetime, dict, float, int, list, str, none_type** | Application dependencies are used to define what is required in the deployment when --include (-i) is used. Specify application names in the list. | [optional] 
+**dependencies** | [**ApplicationDependenciesConfig**](ApplicationDependenciesConfig.md) |  | [optional] 
 **secured** | **bool** | When true, the application is shielded with a getekeeper | [optional] 
 **uri_role_mapping** | [**[UriRoleMappingConfig]**](UriRoleMappingConfig.md) | Map uri/roles to secure with the Gatekeeper (if `secured: true`) | [optional] 
-**secrets** | **bool, date, datetime, dict, float, int, list, str, none_type** | Define secrets will be mounted in the deployment  Define as  ```yaml secrets:     secret_name: 'value'  ```  Values if left empty are randomly generated | [optional] 
+**secrets** | [**SimpleMap**](SimpleMap.md) |  | [optional] 
 **use_services** | **[str]** | Specify which services this application uses in the frontend to create proxy ingresses. e.g.  ``` - name: samples ``` | [optional] 
-**database** | **bool, date, datetime, dict, float, int, list, str, none_type** |  | [optional] 
+**database** | [**DatabaseDeploymentConfig**](DatabaseDeploymentConfig.md) |  | [optional] 
 **resources** | [**[FileResourcesConfig]**](FileResourcesConfig.md) | Application file resources. Maps from deploy/resources folder and mounts as configmaps | [optional] 
-**readiness_probe** | **bool, date, datetime, dict, float, int, list, str, none_type** |  | [optional] 
+**readiness_probe** | [**ApplicationProbe**](ApplicationProbe.md) |  | [optional] 
 **startup_probe** | [**ApplicationProbe**](ApplicationProbe.md) |  | [optional] 
 **liveness_probe** | [**ApplicationProbe**](ApplicationProbe.md) |  | [optional] 
+**source_root** | [**Filename**](Filename.md) |  | [optional] 
+**name** | **str** |  | [optional] 
+**jupyterhub** | [**JupyterHubConfig**](JupyterHubConfig.md) |  | [optional] 
 **any string name** | **bool, date, datetime, dict, float, int, list, str, none_type** | any string name can be used but the value must be the correct type | [optional]
 
 [[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md)

docs/model/OrchestrationEvent.md → docs/model/CDCEvent.md

@@ -1,14 +1,14 @@
-# OrchestrationEvent
+# CDCEvent
 
-A message sent to the orchestration queue. Applications can listen to these events to react to CRUD events happening on other applications.
+A message sent to the orchestration queue. Applications can listen to these events to react to data change events happening on other applications.
 
 ## Properties
 Name | Type | Description | Notes
 ------------ | ------------- | ------------- | -------------
 **operation** | **str** | the operation on the object e.g. create / update / delete | 
 **uid** | **str** | the unique identifier attribute of the object | 
 **message_type** | **str** | the type of the message (relates to the object type) e.g. jobs | 
-**meta** | [**OrchestrationEventMeta**](OrchestrationEventMeta.md) |  | 
+**meta** | [**CDCEventMeta**](CDCEventMeta.md) |  | 
 **resource** | [**FreeObject**](FreeObject.md) |  | [optional] 
 **any string name** | **bool, date, datetime, dict, float, int, list, str, none_type** | any string name can be used but the value must be the correct type | [optional]
 

docs/model/OrchestrationEventMeta.md → docs/model/CDCEventMeta.md

@@ -1,4 +1,4 @@
-# OrchestrationEventMeta
+# CDCEventMeta
 
 
 ## Properties