diff --git a/docs/accounts.md b/docs/accounts.md index c343865e..9b2a8579 100644 --- a/docs/accounts.md +++ b/docs/accounts.md @@ -27,4 +27,92 @@ harness: - administrator ``` -See the [Gogatekeeper official documentation](https://github.com/gogatekeeper/gatekeeper/blob/master/docs/user-guide.md) for more. \ No newline at end of file +See the [Gogatekeeper official documentation](https://github.com/gogatekeeper/gatekeeper/blob/master/docs/user-guide.md) for more. + + +## Backend development +### Secure and enpoint with the Gatekeeper + +The simplest solution to give authorized access to some api endpoint is to configure the gatekeeper (see above). + +```yaml +harness: + ... + secured: true + uri_role_mapping: + - uri: /* + methods: + - POST + - PUT + - DELETE + roles: + - administrator +``` + +### Secure an enpoint with OpenAPI + +In every api endpoint that you want to secure, add the bearerAuth security as in the example: + +```yaml +paths: + /valid: + get: + summary: Check if the token is valid. Get a token by logging into the base url + security: + - bearerAuth: [] +``` + +In the components section, add the following +```yaml +components: + securitySchemes: + bearerAuth: + type: http + scheme: bearer + bearerFormat: JWT + x-bearerInfoFunc: cloudharness.auth.decode_token +``` + +See the examples: + +* [Secured with openapi](/applications/samples/backend/samples/controllers/auth_controller.py) (actually a normal api, the openapi configuration does everything) +* [Openapi configuration: add bearerAuth](/applications/samples/api/samples.yaml#L20) +* [Openapi configuration: configure bearer handler](/applications/samples/api/samples.yaml#L141) + + +### Use the AuthClient + +The Cloudharness AuthClient is a handy wrapper for the Keycloak REST API. +This wrapper class can be used to retrieve the current user of the http(s) request +or to retrieve the Keycloak groups with all users etc. + +All functions of the AuthClient class are wrapped by the `with_refreshtoken` decorator +to auto refresh the token in case the token is expired. There is no need to manually +refresh the token. + +`AuthClient` uses the `admin_api` account to log in into the Keycloak admin REST api +the password is stored in the `accounts` secret and is retrieve using the Cloudharness +`get_secret` function (imported from `cloudharness.utils.secrets`) + +For more information about the usage of the `AuthClient` see the Python doc strings + + +**Important note:** + +it is mandatory that the application deployment has a hard dependency to the +`accounts` application. This dependency will mount the accounts secret to the pods. + +
+ +Examples: +```python +from cloudharness.auth.keycloak import AuthClient +from cloudharness.models import User + +ac = AuthClient() + +current_user: User = ac.get_current_user() +email = current_user.email + +all_groups = ac.get_groups(with_members=True) +``` diff --git a/docs/applications/development/backend-development.md b/docs/applications/development/backend-development.md index 75912acd..34d044bb 100644 --- a/docs/applications/development/backend-development.md +++ b/docs/applications/development/backend-development.md @@ -1,115 +1,105 @@ # Develop in the backend with CloudHarness -## Base images and libraries -TODO - ## Create a default Backend with Flask and Openapi -TODO -## Use the CloudHarness Python library +Although there is no restriction on the technology used to build +your application, starting from a semi auto-generated Python/Flask application is the +recommended way to create a backend microservice within a +CloudHarness based solution. -### Get applications references and configurations -TODO +To create the initial scaffolding, run: -### Check authentication and Authorization +``` +harness-application app-name +``` -#### Secure a RESTful API +The development then can go through the following steps: +1. Edit the openapi file at `applications/app-name/api/openapi.yaml`. Can edit the yaml file directly or use more friendly interfaces like Apicurio Studio or SwaggerHub. It is recommended to use a different tag for every resource type: one controller is generated for each tag. +1. Regenerate the application code stubs with `harness-generate . -i` +1. *(optional)* edit the setup.py and requirements.txt according to your library requirements +1. Implement your logic inside `backend/app_name/controllers`. It is recommended to implement the actual application logic on custom files implementing the business logic and the service logic. +1. *(optional)* add **/controllers to the .openapi-generator-ignore so that the controllers won't be overwritten by the next generation +1. *(optional)* customize the Dockerfile +1. *(optional)* add other custom endpoints -Note: this document is not a tutorial on how to secure an application on Cloud Harness. The aim is giving the CH developer an insight on how it's implemented. To see an example of a secured api, see samples application: +The application entry point is `backend/__main__.py`: it can be run as +a simple Python script or module to debug locally. -* [Secured backend api](/applications/samples/backend/samples/controllers/auth_controller.py) (actually a normal api, the openapi configuraition does everything) -* [Openapi configuration: add bearerAuth](/applications/samples/api/samples.yaml#L20) -* [Openapi configuration: configure bearer handler](/applications/samples/api/samples.yaml#L141) +### Base libraries and images -Following some insights to have a web application dashboard secured with username and password and then interact with a RESTful API using JWT +The simplest way to use the shared CloudHarness functionality in an +application is to inherit your application Docker image from one +of the base images. +The base images include preinstalled the CloufHarness common libraries. -Using OpenAPI to generate the code, we introduce the following security +Since the images are built together with the rest of the system, +we use arguments for the reference, like: -```yaml -security: - - bearerAuth: [] -components: - securitySchemes: - bearerAuth: - type: http - scheme: bearer - bearerFormat: JWT +```dockerfile +ARG $CLOUDHARNESS_BASE +FROM $CLOUDHARNESS_BASE +... ``` -Then we look for `security_controller.py` file: +For the image dependency to be recognized by the build scripts, +the dependencty must be declared in the `values.yaml` file of your application, e.g.: -```python -def info_from_bearerAuth(token): - SCHEMA = 'https://' - AUTH_DOMAIN = os.environ.get('AUTH_DOMAIN') - AUTH_REALM = os.environ.get('AUTH_REALM') - BASE_PATH = f"//{os.path.join(AUTH_DOMAIN, 'auth/realms', AUTH_REALM)}" - AUTH_PUBLIC_KEY_URL = urljoin(SCHEMA, BASE_PATH) - - # We extract KC public key to validate the JWT we receive - KEY = json.loads(requests.get(AUTH_PUBLIC_KEY_URL, verify=False).text)['public_key'] - - # Create the key - KEY = f"-----BEGIN PUBLIC KEY-----\n{KEY}\n-----END PUBLIC KEY-----" - - try: - # Here we decode the JWT - decoded = jwt.decode(token, KEY, audience='account', algorithms='RS256') - except: - current_app.logger.debug(f"Error validating user: {sys.exc_info()}") - return None - - # Here we proceed to do all the validation we need to check if we grant access to the RESTful API - valid = 'offline_access' in decoded['realm_access']['roles'] - current_app.logger.debug(valid) - return {'uid': 'user_id' } +```yaml +harness: + dependencies: + build: + - cloudharness-base ``` -#### Using the AuthClient +Every image defined as a base image or a common image can be used as a +build dependency. -The Cloudharness AuthClient is a handy wrapper for the Keycloak REST API. -This wrapper class can be used to retrieve the current user of the http(s) request -or to retrieve the Keycloak groups with all users etc. +For more details about how to define your custom image and the available images, see [here](../../base-common-images.md) -All functions of the AuthClient class are wrapped by the `with_refreshtoken` decorator -to auto refresh the token in case the token is expired. There is no need to manually -refresh the token. +## Use the CloudHarness runtime Python library -`AuthClient` uses the `admin_api` account to log in into the Keycloak admin REST api -the password is stored in the `accounts` secret and is retrieve using the Cloudharness -`get_secret` function (imported from `cloudharness.utils.secrets`) +The CloudHarness runtime library shares some common functionality that +helps the backend development in Python. +The runtime library depends on the `cloudharness_models` library, which builds the common +ground to understand and use the relevant data types. -
+The main functionality provided is: +- Access to the solution configuration and secrets +- Access and manipulate users, authentication and authorization +- Create and listen to orchestration events +- Create and monitor workflow operations -For more information about the usage of the `AuthClient` see the Python doc strings - -
- -**Important note:** +### Get applications references and configurations -it is mandatory that the application deployment has a hard dependency to the -`accounts` application. This dependency will mount the accounts secret to the pods. +The applications configuration api gives access to a proxy object +containing all the data from the values.yaml file at runtime. -
+The object returned is of type `cloudharness.applications.ApplicationConfiguration`, +a subtype of the wrapper [ApplicationConfig](../../model/ApplicationConfig.md). -Examples: ```python -from cloudharness.auth.keycloak import AuthClient - -ac = AuthClient() +from cloudharness import applications -def example1(): - current_user = ac.get_current_user() +uut: applications.ApplicationConfiguration = applications.get_configuration('app1') -def exampl2(): - all_groups = ac.get_groups(with_members=True) +uut.is_auto_service() # has a service? +uut.is_auto_deployment() # has a deployment? +uut.is_sentry_enabled() # is sentry enabled? +uut.image_name # get the image name +uut.get_public_address() # get the public (external) address, as configured in Ingress +uut.get_service_address() # internal address to make calls to this application ``` +### Check authentication and Authorization + +See [accounts specific documentation](../../accounts.md#Backend-development). ### Run workflows -See [the workflows api](./workflows-api.md) dedicated document. +See the [workflows api](./workflows-api.md) dedicated document. + +### Events and orchestration ## Debug inside the cluster diff --git a/docs/events.md b/docs/events.md index 086b7a06..28f16fbf 100644 --- a/docs/events.md +++ b/docs/events.md @@ -24,3 +24,58 @@ Also add to your hosts file 127.0.0.1 kafka-0.broker.mnp.svc.cluster.local bootstrap.mnp.svc.cluster.local ``` +## Backend library + +### Data Create, Delete and Change Events + +Data change events are a special kind of event used to notify the system that some +data is created/changed/deleted. + +The best way to send a CDC Event is by a decorator in a service function: + +```python +from cloudharness.events.decorators import send_event + +@send_event(message_type="my_object", operation="create") +def create_myobject(self, body): + created_object = ... # database logic + return created_object +``` + + +The above event can be consumed as: + +```python +from cloudharness.events.client import EventClient +from cloudharness.models import CDCEvent + +def handler(app, event_client, message: CDCEvent): + ... + +event_client = EventClient("my_object") +event_client.async_consume(handler=handler, group_id="ch-notifications") +``` + +For a concrete code example of the CDC events, see the [notification application](/applications/notifications/server/notifications/controllers/notifications_controller.py) + +### Consume and handle a generic event + +```python +from cloudharness.events.client import EventClient + +def my_callback(event_client, message): + ... + +client = EventClient("my-topic") +client.async_consume(group_id="my-group", handler=my_callback) +``` + + +### Produce a generic event + +```python +from cloudharness.workflows.utils import notify_queue + +my_message = {"a": "b"} +notify_queue("my-topic", my_message) +``` \ No newline at end of file diff --git a/docs/model/ApplicationHarnessConfig.md b/docs/model/ApplicationHarnessConfig.md index 173dbfa0..6d8fc882 100644 --- a/docs/model/ApplicationHarnessConfig.md +++ b/docs/model/ApplicationHarnessConfig.md @@ -4,21 +4,24 @@ ## Properties Name | Type | Description | Notes ------------ | ------------- | ------------- | ------------- -**deployment** | **bool, date, datetime, dict, float, int, list, str, none_type** | Defines reference deployment parameters. Values maps to k8s spec | [optional] -**service** | **bool, date, datetime, dict, float, int, list, str, none_type** | Defines automatic service parameters. | [optional] +**deployment** | [**DeploymentAutoArtifactConfig**](DeploymentAutoArtifactConfig.md) | | [optional] +**service** | [**ServiceAutoArtifactConfig**](ServiceAutoArtifactConfig.md) | | [optional] **subdomain** | **str** | If specified, an ingress will be created at [subdomain].[.Values.domain] | [optional] **aliases** | **[str]** | If specified, an ingress will be created at [alias].[.Values.domain] for each alias | [optional] **domain** | **str** | If specified, an ingress will be created at [domain] | [optional] -**dependencies** | **bool, date, datetime, dict, float, int, list, str, none_type** | Application dependencies are used to define what is required in the deployment when --include (-i) is used. Specify application names in the list. | [optional] +**dependencies** | [**ApplicationDependenciesConfig**](ApplicationDependenciesConfig.md) | | [optional] **secured** | **bool** | When true, the application is shielded with a getekeeper | [optional] **uri_role_mapping** | [**[UriRoleMappingConfig]**](UriRoleMappingConfig.md) | Map uri/roles to secure with the Gatekeeper (if `secured: true`) | [optional] -**secrets** | **bool, date, datetime, dict, float, int, list, str, none_type** | Define secrets will be mounted in the deployment Define as ```yaml secrets: secret_name: 'value' ``` Values if left empty are randomly generated | [optional] +**secrets** | [**SimpleMap**](SimpleMap.md) | | [optional] **use_services** | **[str]** | Specify which services this application uses in the frontend to create proxy ingresses. e.g. ``` - name: samples ``` | [optional] -**database** | **bool, date, datetime, dict, float, int, list, str, none_type** | | [optional] +**database** | [**DatabaseDeploymentConfig**](DatabaseDeploymentConfig.md) | | [optional] **resources** | [**[FileResourcesConfig]**](FileResourcesConfig.md) | Application file resources. Maps from deploy/resources folder and mounts as configmaps | [optional] -**readiness_probe** | **bool, date, datetime, dict, float, int, list, str, none_type** | | [optional] +**readiness_probe** | [**ApplicationProbe**](ApplicationProbe.md) | | [optional] **startup_probe** | [**ApplicationProbe**](ApplicationProbe.md) | | [optional] **liveness_probe** | [**ApplicationProbe**](ApplicationProbe.md) | | [optional] +**source_root** | [**Filename**](Filename.md) | | [optional] +**name** | **str** | | [optional] +**jupyterhub** | [**JupyterHubConfig**](JupyterHubConfig.md) | | [optional] **any string name** | **bool, date, datetime, dict, float, int, list, str, none_type** | any string name can be used but the value must be the correct type | [optional] [[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md) diff --git a/docs/model/OrchestrationEvent.md b/docs/model/CDCEvent.md similarity index 82% rename from docs/model/OrchestrationEvent.md rename to docs/model/CDCEvent.md index b4819f58..ef9c1a49 100644 --- a/docs/model/OrchestrationEvent.md +++ b/docs/model/CDCEvent.md @@ -1,6 +1,6 @@ -# OrchestrationEvent +# CDCEvent -A message sent to the orchestration queue. Applications can listen to these events to react to CRUD events happening on other applications. +A message sent to the orchestration queue. Applications can listen to these events to react to data change events happening on other applications. ## Properties Name | Type | Description | Notes @@ -8,7 +8,7 @@ Name | Type | Description | Notes **operation** | **str** | the operation on the object e.g. create / update / delete | **uid** | **str** | the unique identifier attribute of the object | **message_type** | **str** | the type of the message (relates to the object type) e.g. jobs | -**meta** | [**OrchestrationEventMeta**](OrchestrationEventMeta.md) | | +**meta** | [**CDCEventMeta**](CDCEventMeta.md) | | **resource** | [**FreeObject**](FreeObject.md) | | [optional] **any string name** | **bool, date, datetime, dict, float, int, list, str, none_type** | any string name can be used but the value must be the correct type | [optional] diff --git a/docs/model/OrchestrationEventMeta.md b/docs/model/CDCEventMeta.md similarity index 97% rename from docs/model/OrchestrationEventMeta.md rename to docs/model/CDCEventMeta.md index 8b7436d8..a90f4b95 100644 --- a/docs/model/OrchestrationEventMeta.md +++ b/docs/model/CDCEventMeta.md @@ -1,4 +1,4 @@ -# OrchestrationEventMeta +# CDCEventMeta ## Properties diff --git a/docs/model/HarnessMainConfig.md b/docs/model/HarnessMainConfig.md index 608ff2b4..b563b5c5 100644 --- a/docs/model/HarnessMainConfig.md +++ b/docs/model/HarnessMainConfig.md @@ -14,7 +14,9 @@ Name | Type | Description | Notes **tag** | **str** | Docker tag used to push/pull the built images. | [optional] **env** | [**[NameValue]**](NameValue.md) | Environmental variables added to all pods | [optional] **privenv** | [**NameValue**](NameValue.md) | | [optional] -**backup** | **bool, date, datetime, dict, float, int, list, str, none_type** | | [optional] +**backup** | [**BackupConfig**](BackupConfig.md) | | [optional] +**name** | **str** | Base name | [optional] +**task_images** | [**SimpleMap**](SimpleMap.md) | | [optional] **any string name** | **bool, date, datetime, dict, float, int, list, str, none_type** | any string name can be used but the value must be the correct type | [optional] [[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md) diff --git a/docs/model/JupyterHubConfig.md b/docs/model/JupyterHubConfig.md new file mode 100644 index 00000000..7c7f2361 --- /dev/null +++ b/docs/model/JupyterHubConfig.md @@ -0,0 +1,15 @@ +# JupyterHubConfig + + +## Properties +Name | Type | Description | Notes +------------ | ------------- | ------------- | ------------- +**args** | **[str]** | arguments passed to the container | [optional] +**extra_config** | [**SimpleMap**](SimpleMap.md) | | [optional] +**spawner_extra_config** | [**FreeObject**](FreeObject.md) | | [optional] +**application_hook** | **bool, date, datetime, dict, float, int, list, str, none_type** | change the hook function (advanced) Specify the Python name of the function (full module path, the module must be installed in the Docker image) | [optional] +**any string name** | **bool, date, datetime, dict, float, int, list, str, none_type** | any string name can be used but the value must be the correct type | [optional] + +[[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md) + + diff --git a/docs/model/URL.md b/docs/model/PathSpecifier.md similarity index 51% rename from docs/model/URL.md rename to docs/model/PathSpecifier.md index 85c7a437..baa172d4 100644 --- a/docs/model/URL.md +++ b/docs/model/PathSpecifier.md @@ -1,8 +1,11 @@ -# URL +# PathSpecifier -Type | Description | Notes -------------- | ------------- | ------------- -**str** | | + +## Properties +Name | Type | Description | Notes +------------ | ------------- | ------------- | ------------- +**value** | **str** | | [[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md) + diff --git a/docs/model/UriRoleMappingConfig.md b/docs/model/UriRoleMappingConfig.md index f3e3874d..4ea3fb85 100644 --- a/docs/model/UriRoleMappingConfig.md +++ b/docs/model/UriRoleMappingConfig.md @@ -5,7 +5,7 @@ Defines the application Gatekeeper configuration, if enabled (i.e. `secured: tru ## Properties Name | Type | Description | Notes ------------ | ------------- | ------------- | ------------- -**uri** | [**Filename**](Filename.md) | | +**uri** | [**PathSpecifier**](PathSpecifier.md) | | **roles** | **[str]** | Roles allowed to access the present uri | **any string name** | **bool, date, datetime, dict, float, int, list, str, none_type** | any string name can be used but the value must be the correct type | [optional] diff --git a/docs/model/UserRole.md b/docs/model/UserRole.md new file mode 100644 index 00000000..412ab624 --- /dev/null +++ b/docs/model/UserRole.md @@ -0,0 +1,18 @@ +# UserRole + + +## Properties +Name | Type | Description | Notes +------------ | ------------- | ------------- | ------------- +**attributes** | **{str: (bool, date, datetime, dict, float, int, list, str, none_type)}** | | [optional] +**client_role** | **bool** | | [optional] +**composite** | **bool** | | [optional] +**container_id** | **str** | | [optional] +**description** | **str** | | [optional] +**id** | **str** | | [optional] +**name** | **str** | | [optional] +**any string name** | **bool, date, datetime, dict, float, int, list, str, none_type** | any string name can be used but the value must be the correct type | [optional] + +[[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md) + +