Quickstart Deployment Mode

[Cccccrystall]

Hi, everyone!

At present, there are no Pipelines or dashboards in the user's initial page after StreamPipes is installed. Introducing Quickstart deployment mode could initiate all elements of StreamPipes, such as the backend, message broker, user interface, and so on. This mode is designed to produce a greater number of StreamPipes assets straightaway upon startup. With the introduction of this new implementation style, it is anticipated that each StreamPipes module will come with pre-existing assets, including data sources, pipelines, dashboard. The Quickstart mode could help users could easily explore StreamPipes’features.

My Implementations of the Quickstart Deployment are as follows:

• First, I'll acquire the assets locally via Asset Management. These assets include default pipelines, dashboards, functions, among others. We could create asset instance via the UI interface, download it locally, and obtain the json files for each element. At present, we can manually upload the asset file to obtain the configured asset. The next step entails automating this process.

• Second, I will encapsulate the preset asset configuration into a new docker image. This docker image will also include a help function and trigger for interpreting and loading the asset configuration. This new docker image will be added based on the original docker. Prefined assets can be set up as a container during the user installation.

However, before we proceed with this deployment, there are a few more details I'd like to discuss with you and hope to get your invaluable insights and suggestions.

• Where should we put the codes of loading predefined resources? Docker or in the codebase?

• Do we have the related code for loading the resources? If we have, it is implemented in which part of the codebase?

• For each asset plan, how to choose and match resources as preset resources? Since there are various resources for different targets. Also, is it possible to incorporate a drop-down box for "Example assets", thereby allowing users to choose their preferred assets from the library based on their needs?

[tenthe]

Hi @Cccccrystall,
thank you very much for the detailed description. I like your solution very much.
I have only one comment. Do we need a docker container for this function? Maybe we can find a solution where we integrate it into the code base and trigger it from the GUI.
Because then it would be a bit easier for the user, otherwise a user must edit the docker-compose.yml when getting started.

What resources would you expect to see in this quick start mode?
Cheers,
Philipp

[Cccccrystall]

Hi, @tenthe ! Thanks for your suggestion! It helps a lot! And sorry for replying late.
And hi, Tim! @bossenti I am currently trying to deploy autoload resources as scripts into docker, maybe we could discuss how to improve it when I have a preliminary version.

[bossenti]

Sounds great 😊

[Cccccrystall]

Hi, Tim @bossenti !
I encountered some problems when I tried to deploy docker, and I would like to discuss them with you. Based on the scenario we discussed earlier, I should upload the default asset and wrap the process into docker so that the user can see the default asset after installing it with docker. The problem with this solution is that the jwt token obtained by the user login cannot be obtained during the docker deployment phase (the user has not logged in yet), because both requests (preview and upload) to upload the asset require authentication.

If we use docker to implement the whole process directly, is there any secure way to get the jwt token? If the request which upload the asset is encapsulated in docker, there needs to be a way to obtain the token at the deployment stage.

The potential solution may be: docker is used to encapsulate the asset. After the user logs in, we get the list of paths to obtain the asset through the defined api (get_asset), and then send two requests to upload the asset on the front end.

[dominikriemer]

Hi @Cccccrystall do you already have some code available so that we can have a look and find a good way how to import the assets in a secure way?

[Cccccrystall]

Hi, @dominikriemer! I have add a PR to show the current bash script, which is "Adding the POC of uploading default asset(not official PR) #1933".

[Cccccrystall]

Purpose

introduces a new docker-compose.quickstart.yml file and related configurations, aiming to streamline and enhance the user experience for quick project setups.

Solution

How to use Quickstart Deployment Mode

1. Run Streampipes core
   Use "mvn clean package" in the streampipes root folder
2. Start the UI of Streampipes
   Enter the ui folder and use "npm install" and "npm run build"
3. Start docker in quick mode
   Enter the installer/compose folder
   Use "docker-compose -f docker-compose.quickstart.yml build script-runner" to build the docker
   Use "docker-compose -f docker-compose.quickstart.yml up -d" to start all services!
   4.After services are started, go to http://localhost/, you could see the defalt assets!

New Docker Compose File:
 
Introduced docker-compose.quickstart.yml, derived from the existing docker-compose.full.yml.
Added additional configuration for the script-runner service, detailing build context, volume mapping (including the mapping of zip_folder), network dependencies, and service dependencies.
 script-runner: build:. volumes: -./zip_folder:/zip_folder networks: spnet: depends_on: -backend
Dockerfile for Script-Runner:
 
Created a Dockerfile specifying the base image as ubuntu:latest.
Installed necessary utilities: curl and jq.
Added and made executable two Bash scripts: upload.sh and wait-for-it.sh.
 
Upload Script:
 
The script checks environment variables for user credentials, logs in to obtain an access token, and processes and uploads ZIP configuration files located in /zip_folder to the backend service.

 
ZIP Configuration Files and Folder Mapping:
 
Included a new folder zip_folder containing ZIP configuration files to be used by the upload.sh script.
The zip_folder is mapped to the script-runner container, allowing direct access and processing of the ZIP files within the container.
The introduction of zip_folder mapping facilitates easy additions of default ZIP configurations. Whenever there is a need to introduce new default configurations, we can simply add the corresponding ZIP files to the zip_folder, making the process more efficient and user-friendly.
 
Wait-for-it Script:
 
Incorporated the wait-for-it.sh script, enhancing the container’s capability to wait for the availability of the dependent backend service before executing upload.sh. This script is from: https://github.com/vishnubob/wait-for-it/blob/master/wait-for-it.sh

Usage Instructions:

To utilize the new configurations, the user should build the script-runner service with the command: docker-compose -f docker-compose.quickstart.yml build script-runner.
Subsequently, to trigger the setup, the user should run: docker-compose -f docker-compose.quickstart.yml up -d.
 
Future Plan:
As part of our ongoing efforts to enhance user experience, we plan to package the script-runner service as a separate Docker image and upload it to Docker Hub. This step will allow users to skip the build step and trigger the service directly, thereby further simplifying the setup process.
 
These enhancements are designed to facilitate easier and more efficient project setups, especially beneficial for users seeking a quickstart option. By introducing the script-runner service with automated script executions and additional configurations, we are enabling seamless interactions and operations for different project use cases.

Here is the Demo: