Shareable Viz CLI

[stephkaiser]

Along with providing users the ability to host and share Viz via Kedro-Viz, users should also be able to host and share Viz via the CLI. This ticket is created to track the work needed for that implementation.

Shareable Viz flows here - https://miro.com/app/board/uXjVM8oNLys=/?share_link_id=39391792422

Related - #1116

[tynandebold]

Todo: @amandakys to spec out all the different steps in a prototype.

[tynandebold]

Relates to #1574

[noklam]

I have some more opinion about it, when this is picked up do ping me! Essentially when I try to build the GH Page support separately in #1574, I found that I also need a CLI so I hacked one already.

To make it more maintainable, I think the CLI should be share among different backend (i.e. S3, github page and more).

[amandakys]

An AWS focused CLI that maps to our existing UI flow

kedro viz deploy 

AWS Bucket Info
===============
Bucket Region: 
Bucket Name: 

...creating your webpage (23s) 

Success! Kedro Viz has been deployed. It can be accessed at: 
https://kedroviz-repo/project-123/folder-1/my-hosted-viz-pipeline-quantumblack

This interactive flow would be supported by two flags

--region            allow users to specify region inline
--bucket-name       allow users to specify bucket name inline 

Example

kedro viz deploy --region=eu_west_2 --bucket-name=hosted-viz

...creating your webpage (23s) 

Success! Kedro Viz has been deployed. It can be accessed at: 
https://kedroviz-repo/project-123/folder-1/my-hosted-viz-pipeline-quantumblack

If the user hasn't set up credentials

kedro viz deploy 

Deploying and hosting Kedro-Viz requires AWS access keys. 
To use this feature, please supply your AWS access keys as environment variables. 
More information can be found at: https://kedro-docs.org/visualisation/share_kedro_viz.html

Supporting Generic Deployment

Allow users to generate a static folder that they can deploy however/wherever they like

kedro viz build 

This command is the convention for static site deployment. This command can be run as part of a CI/CD flow to build and re-build an updated Kedro Viz and deploy to the user's chosen platform.

If we provide this command, it will be an important step to supporting generic deployment, as without any further work on our part, people will be able to deploy wherever they want if they put in some legwork themselves. We can then choose to simplify the process for them if we want by proving pre-written github actions or other convenience code snippets, but we wont be a blocker.

@noklam has hacked together a working prototype of this command based on the s3 deployer <3 and showed how this can then be used with github actions to deploy to github pages.

Allow users to specify which platform

kedro viz deploy --platform=aws

Typically, build and deploy are 2 separate steps that we combine in our UI flow. If we have a build command we could also have a deploy command that deploys the most recent build. i.e. you can make changes to the branch, but if you don't run build it will deploy that last build. In this way we could also allow people to deploy their build to multiple different platforms using the --platform flag.

In addition, if we do start with the AWS focused CLI workflow, this could be somewhere to move the feature, when we become more generic. It would very much be an extension on build, more nice to have than essential

For example

kedro viz deploy --platform=aws

AWS Bucket Info
===============
Bucket Region: 
Bucket Name: 

...creating your webpage (23s) 

Success! Kedro Viz has been deployed. It can be accessed at: 
https://kedroviz-repo/project-123/folder-1/my-hosted-viz-pipeline-quantumblack

Managing Credentials

As per discussion with @tynandebold, we decided that we would not ask users to input their credentials as part of the CLI flow, as this was not common practice. Instead the command will read credentials from where users have defined it, similar to how the UI assumes credentials are set up properly.

Currently for Viz features, we ask users to set up their credentials as environment variables. In some user interviews for the Shareable Viz feature, we heard that some users were actively using multiple credentials for different projects, or regularly needed to update their credentials due to short session times. For this Tynan suggested that we delegate to how AWS handles this problem using shared config and credentials files but we were worried this would further tie us unnecessarily to AWS, where one of our goals for the feature was to extend it to support other deployment platforms.

@noklam provided further context on how credentials are managed on the framework side.

1. with credentials.yml

# catalog.yml
my_dataset:
   ...
   credentials: s3

# credentials.yml
s3:
  key: XXX
  secret: XXX

2. with environment variables

# catalog.yml
my_dataset:
  ...
  credentials:
     key: ${os.env: AWS_XXXX}
     secret: ${os.env: AWS_XXXX}

3. with both

# catalog.yml
my_dataset:
  ...
  credentials: s3

# credentials.yml
s3:
    key: ${os.env: AWS_XXXX}
    secret: ${os.env: AWS_XXXX}

Next Steps

Further discussion is needed to decide on how we can standardise credential management across Viz and Framework side, and what method will more easily enable Shareable Viz to be an AWS agnostic feature down the line.

I would also advocate for us taking the time to consider including the build command as a fast follow after our initial release of the Shareable Viz feature. This would allow us to initially support generic deployment, while we expand our UI and docs and consider other features to improve the generic deployment workflow.

[stephkaiser]

Nicely considered and written @amandakys and thank you @noklam!!

I don't have any feedback here - the flow looks good to me, and I love the forward thinking on a generic deployment approach (platform flag is cool) as well as considering about credentials management and standardisation across kedro.

For next steps - do you think this is something for us to bring to tech design, or should we organise a separate discussion?

[noklam]

Typically, build and deploy are 2 separate steps that we combine in our UI flow.
👍🏼

kedro viz deploy
As discussed, I am not sure if this is feasible without breaking change because currently kedro viz is a command but not a command group. In my PoC I use kedro viz-build.

For next steps - do you think this is something for us to bring to tech design, or should we organise a separate discussion?

I will summarise the learnings I have and put together some diagrams. For meeting/tech design I will let @amandakys decide how we should go about it.

[amandakys]

At our meeting 30/10/23 we decided the following action items with attendance: @tynandebold @merelcht @stephkaiser @MehdiNV @noklam @NeroOkwa

1. Change kedro viz to kedro viz run
   Change kedro viz to kedro viz run, make viz a command group  #1616
2. Implement kedro viz build to give users access to build files
   Kedro Viz Build  #1615
3. Implement AWS-focused CLI for shareable URLs feature
   AWS focused Shareable Viz CLI  #1636

---

Details

1. Change kedro viz to kedro viz run in order to support future keyword commands like kedro viz deploy and kedro viz build - high priority as it blocks release of all other work
   This is a breaking change, and can be preceded by another release to provide a deprecation warning. At initial release we can provided an informative error message when people use kedro viz like Did you mean kedro viz run?

2. Move forward with kedro viz build as the functionality already exists, we just have to expose it to users via the CLI

3. Move forward with an AWS focused CLI

kedro viz deploy 

AWS Bucket Info
===============
Bucket Region: 
Bucket Name: 

...creating your webpage (23s) 

Success! Kedro Viz has been deployed. It can be accessed at: 
https://kedroviz-repo/project-123/folder-1/my-hosted-viz-pipeline-quantumblack

We decided that we would proceed with a CLI that aligned with the current state of our UI deployment flow.
This will be complemented by the kedro viz build command that will allow users to deploy their build to any static site deployment service, but without direct support from Kedro Viz.

---

Credentials

Credentials will continue to be managed using environment variables as is the convention. If we later decide that we would like to use a more general credential management system we will revisit the options presented here using the credentials.yml file.

---

Generic Deployment

kedro viz build will provide the first steps to supporting generic deployment, by delegating the actual deployment step to the user, thus giving them greater choice and control over their deployment platform.

In the future, we may decide to build more integrated support for different deployment services in the same way that we support AWS, but we will re-evaluate this based on feature uptake. In this case we may introduce the kedro viz deploy --platform flag to specific which platform to deploy to. We would also need to update our UI flow to reflect the greater choice in platform options. #1531 tracks our ideas for further development on the shareable viz feature to support other platform.

---

Proposed Implementation Order

1. Change kedro viz to kedro viz run
2. AWS focused CLI
3. Kedro viz build - with changes in UI copy to docs on how to deploy to something other than AWS

[takikadiri]

Sharing a static kedro viz is a really appreciated feature for our team, as it can streamlines the collaboration between different personae of the data science project (including the stakeholders).

Currently we can't use the aws s3 shareable link, because we have a custom s3 endpoint in our private cloud. Moreover, we want to deploy the static kedro viz through the CI, to either Gitlab pages, github pages or private s3 (we don't decide yet).

We're looking forward to have the kedro viz build as it would open more deployment and possibilities.

[NeroOkwa]

Thanks @takikadiri for the feedback, it would be great to speak to you (and your team) to get more context on this and any other pain points you have with Kedro-Viz. Would you be open to that ?

cc @rashidakanchwala

[noklam]

@takikadiri that should be possible. I had an old demo that deploying static site with github action. I haven't updated it with the new kedro viz build command, but it's fairly easy to do so once the command is released.