Using cromshell with Cromwell on Azure (kubectl port forwarding)

Cromwell provides a REST API that allows you to submit, abort, and get the status on workflows running on your Cromwell instance. There are several tools available to provide a user friendly interface to this REST API, such as cromshell. These tools all work off the same REST API and are supported by Cromwell on Azure by using port-forwarding to make the remote Cromwell AKS pod accessible from your computer.

Requirements

• WSL, Linux, OS X terminal, or Powershell environment
• Azure CLI installed/logged in, installation
• kubectl CLI, installation
  • See OS X kubectl install for ARM M1 OS X machines
• Azure Subscription ID, resource-group, and Kubernetes Service resource name
• (optional) tenant domain name if you are in a multi-tenant setup

Setting up/authenticating kubectl

1. Use az login or az login --tenant "example.onmicrosoft.com" to authenticate.
2. Next authenticate to your Kubernetes instance, note this will overwrite your current kubectl config

AZURE_SUBSCRIPTION_ID="12345678-1234-1234-1234-123456789012"
AZURE_RESOURCE_GROUP_NAME="cromwell-orange-rg"
AZURE_AKS_NAME="cromwell-0dd78956caae"
az aks get-credentials --resource-group $AZURE_RESOURCE_GROUP_NAME --name $AZURE_AKS_NAME --subscription $AZURE_SUBSCRIPTION_ID --file ~/.kube/config

Note, if you don't want to hard-code your AZURE_AKS_NAME you can also script this to retrieve the Kubernetes name (assuming you have 1 Kubernetes instance in your specific subscription, resource group). It is less error prone if you just record your AKS_NAME though.

AZURE_AKS_NAME=$(az aks list -g $AZURE_RESOURCE_GROUP_NAME --subscription $AZURE_SUB_ID --query "[].name" --output tsv)

3. Confirm that your kubectl can talk to the resources you expect:

> kubectl -n coa get svc
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
cromwell   ClusterIP   10.1.4.161   <none>        8000/TCP   30d
tes        ClusterIP   10.1.4.225   <none>        80/TCP     30d

4. Double check that the pod names that you see exactly match what you expect from the Azure portal. If these numbers don't match you might be connected to the wrong tenant, resource-group, or subscription.

Setup port-forwarding

Run kubectl port-forward -n coa svc/cromwell 8000:8000 to port forward the Cromwell REST API (defaults to port 8000 in CoA) to your localhost port 8000. The kubectl command needs to stay running (and actively forwarding) connections the entire time you want to use the Cromwell REST API/cromshell. So, leave it running in another terminal window (and use the bash loop below to restart it if anything goes wrong).

Now if you open up a browser to http://localhost:8000 you'll be presented with the Swagger page for the Cromwell server.

This command is documented here kubectl docs

Test that everything works

You can also test that everything works by using curl (and jq):

curl -s -X POST "http://localhost:8000/api/workflows/v1/query" -H "accept: application/json" -H "Content-Type: application/json" -d "[{\"status\":\"Running\"},{\"status\":\"Submitted\"}]" | jq

{
  "results": [
    {
      "id": "41e32a30-101d-4b68-a77b-02053fc21460",
      "name": "GatherSampleEvidence",
      "start": "2023-01-01T10:01:00.000Z",
      "status": "Running"
      "submission": "2023-01-01T10:00:00.000Z"
    },
  ],
  "totalResultsCount": 1
}

Check with cromshell:

> export CROMWELL_URL=localhost:8000;cromshell status 113556d4-ac73-492e-bf9d-03eb5edcf469
Sub-Command: status
Using workflow-id == 113556d4-ac73-492e-bf9d-03eb5edcf469
Using workflow server URL == localhost:8000
    ,,    ,,     ,,
    \‾\,-/‾/====/‾/,
     \/=<‾><‾><‾><‾>-,
     / (\‾\‾|‾/‾/‾/‾
    \‾x/ ˙'-;-;-'˙
     ‾‾
{
  "status": "Failed",
  "id": "113556d4-ac73-492e-bf9d-03eb5edcf469"
}

Troubleshooting

If your kubectl command keeps disconnecting use the following bash script to restart the command in case it fails:

#!/bin/bash

while true; do
    kubectl port-forward -n coa svc/cromwell 8000:8000
    if [ $? -eq 0 ]; then
        break
    fi
    sleep 1
done