Problem using Nexus to proxy docker hub

[dvoet]

Describe the bug
I am trying to setup nexus in the AzureTRE environment to proxy docker hub. I believe I have nexus setup correctly and appropriate docker hosts added the the firewall rules. When I try to run docker login from a VM running in a workspace shared service subnet I see error similar to what is described here. According to Sonatype docs docker needs to be exposed on a different port than the standard ssl port nexus is already running on. However nexus is running in an Azure App Service and I don't see a way to expose custom ports.
Steps to reproduce

1. setup Nexus to proxy docker hub
2. add *.docker.io and *.docker.com to firewall rules to allow nexus to access them
3. create workspace and start vm in shared services subnet
4. connect to vm using bastion and run docker login nexus-[tre_id].azurewebsites.net:[port]
5. see timeout error

Acceptance criteria

• (Nexus Docker Hub) Change Nexus hosting to VM #1479
• (Nexus Docker Hub) [Spike] Generate Letsencrypt SSL cert #1480
• (Nexus Docker Hub) Consume SSL cert in Nexus and connect via User resource VM #1481
• Configure Docker Hub proxy in Nexus using a standalone port

[oliver7598]

Can this be assigned to me

[tamirkamara]

@oliver7598 do you have an idea on how to address this?

[oliver7598]

@tamirkamara myself and @jjgriff93 are going to pair on this as a priority over the next sprint

[tamirkamara]

Sounds good. Can you please share more details on the plan once you have it?
IIRC, I was the one who added this to begin with, so let me know if there's something I can help with...

[marrobi]

This doc will be useful - might be we add a second app service just for docker?

[jjgriff93]

Current challenge is exposing a second port in App Service to allow docker hub connections, which is prevented by this limitation: https://docs.microsoft.com/en-us/archive/blogs/waws/things-you-should-know-web-apps-and-linux#you-can-only-expose-one-port-to-the-outside-worldapplies-to-web-app-for-containers

[jjgriff93]

Potential solution is moving nexus to container instances which does support multiple ports, Oliver and I will do some investigation into this

[marrobi]

Container instances could work, still need an SSL certificate. VM is potentially a better option - easier to mount disks, backup etc.

Either way I don't think we can avoid generating SSL certificates with ACI or a VM.

[marrobi]

@jjgriff93 do you think this would work...

Create a new web app - that shares the same config and data store. The second web app exposes the docker proxy port. The first (existing web app) is used for the user interface. Not sure if that would work.

[jjgriff93]

We had this thought also, will require some testing as to whether Nexus would support that - I don't think we'd be able to configure anything because we couldn't post configuration to the Nexus API to set up the docker proxy if we've exposed a different port for that instance

[jjgriff93]

Have set up a meeting to discuss further after tomorrow's stand-up - @tamirkamara have invited you if you're able to attend as your input would be valued. If not feel free to post your thoughts on this ticket. Summary of the planned meeting discussion:

• How best to enable two exposed ports (limited to one for app service) so we can proxy docker hub on a separate port to the general nexus port (as per https://help.sonatype.com/repomanager3/nexus-repository-administration/formats/docker-registry/ssl-and-repository-connector-configuration)
• If we choose to host Nexus on container registry / VM to get around port issues, how we then plan to set up SSL certs in lieu of the app service provided one

[marrobi]

Scrap the multiple web app idea. The SSL certificate needs to be loaded into nexus - can't do that with web apps as we don't have access to the cert.

Could not configure HTTPS connector on port 5000 for docker repository dockerhub

Searching got me:
"Docker registries are required to use HTTPS. This message means that you have not configured Jetty with a keystore from which it can load a TLS certificate for the Docker HTTPS connectors"

I see no option that to use a trusted SSL certificate, and in that case would run in docker on a VM/VMSS.

To get a trusted SSL, either

1. Need to procure a wildcard cert that can be used throughout the TRE (in the past has been a requirement for many production use cases).
2. Expose an endpoint publicly, get a lets encrypt cert, then make private - messy.
3. Use self signed certs and configure VMs to trust these certs... also not great.

[stuartleeks]

Possible approach for working with private VM endpoint proposed by @marrobi in a discussion on this (in terms of supporting dev environments)

 ┌──────────────────────────────────────────┐    ┌────────────────────────────────┐
 │                                          │    │                                │
 │ Public IP: mytre123.uksouth.cloudapp.net │    │  Storage static site           │
 │                                          ├───►│                                │
 │                                          │    │                                │
 └──────────────────────────────────────────┘    └────────────────────────────────┘
                                                        ▲
                                                        │
                                                        │
                                                        │
                                                        │
                                                        │
 ┌──────────────────────────────────┐                   │
 │                                  ├───────────────────┘
 │  Letsencrypt cert                │
 │  processor (run on schedule)     ├────────────────────┐
 │                                  │                    │
 └──────────────────────────────────┘                    │
                                                         │       ▼
                                                         │
                                                         │
                                                         │
                                                 ┌───────▼────────────────────────┐
                                                 │                                │
                                                 │  Key Vault                     │
                                                 │                                │
                                                 └────────────────────────────────┘
                                                         ▲
                                                         │
                                                         │
                                                         │
                          ┌──────────────────────────────┼────────────────────────────────────┐
                          │                              │                                    │
                          │   VNET                       │                                    │
                          │                              │                                    │
                          │                              │                                    │
                          │                      ┌───────┴─────────────────────────────┐      │
                          │                      │                                     │      │
                          │                      │ VM                                  │      │
                          │                      │                                     │      │
                          │                      │                                     │      │
                          │                      └─────────────────────────────────────┘      │
                          │                                                                   │
                          │                                                                   │
                          │                                                                   │
                          │                      ┌─────────────────────────────────────┐      │
                          │                      │                                     │      │
                          │                      │  Private DNS Zone:                  │      │
                          │                      │  map mytre123.uksouth.cloudapp.net  │      │
                          │                      │         to VM IP address            │      │
                          │                      │                                     │      │
                          │                      │                                     │      │
                          │                      │                                     │      │
                          │                      └─────────────────────────────────────┘      │
                          │                                                                   │
                          │                                                                   │
                          │                                                                   │
                          └───────────────────────────────────────────────────────────────────┘
                          

[jjgriff93]

Have added two new issues (see above) to start tracking the work on this

[martinpeck]

The three tasks associated with this bug are close, and the PR #1584 is merged, so closing this one out.

Thanks all!

[jjgriff93]

Re-opening as #1584 was a dependency to get this one closed, but now that the dependency is in main we can configure docker hub as a repository in nexus as per this story to resolve the original issue