[Docs] Misc typos/ambiguities in https://www.marblerun.sh/docs

[dimakuv]

Issue description

I'm going through your whole documentation webpage. Random typos/ambiguities/broken links found while reading the documentation:

1. https://www.marblerun.sh/docs/getting-started/quickstart/

   • "The CLI will obtain the Coordinator’s remote attestation quote and verify it against the configuration on our release page." -- the link is broken. Points to https://www.marblerun.sh/docs/getting-started/quickstart/github.com/edgelesssys/marblerun/releases/latest/download/coordinator-era.json but must point to https://github.com/edgelesssys/marblerun/releases/latest/download/coordinator-era.json
   • You have marblerun certificate root $MARBLERUN -o marblerun.crt --insecure but you don't explain what this --insecure option does. Given that Marblerun is already installed in simulation mode, what does this insecure option disable additionally?
   • You have marblerun manifest set tools/manifest.json $MARBLERUN --insecure but you don't explain what this --insecure option does. Isn't setting the very first manifest into Marblerun insecure by design? (And then end users should actually verify the manifest before trusting Marblerun.)
   • You have marblerun namespace add emojivoto --no-sgx-injection but you don't explain why you add --no-sgx-injection option. I assume that's because we run Marblerun in simulation mode without SGX, so there is no sense in adding the Marblerun-sgx label.

2. https://www.marblerun.sh/docs/getting-started/coordinator/

   • You have "EDG_COORDINATOR_CLIENT_ADDR: The listener address for the HTTP server". It would be more specific to replace "HTTP" with "HTTP-REST" since this is how you call it in the very beginning of the page.
   • The last sentence is: "The Client API serves both use-cases with a compact REST-API." However, in all other documentation you seem to use "HTTP-REST API".

3. https://www.marblerun.sh/docs/features/attestation/

   • General question: why do we need the Intermediate certificate? I.e., why does the Coordinator need to be both the Root CA and the Intermediate CA? Is there any benefit in having R - I - M chain instead of a simpler R - M chain? Or is it simply to have a "classic" three-level certificate chain?
   • "Marblerun relies heavily on the Data Center Attestation Primitives (DCAP)" -- the link is outdated (from 2019). I suggest to replace with: https://download.01.org/intel-sgx/sgx-dcap/1.11/linux/docs/DCAP_ECDSA_Orientation.pdf
   • In "Coordinator deployment", you say "It generates a remote attestation quote which contains the hash of the root certificate as well as a hardware measurement of its enclave". However, on the diagram you only show the hash of the root certificate. Please update the text or the diagram (whatever is correct). If the Quote only contains one of these, please describe how the second one is embedded in the first one. If the Quote contains both of these, please describe on the SGX Quote example how you cram both in it (I assume you use 64-byte report_data field and put two 32-byte SHA hashes).
   • In "Marble deployment and attestation", the diagram is a bit confusing. First, I would expect an arrow from "Verify Quote" to "Manifest" -- meaning that the Quote is verified against the contents in the Manifest. Second, I don't understand the "enforce" arrow -- the Marble deployment is not really enforced but rather just happens at runtime, right? If you meant that the Marble is not "admitted" to the mesh if its Quote is bad, then I would replace "enforce" with "admit" or "allow". Finally, under "deploy certificates" you have a strange order: Intermediate - Root - Marble. I would expect Root - Intermediate - Marble.
   • In "Confidential service mesh", you say "Different Marbles can communicate with each other in the same way". This is a confusing sentence. What you mean is that marble-to-marble communication can happen via normal TLS with classic X.509 certificates, since both M certificates are signed by the same I certificate and are rooted in the same R certificate. I would prefer this sentence to be rewritten to be more specific.
   • In the same "Confidential service mesh", you say "the Coordinator’s certificate is used as a trusted CA". This sounds lingiustically wrong (the certificate cannot be a CA), probably "the Coordinator serves as a trusted CA" is a better wording.
   • In the same "Confidential service mesh", the diagram has "Client uses Coordinator's root CA for TLS connection..." -- this sounds wrong. A better phrase would be "Client installs the Coordinator's root certificate to verify TLS connection...". Also, do Marbles send the certificate chain I - M? Or just M?
   • In "Cluster attestation", you don't show on the diagram how the Root certificate is sent to and installed on the Client. Please add this arrow. Also, you again show only "the hash of the root certificate" and do not show the measurement of the Coordinator.

4. https://www.marblerun.sh/docs/features/secrets-management/

   • "the Coordinator injects a Marble’s virtual sealing key through the {{ hex .Marblerun.SealKey }} placeholder in the Parameters section of the manifest" -- using this key requires awareness from the application/framework (i.e., the app must explicitly encrypt and save secrets to disk, using e.g. an environment variable to get the encryption key). I guess you assumed it for EGo and Edgeless RT applications. However, this doesn't work in most cases for Graphene/Occlum/SCONE because the app is unaware, and these frameworks actually do the encryption transparently on certain files. In particular, Graphene relies on special manifest options to mark which files are to be sealed (see [Pal/Linux-SGX] Add sgx.protected_mr{enclave,signer}_files manifest options gramineproject/graphene#2484). You should probably add a note like this, and maybe create an issue in Graphene github to support this kind of files?
   • In "Shared symmetric keys", how is it populated through the Marblerun manifest? I would expect a placeholder similar to {{ hex .Marblerun.SealKey }}. Please add the explanation. Or is this outdated?
   • I am also missing some dicsussion on the kinds of secrets the applications typically need. Is it really enough to have one sealing key + TLS credentials? I like the minimalism of this approach but wonder if the Coordinator could generate something more than that. UPDATE: The Coordinator can also generate Secrets described in section "Marblerun: Secrets"; please add a link to this section.
   • In "TLS credentials", you say "The certificate is signed by the Coordinator’s root CA". But this doesn't seem to be true -- the Coordinator actually uses the Intermediate CA to sign the Marble certificate. Please fix.

5. https://www.marblerun.sh/docs/features/recovery/

   • [no issues]

6. https://www.marblerun.sh/docs/features/transparent-tls/

   • TTLS sounds quite hard to use in unmodified applications, from my experience (I mean, we tried this with Graphene). So at least from Graphene's standpoint, we plan to not support/recommend TTLS.
   • You say "You can use custom credentials defined in the manifest’s secrets". How does this work, what are the exact manifest options to specify? An example would help here.

7. https://www.marblerun.sh/docs/features/auto-injection/

   • I don't know Kubernetes well, but can you have a Kubernetes pod with several Docker containers (and thus several Marbles) in one pod? Or does Marblerun integration with Kubernetes require the one-docker-per-pod scheme?

8. https://www.marblerun.sh/docs/features/runtimes/

   • [no issues]

9. https://www.marblerun.sh/docs/features/planned-features/

   • Do you have in plans the NTP/NTS server or something like this embedded in the Coordinator? This would be extremely helpful! E.g., Graphene could then transparently connect and ping the Coordinator for trusted timestamps.

10. https://www.marblerun.sh/docs/deployment/cloud/

    • In "Azure Kubernetes Services (AKS)", you have this sentence: "This section describes the workflow to create an AKS cluster with confidential compute VMs and deploy Marblerun". But there is no workflow description on this page. Did you forget to put a link? Otherwise just remove this sentence.

11. https://www.marblerun.sh/docs/deployment/on-prem/

    • "You need to install the SGX Driver for systems supporting the Launch Control Configuration." The link is broken.
    • "DCAP is the new attestation mechanism for SGX replacing EPID." The link is broken.
    • "You can find an overview of DCAP in the official Intel docs." This link is very old, please replace with 1.11 link (see above).
    • "The QGL and QVL libraries need to be configured to talk to a Provisioning Certificate Caching Service (PCCS)." This link is very old.
    • "Install the DCAP client libraries". This link is broken.
    • "Currently, the Marblerun Coordinator and EGo docker images..." The links are weird here, was this intended?

12. https://www.marblerun.sh/docs/deployment/kubernetes/

    • In "Out-of-process attestation", you have a typo: "While Marbles build with Ego" (must be "built").
    • What does marblerun install do exactly? The section misses a paragraph or a link with a brief description of what exactly the "Marblerun installation" does.
    • In "Option 1: Install with the Marblerun CLI", you write: "Depending on which domain you expose Coordinator’s client API, you need to set the CommonName accordingly". I don't understand this sentence: do you mean that you should specify --domain as the same one where your Coordinator is deployed? If this is what you mean, then your sentence seems correct but strangely worded.
    • In "Option 2: Install with Helm", you have the same sentence as above. Please fix both.
    • In "Option 2: Install with Helm", you have "See the how to add a service documentation for more information on how to create and set a manifest". The add a service documentation is highlighted as code but it shouldn't be highlighted.

13. https://www.marblerun.sh/docs/deployment/standalone/

    • In "Run your workloads", you only mention EGo and Graphene. Should add Occlum now as well.
    • In "Run your workloads", you have EDG_MARBLE_TYPE=<your_marbler_type> erthost <your_marbler_binary>. What is "marbler"?

14. https://www.marblerun.sh/docs/workflows/define-manifest/

    • In "Manifest:Packages", you write "Future versions of Marblerun will accept any SecurityVersion that is equal or higher than the one specified in Packages for a given combination of SignerID and ProductID". But why in "future versions"? Seems like a trivial change in the code.
    • In "Manifest:Marbles", you use examples like "Package": "backend" and "Package": "frontend". It would be better to have a running example with pkg0 and pkg1 from the previous section. Update: actually, you use backend and frontend names everywhere else, so it makes sense to rename pkg0 -> backend and pkg1 -> frontend.
    • Here "In the remote attestation step, the Coordinator ensures that enclaves run the software-defined in the Packages section", you have a redundant - sign, please remove.
    • Looking at the list "The following named keys and certificates are always available", I don't see Intermediate certificates. This R - I - M vs R - M inconsistency is in many places of the documentation...
    • The manifest's secret has a field UserDefined. If set to true, "it will be uploaded by a user at a later date". How does this function? There is a client API for sending a secret value? Could you describe or add a link?
    • In "Manifest:TLS", you should mention that this is only applicable to EGo and Edgeless RT. Graphene doesn't support this, and I think Occlum as well.

15. https://www.marblerun.sh/docs/workflows/set-manifest/

    • I don't find the included manifest example required (after reading the previous section). I think you can safely remove it to deduplicate your documentation, and just give a link to the previous section.
    • You say "To set the manifest we can use the command line interface, which automatically performs remote attestation before uploading the manifest". But how can a user "automatically perform remote attestation" of the Coordinator? You have a separate section where you describe how to gain trust in the Coordinator (https://www.marblerun.sh/docs/workflows/verification/), so what does this sentence even mean?
    • You say "If the manifest contains a RecoveryKeys entry, you will receive a JSON reply including a recovery secret, encrypted with the public key supplied in RecoveryKeys". Why do you call it a "recovery secret"? Isn't it more specific to call it a "recovery encryption key"? From what I understood about recovery in Marblerun, the Coordinator generates an encryption key with which its state is encrypted, and this encryption key is itself encrypted with the public RecoveryKey.

16. https://www.marblerun.sh/docs/workflows/add-service/

    • In Step 1, you don't have a link to "Building Services -> Occlum".
    • Also, you don't have a proper "Building Services" documentation for Edgeless RT (the link points to one of the examples on GitHub). But I guess that's because Edgeless RT is getting deprecated (or simply not that interesting to merit a separate page).
    • "Skip this step, when using EGo with TTLS." -- here it would be good to add a link to TTLS.
    • In Step 2.1, you seem to be assuming only Edgeless RT-based enclave applications. Please add a note that for other frameworks, the enclave-signing process is different, and MRENCLAVE/MRSIGNER/etc are taken from their respective tools.
    • In Step 2.2, there is a typo: "environments variables". Remove the s.
    • In Step 2.2, the link doesn't work for me: "Otherwise, as discussed in Step Dev #1.1".
    • In Step 2.2., you write "If your service is written in Go". But in other places you use the phrase "When using EGo" -- which I find more specific.
    • In Step 3, the bash command line needs to be horizontally scrolled (even on my big screen). This is annoying; I suggest to use the \ line-break notation.
    • In Step 3, you use erthost which again assumes the Edgeless RT-based enclave app. Please add a note on other frameworks.
    • In Step 3, you write "When you deploy the Coordinator using our Helm repository as is described in our deploying Marblerun hands-on" -- the link doesn't jump to anything meaningful.
    • In Step 4, the indentation for spec -> container -> env name/value pairs seems broken. See e.g. a correct example here: https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/

17. https://www.marblerun.sh/docs/workflows/verification/

    • You write "Marblerun provides a simple REST-API". Please rename to "HTTP-REST API".
    • In the example you give (era -c coordinator-era.json ...), you never explicitly mention that the era tool compares the values in Coordinator Quote against the values in coordinator-era.json. Please add a sentence like this.
    • In another example you give (marblerun manifest verify manifest.json $MARBLERUN), it's unclear what manifest.json is -- is it your local version of the manifest file (which will be hashed and compared against the received hash)? Please add a sentence.

18. https://www.marblerun.sh/docs/workflows/update-manifest/

    • You write "If no administrator has been initially set up, no updates can be applied". What "administrator"? You talk only about users. I guess it is "a user who belongs to the role that has corresonding access rights"?
    • You write "Please do not define other values except than SecurityVersion value for a package". There is a typo than -> the.

19. https://www.marblerun.sh/docs/workflows/managing-secrets/

    • "For example, an application might require an access token, to establish a secure connections to an external service". Typo: a secure -> secure.
    • "Similarly, predefined users can retrieve particular secrets from the Coordinatorvia the...". Typo: Coordinator via.
    • "For example, the following command to requests a secret named..." Typo: remove to.
    • "If the user has the require permissions". Typo: required.
    • "via the REST-API". Change to HTTP-REST API.
    • I do not understand the first example in "Setting a secret" at all. Where is user-certificate secret name in the command line? How does marblerun secret command understands that the file contains two things (the cert and the key)?

20. https://www.marblerun.sh/docs/workflows/updates/

    • [no issues but I am bad at Kubernetes updates]

21. https://www.marblerun.sh/docs/workflows/recover-coordinator/

    • In the first option, it is unclear where this recovery_key_encrypted_base64 file comes from. Please specify that this file was created by the user and populated with contents that were sent by Marblerun Coordinator when the user set a new manifest (with the RecoveryKeys option).

22. https://www.marblerun.sh/docs/building-services/graphene/

    • "For a better illustration of the differences between the two premain variants..." I guess this is outdated. There are no two premain variants these days.
    • In newer Graphene versions, we deprecate loader.insecure__use_host_env = 1 and instead use proper booleans: loader.insecure__use_host_env = true. Not critical though. (Same for sgx.remote_attestation = 1.)
    • You say "Ideally, these would be placed in the Marble’s in-memory filesystem". I don't know what this means but Graphene supports tmpfs feature: https://graphene.readthedocs.io/en/latest/manifest-syntax.html#fs-mount-points

That's it. I didn't go through some other pages. My main goal was to understand Marblerun in details, so I decided to also document everything that didn't work/looks wrong to me. Your documentation is very good, thanks for this!

---

P.S. How does TTLS work for your workloads? From what I remember, transparently wrapping non-TLS applications in TLS connections is painful and has corner cases. Did it work for you? Did you have to change something in Google Go runtime libraries? Did you encounter unsupported cases?

[m1ghtym0]

Thanks, @dimakuv for going through this!

[Nirusu]

@dimakuv I hope you don't mind that I edited your bullet point list to a task list. Should help us getting through this list a little bit more organized, we'll get back to you with changes & clarifications soon! :)

[dimakuv]

Just found one more bug in your QuickStart documentation.

The documentation is all in simulation mode, but you have this command:

marblerun status $MARBLERUN

This doesn't work in simulation mode (tried against Marblerun deployed in Azure Kubernetes Service):

$ marblerun status localhost:4433
No era config file specified, getting config from https://github.com/edgelesssys/marblerun/releases/download/v0.4.0/coordinator-era.json
Got latest config
Error: no quote received

Adding --insecure option however makes it succeed:

$ marblerun status localhost:4433 --insecure
Warning: skipping quote verification
3: Coordinator is running correctly and ready to accept marbles.

[Nirusu]

Already addressed in https://github.com/edgelesssys/marblerun.sh/pull/72 :)

[katexochen]

@dimakuv Here are some answers to your questions regarding the attestation process in addition to edgelesssys/docs.edgeless.systems#19

Is there any benefit in having R - I - M chain instead of a simpler R - M chain?

The chain helps to separate Marbles between different updates, e.g. with different security versions, since the intermediate certificate changes during an update. I've added a new paragraph to the section to address this.

If the Quote contains both of these, please describe on the SGX Quote example how you cram both in it (I assume you use 64-byte report_data field and put two 32-byte SHA hashes).

The report_data field only contains the hash of the root certificate. The hardware measurement, which is included, is the MRENCLAVE field of the report.

Also, do Marbles send the certificate chain I - M? Or just M?

This depends on how Marbles use the certificates, e.g. EGo's GetTLSConfig function uses the chain I - M.

[m1ghtym0]

Do you have in plans the NTP/NTS server or something like this embedded in the Coordinator? This would be extremely helpful! E.g., Graphene could then transparently connect and ping the Coordinator for trusted timestamps.

We haven't discussed how would expose the trusted time to Marbles, but this is a great idea:-)
The initial step would be to get a trusted time in the Coordinator.
I'll add this to the proposal.

[m1ghtym0]

I don't know Kubernetes well, but can you have a Kubernetes pod with several Docker containers (and thus several Marbles) in one pod? Or does Marblerun integration with Kubernetes require the one-docker-per-pod scheme?

@dimakuv You can have multiple containers per pod. Currently, we inject those env vars to every container in the pod.
That should probably be mentioned in the docs. Technically, we could also move those labels to containers. Probably makes more sense.

[EDIT]
Labels are only available on the Pod level. Makes sense in the Kubernetes context. We need to think about something else here or live with the fact that we injecting into sidecars etc.

[m1ghtym0]

Looking at the list "The following named keys and certificates are always available", I don't see Intermediate certificates. This R - I - M vs R - M inconsistency is in many places of the documentation...

@dimakuv I see why this is confusing, but I'm not sure what's the best way of addressing this.
The R - I - M is essentially an implementation detail that we explain in our docs to give a throughout understanding of how MarbleRun works. It surfaces for the user when interacting with the client API, hence, it is relevant to explain it.
For the Marble side, it is completely transparent that there is another certificate in the mix.
It only knows one root certificate and its own leaf certificate.
In other words, depending on which side you come from, the MarbleRun's certificate chain either looks like R - I -M or R - M to you.
This is why we have a bit of an ambiguity here.

[3u13r]

@dimakuv Regarding point 6.1:

TTLS sounds quite hard to use in unmodified applications, from my experience (I mean, we tried this with Graphene). So at least from Graphene's standpoint, we plan to not support/recommend TTLS.

We currently only support TTLS with EGo/Edgeless RT. We also added a warning about that here. Furthermore, our current TTLS implementation just supports TCP sockets. This satisfies our needs, as we just want to transparently encrypt the traffic between marbles or lift outward facing http servers to https.
From our tests, TTLS works great for http and grpc requests in Go without changing its runtime. We had some issues with handling TLS errors, but they are solved now.

[m1ghtym0]

Hi @dimakuv,
We are done with the checkboxes. There were a couple of points that we tried to answer in this thread.
Please let us know if you have further questions/remarks/things we need to address.
Thanks so much already for taking the time, going through the whole documentation, and finding all those issues!!:-)
Very much appreciated!

[dimakuv]

Thanks! Closing this issue.