Securing Container Images

Module Objectives

Enable GCR vulnerability scanning
View and filter vulnerability occurrences
Redeploy the cluster with Binary Authorization enabled
Configure and test custom Binary Authorization policy

Enable GCR Vulnerability Scanning

Google Container Registry may scan images for known vulnerabilities. It will notify you by adding a Vulnerabilities column in the image build list. You may extend the support further by enabling Cloud Pub/Sub queue notifications to trigger automated response and Binary Authorization to prevent insecure images from running on top of a Kubernetes cluster.

From Google Web Console go to Container Registry -> Settings
In the Settings click Enable Container Analysis API

Note: Container registry scanning will be enabled automatically
Now each image screen will show the vulnerabilities per build
Click on the vulnerabilities column to show the list of vulnerabilities. Each has a severity column which tells how dangerous the vulnerability is. The package column shows the component of the image that has the bug and a link to the CVE describing it.

Note: As of writing, scans currently support Alpine, Ubuntu and Debian based images

View and Filter Vulnerability Occurrences

Use the gcloud command to view a summary of vulnerabilities for an image

Here is an example of viewing vulnerability summaries with gcloud.

gcloud beta container images list-tags --show-occurrences \
  gcr.io/project-altoros-workshop/sample-k8s-app \
  --occurrence-filter='kind="PACKAGE" AND has_prefix(resource_url, "gcr.io/project-altoros-workshop/sample-k8s-app")'

DIGEST        TAGS   TIMESTAMP            VULNERABILITIES
1dd9b5827cc3  1.0.2  2018-11-30T10:36:17  CRITICAL=5,HIGH=35,LOW=21,MEDIUM=184
fbccfc8b25fe  1.0.0  2018-11-26T12:29:40  CRITICAL=5,HIGH=35,LOW=21,MEDIUM=184

This example shows the vulnerability summary for the image sample-k8s-app in the project project-altoros-workshop. Modify the request to show data on the image in your own repository.

Note: To get a vulnerabilities list, we need to talk to the API directly. There is no subcommands for the gcloud tool yet

Get an authentication token. You will put it into each request so Google Cloud can authenticate it and grant the same rights your user has.
```
gcloud config config-helper --format='value(credential.access_token)'
```
List vulnerability occurrences for your project
```
curl -s -XGET \
  -H "Authorization: Bearer $(gcloud config config-helper --format='value(credential.access_token)')" \
  https://containeranalysis.googleapis.com/v1beta1/projects/<PROJECT_ID>/occurrences?filter=kind%3D%22PACKAGE%22 | jq .
```
- curl is a tool to make HTTP requests from the command line
- -s option tells to hide download statistics information
- -XGET sets the HTTP method GET, a method to read data
- -H Authorization sets the authorization header
- https://containeranalysis.googleapis.com/v1beta1/ is the API we are going to use
- <PROJECT_ID> is substituted forthe actual value of your project ID
- filter=kind%3D%22PACKAGE%22 shows only occurrences of type PACKAGE
- jq . pretty-prints the json result
Lets list the names of all the packages affected, modify and run the previous command
```
jq ".occurrences[].installation.installation.name"
```
Note: The package name is under .installation.installation.name, we can filter this with jq
```
"libss2"
"procps"
"python-configobj"
"debconf"
```

There are less then 287 items in the list. Where are all the others? GCP API paginates the output so there is only a limited number of items in each page. If there are more items in the list the output has a nextPageToken item. You may repeatedly query the results using the pageToken parameter until you get all items. This exercise requires writing code so we omit it from the workshop.

Redeploy the Cluster with Binary Authorization Enabled

Enable the binary auth API

gcloud services enable binaryauthorization.googleapis.com

Create a cluster with the binary API enabled

gcloud beta container clusters create \
  --enable-binauthz \
  --zone us-west2-b \
  gke-workshop-1

Take a look at cluster default policy

gcloud beta container binauthz policy export  > manifests/policy.yaml
cat manifests/policy.yaml

Look through the policy YAML reference doc

Now check that you can deploy containers with Binary Auth enabled. The default security rule is ALWAYS_ALLOW

kubectl run nginx --image=nginx
kubectl get pods

NAME                   READY   STATUS    RESTARTS   AGE
nginx-8586cf59-zjxl4   1/1     Running   0          4s

kubectl delete deployment/nginx

Deny Policy

Let's modify the policy to deny all the containers except needed by GKE cluster

Modify the policy file manifests/policy.yaml and set evaluationMode: ALWAYS_DENY

Update the policy

gcloud beta container binauthz policy import manifests/policy.yaml

Try to deploy again

kubectl run nginx --image=nginx
kubectl get pods

Note: You will get a message No resources found. Let’s get more information

kubectl describe rs -l run=nginx

Warning  FailedCreate  41s  replicaset-controller  Error creating: pods "nginx-8586cf59-lbnjh" is forbidden: image policy webhook backend denied one or more images: Denied by default admission rule. Overridden by evaluation mode

The policy works!

Break Glass

There are special cases when you want to deploy without respect for the policy. It may happen during some emergencies.

Edit the nginx deployment
```
kubectl edit deployment/nginx
```
Add an annotation to the Pods metadata section (not Deployment metadata)
```
annotations:
  alpha.image-policy.k8s.io/break-glass: "true"
```

The Pod will be created after you exit the editor

kubectl get pods --watch

NAME                     READY   STATUS    RESTARTS   AGE
nginx-7bf86b9677-sxqxm   1/1     Running   0          4s

This is useful in emergency scenarios, typically for production related issues

Let's clean up for the next section. Remove the annotation we just added from the Pod metadata on the deployment
```
kubectl edit deployment/nginx
```

Whitelist Docker Library

What if I want to allow running all the images from the Docker official library?

Edit the manifests/policy.yaml and whitelist the Docker registry

All images from the library have the prefix registry.hub.docker.com/library/*

Add this under admissionWhitelistPatterns:
```
- namePattern: registry.hub.docker.com/library/*
```

Update the policy

gcloud beta container binauthz policy import manifests/policy.yaml

Test the Deployment with the nginx image

Because the controller doesn't know the nginx image comes from a standard registry, it just applies a regular expression filter!
Edit the Deployment and put the whole path to the image
```
image: registry.hub.docker.com/library/nginx
```
You should now see the container running
Clean up by deleting the nginx Deployment
```
kubectl delete deployment/nginx
```

Signing Images

In this exercise you will set up an attestor. It is an entity-like CI/CD system that signs the images with PGP and clusters will accept only the images signed by all the attestors defined in the policy.

Export configuration variables
```
export ATTESTOR=test-attestor
export NOTE_ID=test-attestor-note
```
Important: If it is not set already, set PROJECT_ID to your project ID.
- ATTESTOR is the id of the attestor
- NOTE_ID is the id of a note (image tag) in the container registry

Create text file note.json with the json payload

{
  "name": "projects/${PROJECT_ID}/notes/${NOTE_ID}",
  "attestation_authority": {
    "hint": {
      "human_readable_name": "Attestor Note"
    }
  }
}

Create a note

curl -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $(gcloud auth print-access-token)"  \
  --data-binary @note.json  \
  "https://containeranalysis.googleapis.com/v1beta1/projects/${PROJECT_ID}/notes/?noteId=${NOTE_ID}"

  {
    "name": "projects/<your-project-id>/notes/test-attestor-note",
    "kind": "ATTESTATION",
    "createTime": "2018-12-03T14:08:45.821912Z",
    "updateTime": "2018-12-03T14:08:45.821912Z",
    "attestationAuthority": {
      "hint": {
        "humanReadableName": "Attestor Note"
      }
    }
  }

Verify the note was created

curl \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  "https://containeranalysis.googleapis.com/v1beta1/projects/${PROJECT_ID}/notes/${NOTE_ID}"

  {
    "name": "projects/<your-project-id>/notes/test-attestor-note",
    "kind": "ATTESTATION",
    "createTime": "2018-12-03T14:08:45.821912Z",
    "updateTime": "2018-12-03T14:08:45.821912Z",
    "attestationAuthority": {
      "hint": {
        "humanReadableName": "Attestor Note"
      }
    }
  }

Create an attestor

gcloud beta container binauthz attestors create ${ATTESTOR} \
  --attestation-authority-note=${NOTE_ID} \
  --attestation-authority-note-project=${PROJECT_ID}

Verify the attestor was created
```
gcloud beta container binauthz attestors list
```
Note: Without a PGP keypair associated, the attestor can do nothing useful

Generate a PGP keypair

gpg --batch --gen-key <(
  cat <<- EOF
    Key-Type: RSA
    Key-Length: 2048
    Name-Real: "Test Attestor"
    Name-Email: "test-attestor@example.com"
    %commit
EOF
)

List the keys

gpg --list-keys "test-attestor@example.com"

gpg: checking the trustdb
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
pub   rsa2048 2018-12-03 [SCEA]
      XXXXX
uid           [ultimate] "Test Attestor" <"test-attestor@example.com">

Copy and save the public fingerprint (get from the output above)

FINGERPRINT=XXXXX
gpg --armor --export ${FINGERPRINT} > generated-key.pgp

Add the PGP key to the attestor

gcloud beta container binauthz attestors public-keys add \
    --attestor=${ATTESTOR} \
    --public-key-file=generated-key.pgp

Create a policy manifests/policy-attestor.yml

name: projects/${PROJECT_ID}/policy
admissionWhitelistPatterns:
- namePattern: gcr.io/google_containers/*
- namePattern: gcr.io/google-containers/*
- namePattern: k8s.gcr.io/*
- namePattern: gcr.io/stackdriver-agents/*
defaultAdmissionRule:
  evaluationMode: REQUIRE_ATTESTATION
  enforcementMode: ENFORCED_BLOCK_AND_AUDIT_LOG
  requireAttestationsBy:
    - projects/${PROJECT_ID}/attestors/${ATTESTOR}

Import the policy

gcloud beta container binauthz policy import manifests/policy-attestor.yaml

Test the policy
```
kubectl apply -f manifests/frontend.yaml
```
```
kubectl get pods
```
Note: You will get a message No resources found. This is okay as we still need to create attestation for the image to run it in the cluster.

Create an attestation

Important: You will have different project-id and image signature!

IMAGE_PATH="gcr.io/project-aleksey-zalesov/sample-k8s-app"
IMAGE_DIGEST="sha256:1dd9b5827cc3da6db2b89d8d0a4bc5d4c1caa98a5efba62da61bfd206ce03af3"

gcloud beta container binauthz create-signature-payload \
--artifact-url=${IMAGE_PATH}@${IMAGE_DIGEST} > signature.json

gpg \
  --local-user "test-attestor@example.com" \
  --armor \
  --sign signature.json
  --output signature.pgp \

gcloud beta container binauthz attestations create \
  --artifact-url="${IMAGE_PATH}@${IMAGE_DIGEST}" \
  --attestor="projects/${PROJECT_ID}/attestors/${ATTESTOR}" \
  --signature-file=signature.pgp \
  --pgp-key-fingerprint="${FINGERPRINT}"

Verify the attestation

gcloud beta container binauthz attestations list \
    --attestor=$ATTESTOR --attestor-project=$PROJECT_ID

Start the sample application again

Note: that this time you need to reference the image with the SHA256 sum instead of the tag, like this: https://gcr.io/project-altoros-workshop/sample-k8s-app:1dd9b5827cc3da6db2b89d8d0a4bc5d4c1caa98a5efba62da61bfd206ce03af3

This time the application should run correctly
Start the backend Pod
Create attestation for the MySQL image and run the db Pod by yourself

Clean Up

After doing these exercises, delete the cluster to free the resources

gcloud beta container clusters delete \
  --zone us-west2-b \
  gke-workshop-1

Next: Securing the Network and Container Runtime

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

13-secure-container-images.md

13-secure-container-images.md

Securing Container Images

Module Objectives

Enable GCR Vulnerability Scanning

View and Filter Vulnerability Occurrences

Redeploy the Cluster with Binary Authorization Enabled

Deny Policy

Break Glass

Whitelist Docker Library

Signing Images

Clean Up

Files

13-secure-container-images.md

Latest commit

History

13-secure-container-images.md

File metadata and controls

Securing Container Images

Module Objectives

Enable GCR Vulnerability Scanning

View and Filter Vulnerability Occurrences

Redeploy the Cluster with Binary Authorization Enabled

Deny Policy

Break Glass

Whitelist Docker Library

Signing Images

Clean Up