Patch Container Images using Trivy Operator and Copacetic

I recently saw a blog post that claimed to go over the steps to use the k8s Trivy Operator and Copacetic. It turned out to be LLM generated nonsense, but I thought it was an interesting idea, so I decided to make my own proof-of-concept.

Prerequistics

• Docker and Kubernetes experience
• The Trivy Operator running in a test k8s cluster installed via Helm
• A copa namespace in your cluster

Trivy Operator Configuration

I’m assuming you already have the Trivy Operator running. I’m going to give the instructions for making it work with this proof-of-concept, which assumed it is installed using Helm, So you do not have it installed via Helm, you will have to figure out how to make the same changes.

1. Set trivy.vulnType=os
   • As of this writing, copa does not support package vulnerabilities. By the time our copa install recieves the vulnerability report, we will no longer have the context of whether or not the vulnerability is os or package, so I’m setting it here to make sure they are all os vulnerabilies. An alternative workaround might be to configure copa to ignore errors when patching.
2. Set operator.webhookBroadcastURL=http://webhook.copa:9000/hooks/run-webhook

• This will trigger the webhook endpoint (that we will create later on) when there is a new vulnerability report.

Create Webhook Docker Image

I made the decision to have a webhook container trigger a Powershell script that runs the copa tool to patch the container image vulnerabilities provided in the request. So what I did to make that possible is somewhat specific to those decisions.

1. Make a new folder and create the following Dockerfile

FROM --platform=$BUILDPLATFORM cgr.dev/chainguard/wolfi-base AS builder

ARG TARGETARCH

ARG WEBHOOK_VERSION=2.8.2
ARG WEBHOOK_PACKAGE=webhook-linux-${TARGETARCH}.tar.gz
ARG WEBHOOK_PACKAGE_URL=https://github.com/adnanh/webhook/releases/download/${WEBHOOK_VERSION}/${WEBHOOK_PACKAGE}

ARG COPA_VERSION=0.9.0
ARG COPA_PACKAGE=copa_${COPA_VERSION}_linux_${TARGETARCH}.tar.gz
ARG COPA_PACKAGE_URL=https://github.com/project-copacetic/copacetic/releases/download/v${COPA_VERSION}/${COPA_PACKAGE}

WORKDIR /app

COPY ./hooks.yaml /app/
COPY ./script.ps1 /app/

WORKDIR /work
COPY ./copa-report /work/copa-report

RUN apk update && apk add curl \
    && curl -sSL ${WEBHOOK_PACKAGE_URL} -o /tmp/webhook.tar.gz \
    && tar zxf /tmp/webhook.tar.gz -C /work

RUN curl -sSL ${COPA_PACKAGE_URL} -o /tmp/copa.tar.gz \
    && tar zxf /tmp/copa.tar.gz -C /work

FROM docker:27.3.1-cli AS docker

FROM mcr.microsoft.com/powershell:7.4-azurelinux-3.0 AS powershell-amd64

FROM mcr.microsoft.com/powershell:7.4-azurelinux-3.0-arm64 AS powershell-arm64

FROM powershell-${TARGETARCH}

WORKDIR /app

COPY --from=docker /usr/local/bin/docker /usr/local/bin/
COPY --from=builder /work/*/webhook /app
COPY --from=builder /work/copa /app
COPY --from=builder /work/copa-report /usr/local/bin/
COPY --from=builder /app/ /app/

USER 65532:65532

EXPOSE 9000

ENTRYPOINT ["/app/webhook", "-hooks", "hooks.yaml", "-verbose"]

• The builder stage of the dockerfile downloads the webhook package that we will be using as well as the copa tool. It also copies in a couple other files that we will be creating.
• copa uses the docker cli, so we’re also grabbing that from the docker-cli container.
• My test cluster is using arm64, so I set up the Dockerfile so I would be able to build arm64 and amd64 images.
• The container will run the webhook package, along with the webhook configuration that we will define below.

2. Make the following hooks.yaml file in the same folder

- id: run-webhook
  execute-command: "/usr/bin/pwsh"
  pass-arguments-to-command:
  - source: string
    name: /app/script.ps1
  - source: "entire-payload"
  trigger-rule:
    match:
      type: value
      value: VulnerabilityReport
      parameter:
        source: payload
        name: kind
  include-command-output-in-response: true
  include-command-output-in-response-on-error: true

• The hooks.yaml properties as explained here
• I created a webook that will be available at the /hooks/run-webhook endpoint
• When it runs it executes powershell with the script.ps1 script and passes in the request payload as an argument to that script.
• Since The Trivy Operator will trigger the webhook on many different kinds of reports, I added a rule so that only VulnerabilityReport in the kind property of the json payload will execute the powershell

3. Make the following script.ps1 file in the same folder

$payload = $args[0] | ConvertFrom-Json

$report = @{
  apiVersion = 'v1alpha1'
  metadata = @{
    os = @{
      type = $payload.report.os.family
      version = $payload.report.os.name
    }
    config = @{
      arch = 'amd64'
    }
  }
  updates = $payload.report.vulnerabilities | ForEach-Object {
    @{
       name = $_.resource
       installedVersion = $_.installedVersion
       fixedVersion = $_.fixedVersion
       vulnerabilityID = $_.vulnerabilityID
    }
  }
}

$tempFile = New-TemporaryFile
$report | ConvertTo-Json | Out-File $tempFile

$sourceImage = "$($payload.report.registry.server)/$($payload.report.artifact.repository):$($payload.report.artifact.tag)"
/app/copa patch --timeout 5m -r $tempFile.FullName -s report  -i $sourceImage -a 'tcp://localhost:8888'
Remove-Item $tempFile

$destinationImage = "ttl.sh/$($payload.report.artifact.repository):$($payload.report.artifact.tag)-patched"
docker tag "$sourceImage-patched" $destinationImage
docker push $destinationImage

• Although the copa tool supports trivy reports by default, the Trivy Operator does not send the original Trivy report in the webhook payload. It sends the VulnerabilityReport custom resource. This means that my webhook needs to use the details from the payload we have to create a report that copa can consume.
• We don’t have all the same values that would be in the original trivy report, you can see that I hardcoded the architecture in the report to arm64 to match my cluster.
• The script then uses the newly created report can calls copa to patch the image. Once the image is patched it pushes the image.
• We’re going to use buildkit from a sidecar, so when calling copa, the address of buildkit is also included as an argument.
• When calling copa I have to specify a scanner plugin (-s) this is a workaround so that copa will consume something not in the format of a Trivy report. The creation of our dummy scanner plugin is below.

4. Make the following copa-report file in the same folder

#!/bin/bash

cat $1

• This is our dummy copa scanner plugin. copa will call this bash script and then use the output as the vulnerability report it will consume

5. Build the docker image, using the appropriate architecture and tag.

docker buildx build --platform linux/arm64 . -t ttl.sh/copa-webhook

6. Push the image to your registry

docker push ttl.sh/copa-webhook

Create and Apply Kubernetes Manifests

1. Create the Service for the webhook pod

apiVersion: v1
kind: Service
metadata:
  name: webhook
  namespace: copa
  labels:
    app: webhook
spec:
  selector:
    app: webhook
  ports:
    - name: http
      protocol: TCP
      port: 9000
      targetPort: http

2. Create the webhook Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: webhook
  namespace: copa
  labels:
    app: webhook
spec:
  selector:
    matchLabels:
      app: webhook
  template:
    metadata:
      labels:
        app: webhook
    spec:
      containers:
        - name: webhook
          image: ttl.sh/copa-webhook@sha256:da3ffc0e802defe772efc540c44dab14e334a7fd274374ddbbaa61980353af79
          ports:
          - name: http
            containerPort: 9000
          env:
          - name: DOCKER_HOST
            value: tcp://localhost:2375
        - name: buildkit
          image: moby/buildkit:v0.18.1-rootless
          args:
          - --addr
          - tcp://localhost:8888
          - --addr
          - unix:///run/user/1000/buildkit/buildkitd.sock
          - --oci-worker-no-process-sandbox
          securityContext:
            seccompProfile:
              type: Unconfined
            runAsUser: 1000
            runAsGroup: 1000
        - name: docker
          image: docker:27.3.1-dind-rootless
          securityContext:
            privileged: true
          env:
          - name: DOCKER_TLS_CERTDIR
            value: ""
          args:
          - --tls=false

• The webhook container is the image I built. I’m passing in the DOCKER_HOST environment variable in order to configure the docker cli we included to connect to the Docker-in-Docker container we are adding.
• The buildkit container is set to run on the port that script.ps1 is using for the custom buildkit address. Example for running buildkit as rootless in kubernetes is found here.
• And finally we havve the docker container that the docker cli in our webhook container is configured to connect to.

Results

With all the pieces in place and running, I can look at the webhook container logs and I see the webhook getting triggered and the output showing that images are getting patched and pushed.