Stage 4 · Provision
Image Internals & Advanced Builds
Distroless Debugging
Debug minimal images with ephemeral debug containers, busybox sidecars, copied certificates, and static binaries.
The Debugging Challenge
Distroless and scratch images have no shell, no package manager, and no debugging tools. When something goes wrong, you cannot docker exec sh or install tools. This requires alternative debugging strategies that do not compromise the minimal image's security benefits.
Ephemeral Debug Containers
# Attach a debug container to a running pod
kubectl debug -it mypod --image=busybox --target=mycontainer
# This shares the process namespace of mycontainer
# You can see its processes, files, and network
# Use for non-Kubernetes containers
docker run --rm -it --pid=container:mycontainer \
--net=container:mycontainer \
busybox shEphemeral debug containers share the target container's namespaces. You can see its processes, files, and network without modifying the production image. In Kubernetes, kubectl debug makes this easy.
BusyBox Sidecar
apiVersion: v1
kind: Pod
metadata:
name: myapp
spec:
containers:
- name: myapp
image: gcr.io/distroless/static
# Your application container
- name: debug
image: busybox
command: ["sleep", "infinity"]
volumeMounts:
- name: shared-data
mountPath: /data
volumes:
- name: shared-data
emptyDir: {}The debug sidecar runs alongside your application. You can exec into it to inspect shared volumes, check logs, or run diagnostics. The application container remains unmodified.
Static Debugging Tools
# Build static debugging tools
FROM golang:1.22-alpine AS builder
# Build a static curl
RUN apk add --no-cache gcc musl-dev
RUN CGO_ENABLED=0 go build -o /curl https://github.com/curl/curl
# Or copy pre-built static binaries
FROM scratch
COPY --from=builder /curl /usr/local/bin/curl
COPY --from=builder /etc/ssl/certs /etc/ssl/certs
ENTRYPOINT ["/usr/local/bin/curl"]Static binaries compiled with CGO_ENABLED=0 work on scratch images. You can copy individual tools (curl, ping, nslookup) into your debug image without adding a full shell.
Copied Certificates
FROM golang:1.22-alpine AS builder
WORKDIR /app
COPY . .
RUN CGO_ENABLED=0 go build -o server .
FROM gcr.io/distroless/static
# Copy CA certificates for HTTPS
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
# Copy timezone data
COPY --from=builder /usr/share/zoneinfo /usr/share/zoneinfo
COPY --from=builder /app/server /server
ENTRYPOINT ["/server"]Distroless images include CA certificates, but if you need additional certificates or timezone data, copy them from the builder stage. This keeps the image minimal while adding necessary files.
Debug Image Variants
# Stage 1: Build
FROM golang:1.22-alpine AS builder
WORKDIR /app
COPY . .
RUN CGO_ENABLED=0 go build -o server .
# Stage 2: Production (scratch)
FROM scratch AS production
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder /app/server /server
ENTRYPOINT ["/server"]
# Stage 3: Debug (busybox)
FROM busybox AS debug
COPY --from=builder /etc/ssl/certs /etc/ssl/certs
COPY --from=builder /app/server /server
ENTRYPOINT ["/server"]Build the production image: docker build --target production -t myapp:1.0 . Build the debug image: docker build --target debug -t myapp:1.0-debug . Use the debug variant during development, production in production.
In Kubernetes, kubectl debug attaches an ephemeral container to a running pod without modifying the production image: kubectl debug -it mypod --image=busybox --target=mycontainer. This is the cleanest debugging approach for distroless containers.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.