ARM64 Kubernetes: The QEMU Problem with Go Binaries#

If you run Kubernetes on Apple Silicon (M1/M2/M3/M4) via minikube with the Docker driver, you will eventually try to run an amd64-only container image. For most software this works through QEMU emulation. For Go binaries, it crashes hard.

The Problem#

Go’s garbage collector uses a lock-free stack (lfstack) that packs pointers with counter bits in the high bits of a 64-bit integer. QEMU’s user-mode address translation changes the effective address space layout, which breaks this packing assumption. The result:

runtime: lfstack.push invalid packing: node 0xffff8b410100 cnt 0x1 packed 0x8b41010000000001 -> node 0xffff00008b410100
fatal error: lfstack.push

goroutine 1 [running]:
runtime.throw({0x2bfaa07, 0x14})

This is not a random flake. It is a fundamental incompatibility between QEMU user-mode emulation and Go’s runtime memory model on ARM64 hosts. Every Go binary running under QEMU on ARM64 is affected.

Why Rosetta Does Not Help in Minikube#

Docker Desktop on macOS can use Apple’s Rosetta 2 for amd64 emulation, which handles Go binaries correctly. However, minikube with the Docker driver runs its own containerd inside the minikube VM. That containerd does not use Rosetta – it falls back to QEMU. So even if Docker Desktop has Rosetta enabled, images pulled inside minikube still crash.

Diagnosing the Issue#

When a pod is crash-looping with no obvious application error, check the logs for the lfstack signature:

kubectl logs <pod-name> -n <namespace> --previous
# Look for: "runtime: lfstack.push invalid packing"

Confirm the image architecture:

# Check what architecture an image was built for
docker manifest inspect mattermost/mattermost-team-edition:latest | jq '.manifests[].platform'
# If you only see "amd64", you have the problem

Solution 1: Build a Native ARM64 Image#

Many projects publish ARM64 binaries even when they do not publish ARM64 Docker images. Mattermost is a prime example – there is no official ARM64 image, but ARM64 binary tarballs are available:

https://releases.mattermost.com/10.4.2/mattermost-team-10.4.2-linux-arm64.tar.gz

Create a minimal Dockerfile that uses the native binary:

FROM ubuntu:22.04

RUN apt-get update && apt-get install -y ca-certificates curl && rm -rf /var/lib/apt/lists/*

ARG MM_VERSION=10.4.2
RUN curl -L https://releases.mattermost.com/${MM_VERSION}/mattermost-team-${MM_VERSION}-linux-arm64.tar.gz \
    | tar xz -C /opt

RUN mkdir -p /opt/mattermost/data /opt/mattermost/logs /opt/mattermost/config \
    && adduser --system --group mattermost \
    && chown -R mattermost:mattermost /opt/mattermost

USER mattermost
WORKDIR /opt/mattermost

EXPOSE 8065
ENTRYPOINT ["/opt/mattermost/bin/mattermost"]

Build it directly inside minikube’s Docker daemon so you do not need a registry:

# Point your shell's Docker client at minikube's Docker
eval $(minikube docker-env)

# Build the image -- it is now available inside minikube
docker build -t mattermost-arm64:10.4.2 ./docker/mattermost-arm64/

Then reference the local image in your Helm values with imagePullPolicy: Never:

image:
  repository: mattermost-arm64
  tag: "10.4.2"
  pullPolicy: Never

Solution 2: QEMU for Non-Go amd64 Images#

If you need to run an amd64 image that is not built with Go (Python, Node.js, C/C++ applications), QEMU works fine. Install the binfmt handlers:

docker run --privileged --rm tonistiigi/binfmt --install amd64

This registers QEMU as the handler for amd64 binaries. It persists across Docker restarts but not across host reboots.

Decision Flowchart#

When you encounter an amd64-only image on ARM64 Kubernetes:

  1. Is the application written in Go? Check the Dockerfile or project repository.
  2. If yes: Do NOT use QEMU. Find or build a native ARM64 image.
  3. If no: Install QEMU binfmt support and run the amd64 image directly.
  4. If unsure: Try running it. If you see the lfstack.push crash, it is Go – switch to a native build.

Key Takeaways#

  • The lfstack.push invalid packing error is always QEMU + Go on ARM64. There is no workaround short of native binaries.
  • eval $(minikube docker-env) lets you build images directly into minikube without pushing to a registry.
  • Check for ARM64 binary releases before assuming you need to cross-compile from source.
  • QEMU is perfectly fine for non-Go workloads – do not avoid it entirely, just for Go.