[Ubuntu 24.04] podman broken on 20260726.254: uses /usr/bin/crun 1.14.1 instead of the crun 1.28 in /usr/local/bin

[shyney7]

Description

Every podman run and podman build fails on the ubuntu-24.04 image with:

Error: OCI runtime error: crun: unknown version specified

The image has podman 5.8.4 and crun 1.28, but podman is not using that crun. It resolves the runtime to /usr/bin/crun, which is still the Ubuntu archive build at 1.14.1:

$ podman --version
podman version 5.8.4
$ podman info --format '{{.Host.OCIRuntime.Path}}'
/usr/bin/crun
$ /usr/bin/crun --version
crun version 1.14.1
$ /usr/local/bin/crun --version
crun version 1.28
$ command -v crun
/usr/local/bin/crun

podman 5.8.4 writes "ociVersion":"1.2.1" into config.json. crun 1.14.1 rejects any version string that does not contain "1.0" (src/libcrun/container.c at the 1.14.1 tag):

if (def->ociVersion && strstr (def->ociVersion, "1.0") == NULL)
    return crun_make_error (err, 0, "unknown version specified");

so nothing starts. crun dropped that check in 1.16.

One thing that makes this annoying to debug: crun --version reports 1.28, because /usr/local/bin comes first on PATH. It looks like a current crun is installed while podman is actually running the old one.

Installing crun 1.28 over /usr/bin/crun fixes it. I confirmed that on a runner that had the broken image.

Platforms affected

• GitHub Actions - Standard Runners

Runner images affected

• Ubuntu 24.04

Image version and build link

Version: 20260726.254.1

https://github.com/actions/runner-images/releases/tag/ubuntu24%2F20260726.254

Is it regression?

Yes. Image 20260720.247.2 works. It has podman 4.9.3, which writes ociVersion 1.0.x, and crun 1.14.1 accepts that. The podman 4.9.3 to 5.8.4 bump is the only container change listed in the release notes for 20260726.254.

Because the rollout is staged, jobs currently pass or fail depending on which image they get. The same commit can pass and fail minutes apart, which is how this first showed up for me.

ubuntu-26.04 is not affected. It has podman 5.7.0 with crun 1.21 at /usr/bin/crun.

Expected behavior

podman runs containers on the ubuntu-24.04 image.

Actual behavior

Every container fails to start with unknown version specified, because podman uses /usr/bin/crun 1.14.1 rather than the 1.28 shipped in /usr/local/bin.

Repro steps

jobs:
  repro:
    runs-on: ubuntu-24.04
    steps:
      - run: |
          podman --version
          podman info --format '{{.Host.OCIRuntime.Path}}'
          /usr/bin/crun --version
          /usr/local/bin/crun --version
          podman run --rm docker.io/library/alpine echo hello

This fails when the job lands on 20260726.254.1. The Image Version: line in the "Set up job" group shows which image the run actually got.

[pierreluc16]

Podman is broken on ubuntu 22.04 and ubuntu 24.04 on image 20260726

ubuntu 22.04

ubuntu 24.04

Workflow to reproduce

name: Podman Repro

on:
  push:
  pull_request:
  workflow_dispatch:

jobs:
  repro:
    strategy:
      matrix:
        os: [ubuntu-22.04, ubuntu-24.04]
    runs-on: ${{ matrix.os }}
    continue-on-error: true
    steps:
      - run: |
          podman --version
          podman info --format '{{.Host.OCIRuntime.Path}}'
          /usr/bin/crun --version || true
          /usr/local/bin/crun --version || true
          podman run --rm docker.io/library/alpine echo hello

[idrassi]

A minimal workaround for podman run is to resolve crun through PATH and pass its absolute path explicitly:

OCI_RUNTIME=$(command -v crun)
podman --runtime "$OCI_RUNTIME" run ...

On the broken runner, this selects /usr/local/bin/crun 1.28 instead of the stale /usr/bin/crun 1.14.1 selected by Podman’s built-in runtime search. It also remains compatible with previous runner images.

Changing PATH alone is insufficient because Podman doesn't use shell command resolution when selecting its OCI runtime.

This is the approach I implemented in PR openssl/openssl#32115 since OpenSSL’s linux-x86 CI jobs are also affected.