`minikube image build` with docker runtime cannot be configured to use buildkit

[briandealwis]

minikube image build --build-env DOCKER_BUILDKIT=1 should enable Buildkit

Steps to reproduce the issue:

1. minikube start on Apple M1
2. minikube image build --alsologtostderr --build-env DOCKER_BUILDKIT=1 --build-opt help . should include buildkit-related options like --platform but still shows the non-buildkit option.

minikube image build --alsologtostderr --build-env DOCKER_BUILDKIT=1 --build-opt help .

$ minikube image build --alsologtostderr --build-env DOCKER_BUILDKIT=1 --build-opt help .
I0908 11:49:16.897210   18832 out.go:286] Setting OutFile to fd 1 ...
I0908 11:49:16.897579   18832 out.go:338] isatty.IsTerminal(1) = true
I0908 11:49:16.897586   18832 out.go:299] Setting ErrFile to fd 2...
I0908 11:49:16.897590   18832 out.go:338] isatty.IsTerminal(2) = true
I0908 11:49:16.897821   18832 root.go:312] Updating PATH: /Users/bdealwis/.minikube/bin
I0908 11:49:16.903470   18832 cli_runner.go:115] Run: docker container inspect minikube --format={{.State.Status}}
I0908 11:49:17.168825   18832 ssh_runner.go:149] Run: systemctl --version
I0908 11:49:17.168953   18832 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0908 11:49:17.309323   18832 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49626 SSHKeyPath:/Users/bdealwis/.minikube/machines/minikube/id_rsa Username:docker}
I0908 11:49:17.408634   18832 build_images.go:137] Building image from path: /var/folders/hj/xkqnc3sn77jgn8jdgpcmt86400kl65/T/build.260211351.tar
I0908 11:49:17.408753   18832 ssh_runner.go:149] Run: sudo mkdir -p /var/lib/minikube/build
I0908 11:49:17.416433   18832 ssh_runner.go:149] Run: stat -c "%s %y" /var/lib/minikube/build/build.260211351.tar
I0908 11:49:17.420427   18832 ssh_runner.go:306] existence check for /var/lib/minikube/build/build.260211351.tar: stat -c "%s %y" /var/lib/minikube/build/build.260211351.tar: Process exited with status 1
stdout:

stderr:
stat: cannot stat '/var/lib/minikube/build/build.260211351.tar': No such file or directory
I0908 11:49:17.420461   18832 ssh_runner.go:316] scp /var/folders/hj/xkqnc3sn77jgn8jdgpcmt86400kl65/T/build.260211351.tar --> /var/lib/minikube/build/build.260211351.tar (6656 bytes)
I0908 11:49:17.436936   18832 ssh_runner.go:149] Run: sudo mkdir -p /var/lib/minikube/build/build.260211351
I0908 11:49:17.443000   18832 ssh_runner.go:149] Run: sudo tar -C /var/lib/minikube/build/build.260211351 -xf /var/lib/minikube/build/build.260211351.tar
I0908 11:49:17.450391   18832 docker.go:249] Building image: /var/lib/minikube/build/build.260211351
I0908 11:49:17.443000   18832 ssh_runner.go:149] Run: sudo tar -C /var/lib/minikube/build/build.260211351 -xf /var/lib/minikube/build/build.260211351.tar
I0908 11:49:17.450391   18832 docker.go:249] Building image: /var/lib/minikube/build/build.260211351
I0908 11:49:17.450560   18832 ssh_runner.go:149] Run: docker build /var/lib/minikube/build/build.260211351 --help

Usage:  docker build [OPTIONS] PATH | URL | -

Build an image from a Dockerfile

Options:
      --add-host list           Add a custom host-to-IP mapping (host:ip)
      --build-arg list          Set build-time variables
      --cache-from strings      Images to consider as cache sources
      --cgroup-parent string    Optional parent cgroup for the container
      --compress                Compress the build context using gzip
      --cpu-period int          Limit the CPU CFS (Completely Fair
                                Scheduler) period
      --cpu-quota int           Limit the CPU CFS (Completely Fair
                                Scheduler) quota
  -c, --cpu-shares int          CPU shares (relative weight)
      --cpuset-cpus string      CPUs in which to allow execution (0-3, 0,1)
      --cpuset-mems string      MEMs in which to allow execution (0-3, 0,1)
      --disable-content-trust   Skip image verification (default true)
  -f, --file string             Name of the Dockerfile (Default is
                                'PATH/Dockerfile')
      --force-rm                Always remove intermediate containers
      --iidfile string          Write the image ID to the file
      --isolation string        Container isolation technology
      --label list              Set metadata for an image
  -m, --memory bytes            Memory limit
      --memory-swap bytes       Swap limit equal to memory plus swap:
                                '-1' to enable unlimited swap
      --network string          Set the networking mode for the RUN
                                instructions during build (default "default")
      --no-cache                Do not use cache when building the image
      --pull                    Always attempt to pull a newer version of
                                the image
  -q, --quiet                   Suppress the build output and print image
                                ID on success
      --rm                      Remove intermediate containers after a
                                successful build (default true)
      --security-opt strings    Security options
      --shm-size bytes          Size of /dev/shm
  -t, --tag list                Name and optionally a tag in the
                                'name:tag' format
      --target string           Set the target build stage to build.
      --ulimit ulimit           Ulimit options (default [])

minikube ssh -- 'DOCKER_BUILDKIT=1 docker build --help'

Usage:  docker build [OPTIONS] PATH | URL | -

Build an image from a Dockerfile

Options:
      --add-host list           Add a custom host-to-IP mapping (host:ip)
      --build-arg list          Set build-time variables
      --cache-from strings      Images to consider as cache sources
      --disable-content-trust   Skip image verification (default true)
  -f, --file string             Name of the Dockerfile (Default is
                                'PATH/Dockerfile')
      --iidfile string          Write the image ID to the file
      --isolation string        Container isolation technology
      --label list              Set metadata for an image
      --network string          Set the networking mode for the RUN
                                instructions during build (default "default")
      --no-cache                Do not use cache when building the image
  -o, --output stringArray      Output destination (format:
                                type=local,dest=path)
      --platform string         Set platform if server is multi-platform
                                capable
      --progress string         Set type of progress output (auto, plain,
                                tty). Use plain to show container output
                                (default "auto")
      --pull                    Always attempt to pull a newer version of
                                the image
  -q, --quiet                   Suppress the build output and print image
                                ID on success
      --secret stringArray      Secret file to expose to the build (only
                                if BuildKit enabled):
                                id=mysecret,src=/local/secret
      --ssh stringArray         SSH agent socket or keys to expose to the
                                build (only if BuildKit enabled) (format:
                                default|<id>[=<socket>|<key>[,<key>]])
  -t, --tag list                Name and optionally a tag in the
                                'name:tag' format
      --target string           Set the target build stage to build.

[afbjorklund]

It seems like RunCmd is broken for the ssh_runner, since cmd.Env was only implemented for the kic_runner.

        err = teeSSH(sess, shellquote.Join(cmd.Args...), outb, errb)

vs.

        for _, env := range cmd.Env {
                args = append(args, "-e", env)
        }

        args = append(
                args,
                cmd.Args...,
        )

Also don't see how it was supposed to run, since it adds the current environment - not only the extras...

[afbjorklund]

The command needs to be prefixed with env, and then it works as intended:

I0908 19:59:55.058297   62796 docker.go:262] Building image: /var/lib/minikube/build/build.146515009
I0908 19:59:55.058378   62796 ssh_runner.go:152] Run: docker build /var/lib/minikube/build/build.146515009 --help
I0908 19:59:55.058394   62796 ssh_runner.go:153] Env: [DOCKER_BUILDKIT=1]
I0908 19:59:55.058738   62796 ssh_runner.go:193] SSH: [env DOCKER_BUILDKIT=1 docker build /var/lib/minikube/build/build.146515009 --help]

At least once cruntime and maybe others using cmd.Env are fixed as well... 😔

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten