Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] k3d image import fails when using containerd for images in Docker Desktop #1538

Open
rrmistry opened this issue Dec 18, 2024 · 3 comments
Open

[BUG] k3d image import fails when using containerd for images in Docker Desktop #1538

rrmistry opened this issue Dec 18, 2024 · 3 comments
Labels
bug Something isn't working

Comments

@rrmistry
Copy link

What did you do

  • Where is this setting in Docker Desktop?
    image

  • How was the cluster created?

    • k3d cluster create demo
      Console output for reference 
      k3d cluster create demo  
INFO[0000] Prep: Network                                
INFO[0000] Created network 'k3d-demo'                   
INFO[0000] Created image volume k3d-demo-images         
INFO[0000] Starting new tools node...                   
INFO[0000] Pulling image 'ghcr.io/k3d-io/k3d-tools:5.7.3' 
INFO[0001] Creating node 'k3d-demo-server-0'            
INFO[0001] Starting node 'k3d-demo-tools'               
INFO[0001] Pulling image 'docker.io/rancher/k3s:v1.30.3-k3s1' 
INFO[0002] Creating LoadBalancer 'k3d-demo-serverlb'    
INFO[0002] Pulling image 'ghcr.io/k3d-io/k3d-proxy:5.7.3' 
INFO[0003] Using the k3d-tools node to gather environment information 
INFO[0003] Starting new tools node...                   
INFO[0003] Starting node 'k3d-demo-tools'               
INFO[0004] Starting cluster 'demo'                      
INFO[0004] Starting servers...                          
INFO[0004] Starting node 'k3d-demo-server-0'            
INFO[0006] All agents already running.                  
INFO[0006] Starting helpers...                          
INFO[0006] Starting node 'k3d-demo-serverlb'            
INFO[0013] Injecting records for hostAliases (incl. host.k3d.internal) and for 3 network members into CoreDNS configmap... 
INFO[0015] Cluster 'demo' created successfully!         
INFO[0015] You can now use it like this:                
kubectl cluster-info

  • What did you do afterwards?

    • Pull any official image
      docker pull redis:7.4.1
    • Import this image into the demo cluster
      k3d image import --cluster=demo redis:7.4.1
      Console output for reference 
      k3d image import --cluster=demo redis:7.4.1
INFO[0000] Importing image(s) into cluster 'demo'       
INFO[0000] Saving 1 image(s) from runtime...            
INFO[0001] Importing images into nodes...               
INFO[0001] Importing images from tarball '/k3d/images/k3d-demo-images-20241218155459.tar' into node 'k3d-demo-server-0'... 
ERRO[0002] failed to import images in node 'k3d-demo-server-0': Exec process in node 'k3d-demo-server-0' failed with exit code '1': Logs from failed access process:
 tr: content digest sha256:fa84a527b1808ee5ad2778e5d89aae791e7040b5c303f1d45569a9ca8d5b61ee: not found
INFO[0002] Removing the tarball(s) from image volume... 
INFO[0003] Removing k3d-tools node...                   
INFO[0003] Successfully imported image(s)               
INFO[0003] Successfully imported 1 image(s) into 1 cluster(s)
    • Check the images available in the k3d server-0 node using node's local contained instance:
      docker exec k3d-demo-server-0 ctr images list --quiet
      Console output for reference 
      docker.io/rancher/klipper-helm:v0.8.4-build20240523
docker.io/rancher/klipper-helm@sha256:c2fd922a9a361ac5ec7ef225a46aaaad1e79ec3acc3cf176f60cd09a11683dd5
docker.io/rancher/klipper-lb:v0.4.7
docker.io/rancher/klipper-lb@sha256:558dcf96bf0800d9977ef46dca18411752618cd9dd06daeb99460c0a301d0a60
docker.io/rancher/local-path-provisioner:v0.0.28
docker.io/rancher/local-path-provisioner@sha256:f77fa34dd9d000daf6fd6029bf05de6b351b4fe4287cbc0ad2abb11a91df2ced
docker.io/rancher/mirrored-coredns-coredns:1.10.1
docker.io/rancher/mirrored-coredns-coredns@sha256:a11fafae1f8037cbbd66c5afa40ba2423936b72b4fd50a7034a7e8b955163594
docker.io/rancher/mirrored-library-traefik:2.10.7
docker.io/rancher/mirrored-library-traefik@sha256:606c4c924d9edd6d028a010c8f173dceb34046ed64fabdbce9ff29b2cf2b3042
docker.io/rancher/mirrored-metrics-server:v0.7.0
docker.io/rancher/mirrored-metrics-server@sha256:20b8b36f8cac9e25aa2a0ff35147b13643bfec603e7e7480886632330a3bbc59
docker.io/rancher/mirrored-pause:3.6
docker.io/rancher/mirrored-pause@sha256:74c4244427b7312c5b901fe0f67cbc53683d06f4f24c6faee65d4182bf0fa893
sha256:31fdd40fc908c2bca850f3d521eb9e62518f5c924f979f5f640d5f29f8c3a448
sha256:5cd7991a1c728f68af15f736e5144584c4666c197e3d772cc1495b1c8e2cff9f
sha256:7d46a07936af93fcce097459055f93ab07331509aa55f4a2a90d95a3ace1850e
sha256:97e04611ad43405a2e5863ae17c6f1bc9181bdefdaa78627c432ef754a4eb108
sha256:ad5487c25b85fd289ddcaf23f148b09d7916e74ef6954c00d44e859629a2d275
sha256:afd63e3b5878f73dfbecfc47a77350dc5aee49ce02e06f6914a0ac913f0e5cc8
sha256:bd1c9d32f4f563008741d68a301e0d476a46d6ab173eb7db725b0384e2ab2995

What did you expect to happen

That the image is available on the k3s docker node.

Disabling containerd in Docker Desktop and re-running the same steps shows the expected output:

> docker exec k3d-demo-server-0 ctr images list --quiet
docker.io/library/redis:7.4.1
...
...
Console output for reference 
docker.io/library/redis:7.4.1
docker.io/rancher/klipper-helm:v0.8.4-build20240523
docker.io/rancher/klipper-helm@sha256:c2fd922a9a361ac5ec7ef225a46aaaad1e79ec3acc3cf176f60cd09a11683dd5
docker.io/rancher/klipper-lb:v0.4.7
docker.io/rancher/klipper-lb@sha256:558dcf96bf0800d9977ef46dca18411752618cd9dd06daeb99460c0a301d0a60
docker.io/rancher/local-path-provisioner:v0.0.28
docker.io/rancher/local-path-provisioner@sha256:f77fa34dd9d000daf6fd6029bf05de6b351b4fe4287cbc0ad2abb11a91df2ced
docker.io/rancher/mirrored-coredns-coredns:1.10.1
docker.io/rancher/mirrored-coredns-coredns@sha256:a11fafae1f8037cbbd66c5afa40ba2423936b72b4fd50a7034a7e8b955163594
docker.io/rancher/mirrored-library-traefik:2.10.7
docker.io/rancher/mirrored-library-traefik@sha256:606c4c924d9edd6d028a010c8f173dceb34046ed64fabdbce9ff29b2cf2b3042
docker.io/rancher/mirrored-metrics-server:v0.7.0
docker.io/rancher/mirrored-metrics-server@sha256:20b8b36f8cac9e25aa2a0ff35147b13643bfec603e7e7480886632330a3bbc59
docker.io/rancher/mirrored-pause:3.6
docker.io/rancher/mirrored-pause@sha256:74c4244427b7312c5b901fe0f67cbc53683d06f4f24c6faee65d4182bf0fa893
sha256:31fdd40fc908c2bca850f3d521eb9e62518f5c924f979f5f640d5f29f8c3a448
sha256:5cd7991a1c728f68af15f736e5144584c4666c197e3d772cc1495b1c8e2cff9f
sha256:7d46a07936af93fcce097459055f93ab07331509aa55f4a2a90d95a3ace1850e
sha256:97e04611ad43405a2e5863ae17c6f1bc9181bdefdaa78627c432ef754a4eb108
sha256:9e1ccee11dad8887f693f60ae28e93f54c7e33beb3dcc5781a9340418d33b598
sha256:ad5487c25b85fd289ddcaf23f148b09d7916e74ef6954c00d44e859629a2d275
sha256:afd63e3b5878f73dfbecfc47a77350dc5aee49ce02e06f6914a0ac913f0e5cc8
sha256:bd1c9d32f4f563008741d68a301e0d476a46d6ab173eb7db725b0384e2ab2995

Which OS & Architecture

  • output of k3d runtime-info (with containerd)

    k3d runtime-info
arch: aarch64
cgroupdriver: cgroupfs
cgroupversion: "2"
endpoint: /var/run/docker.sock
filesystem: UNKNOWN
infoname: docker-desktop
name: docker
os: Docker Desktop
ostype: linux
version: 27.3.1

  • output of k3d runtime-info (without containerd)

    arch: aarch64
cgroupdriver: cgroupfs
cgroupversion: "2"
endpoint: /var/run/docker.sock
filesystem: extfs
infoname: docker-desktop
name: docker
os: Docker Desktop
ostype: linux
version: 27.3.1

Which version of k3d

  • output of k3d version 
    k3d version v5.7.3
k3s version v1.30.3-k3s1 (default)

Which version of docker

  • output of docker version

    Console output for reference 
    > docker version
Client:
 Version:           27.3.1
 API version:       1.47
 Go version:        go1.22.7
 Git commit:        ce12230
 Built:             Fri Sep 20 11:38:18 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

    Server: Docker Desktop 4.36.0 (175267)
    
Engine:
    
Version:          27.3.1
    
API version:      1.47 (minimum version 1.24)
    
Go version:       go1.22.7
    
Git commit:       41ca978
    
Built:            Fri Sep 20 11:41:19 2024
    
OS/Arch:          linux/arm64
    
Experimental:     false
    
containerd:
    
Version:          1.7.21
    
GitCommit:        472731909fa34bd7bc9c087e4c27943f9835f111
    
runc:
    
Version:          1.1.13
    
GitCommit:        v1.1.13-0-g58aa920
    
docker-init:
    
Version:          0.19.0
    
GitCommit:        de40ad0

  • output of docker info

    Console output for reference 
    Client:
 Version:    27.3.1
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  ai: Ask Gordon - Docker Agent (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/rohit/.docker/cli-plugins/docker-ai
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.18.0-desktop.2
    Path:     /Users/rohit/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.30.3-desktop.1
    Path:     /Users/rohit/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.37
    Path:     /Users/rohit/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Alpha) (Docker Inc.)
    Version:  v0.0.15
    Path:     /Users/rohit/.docker/cli-plugins/docker-desktop
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/rohit/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.27
    Path:     /Users/rohit/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /Users/rohit/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     /Users/rohit/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/rohit/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.15.0
    Path:     /Users/rohit/.docker/cli-plugins/docker-scout

    Server:
    
Containers: 3
    
Running: 3
    
Paused: 0
    
Stopped: 0
    
Images: 23
    
Server Version: 27.3.1
    
Storage Driver: overlayfs
    
driver-type: io.containerd.snapshotter.v1
    
Logging Driver: json-file
    
Cgroup Driver: cgroupfs
    
Cgroup Version: 2
    
Plugins:
    
Volume: local
    
Network: bridge host ipvlan macvlan null overlay
    
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
    
Swarm: inactive
    
Runtimes: io.containerd.runc.v2 runc
    
Default Runtime: runc
    
Init Binary: docker-init
    
containerd version: 472731909fa34bd7bc9c087e4c27943f9835f111
    
runc version: v1.1.13-0-g58aa920
    
init version: de40ad0
    
Security Options:
    
seccomp
    
Profile: unconfined
    
cgroupns
    
Kernel Version: 6.10.14-linuxkit
    
Operating System: Docker Desktop
    
OSType: linux
    
Architecture: aarch64
    
CPUs: 9
    
Total Memory: 58.75GiB
    
Name: docker-desktop
    
ID: 6c33f800-b50a-4ea8-b8a6-085b19a86a85
    
Docker Root Dir: /var/lib/docker
    
Debug Mode: false
    
HTTP Proxy: http.docker.internal:3128
    
HTTPS Proxy: http.docker.internal:3128
    
No Proxy: hubproxy.docker.internal
    
Labels:
    
com.docker.desktop.address=unix:///Users/rohit/Library/Containers/com.docker.docker/Data/docker-cli.sock
    
Experimental: false
    
Insecure Registries:
    
hubproxy.docker.internal:5555
    
127.0.0.0/8
    
Live Restore Enabled: false
    

    WARNING: daemon is not using the default seccomp profile

@rrmistry rrmistry added the bug Something isn't working label Dec 18, 2024
@rrmistry
Copy link
Author

I just noticed that k3d latest version is v5.7.5.

I confirmed the same behaviour with k3d v5.7.5 when using containerd in Docker Desktop.

Scripts summary:

k3d cluster create demo
docker pull redis:7.4.1
k3d image import --cluster=demo redis:7.4.1
docker exec k3d-demo-server-0 ctr images list --quiet

Expected docker.io/library/redis:7.4.1 image to be available in the output of ctr images list but it is not.

@rrmistry rrmistry mentioned this issue Dec 19, 2024
Open
@rrmistry
Copy link
Author

I've also tried all 3 modes, but it behaves the same with containerd image storage:

  • k3d image import --mode="auto" ...
  • k3d image import --mode="direct" ...
  • k3d image import --mode="tools-node" ...

@parrott-nexthink
Copy link

Confirmed ... I see this issue as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rrmistry @parrott-nexthink