Error: writing blob: adding layer with blob "sha256:[hash]": processing tar file(write /usr/local/lib/python3.8/site-packages/azure/mgmt/web/v2018_02_01/models/__pycache__/_models_py3.cpython-38.pyc: no space left on device): exit status 1

[kashish19]

Issue Description

Describe your issue

Steps to reproduce the issue

Steps to reproduce the issue

1. dnf install podman
2. podman login
3. podman pull docker://internalserver.com/images/awx-ee:202304-21.11.0

Describe the results you received

Error: writing blob: adding layer with blob "sha256:[hash]": processing tar file(write /usr/local/lib/python3.8/site-packages/azure/mgmt/web/v2018_02_01/models/pycache/_models_py3.cpython-38.pyc: no space left on device): exit status 1

Describe the results you expected

podman pull should just work.

podman info output

host:
  arch: amd64
  buildahVersion: 1.29.0
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.6-1.module+el8.8.0+18098+9b44df5f.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.6, commit: 8c4ab5a095127ecc96ef8a9c885e0e1b14aeb11b'
  cpuUtilization:
    idlePercent: 98.58
    systemPercent: 0.68
    userPercent: 0.74
  cpus: 2
  distribution:
    distribution: '"rhel"'
    version: "8.8"
  eventLogger: file
  hostname: server.com
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 10124
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 10124
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 4.18.0-477.15.1.el8_8.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 1362644992
  memTotal: 3843649536
  networkBackend: cni
  ociRuntime:
    name: crun
    package: crun-1.8.4-2.module+el8.8.0+18669+fa5aca5a.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.4
      commit: 5a8fa99a5e41facba2eda4af12fa26313918805b
      rundir: /tmp/podman-run-10124/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /tmp/podman-run-10124/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_SYS_CHROOT,CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-2.module+el8.8.0+18060+3f21f2cc.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 4130836480
  swapTotal: 4294963200
  uptime: 151h 45m 18.00s (Approximately 6.29 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/awx/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/awx/.local/share/containers/storage
  graphRootAllocated: 2040373248
  graphRootUsed: 1057357824
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /tmp/containers-user-10124/containers
  transientStore: false
  volumePath: /home/awx/.local/share/containers/storage/volumes
version:
  APIVersion: 4.4.1
  Built: 1686839996
  BuiltTime: Thu Jun 15 14:39:56 2023
  GitCommit: ""
  GoVersion: go1.19.6
  Os: linux
  OsArch: linux/amd64
  Version: 4.4.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 1.8G 0 1.8G 0% /dev
tmpfs 1.8G 252K 1.8G 1% /dev/shm
tmpfs 1.8G 676K 1.8G 1% /run
tmpfs 1.8G 0 1.8G 0% /sys/fs/cgroup
/dev/mapper/vg_root-lv_root 52G 4.3G 45G 9% /
/dev/sda3 975M 68K 924M 1% /crashdump
/dev/mapper/vg_root-lv_var 25G 646M 23G 3% /var
/dev/mapper/vg_root-lv_tmp 2.0G 196K 1.8G 1% /tmp
/dev/sda1 975M 304M 621M 33% /boot
/dev/mapper/vg_root-lv_home 2.0G 1009M 820M 56% /home
/dev/mapper/vg_root-lv_var_log 9.8G 1.3G 8.0G 14% /var/log
/dev/mapper/vg_root-lv_var_log_audit 2.0G 17M 1.8G 1% /var/log/audit
tmpfs 367M 0 367M 0% /run/user/15521

[rhatdan]

It would seem that you are running out of disk space somewhere.

Is the image huge?

Can you pull the ubi9 image?

[kashish19]

It got fixed
I modified $HOME/.config/containers/storage.conf to:
[storage]
driver = "overlay"
graphroot = "/var/lib/containers/storage"

It solved this issue for me
Thanks @rhatdan for quick response