If you are using the overlay2 storage driver, you can place limits on the rootfs within a container but only if using an xfs backing filesystem (not ext4).
As a quick test of your Docker install, check your Docker storage driver and backing filesystem, then attempt to spin up a small alpine image with a 12Mb limit on the root filesystem “/”.
$ sudo docker info 2>&1 | grep -i "Storage Driver" -A1 Storage Driver: overlay2 Backing Filesystem extfs $ sudo docker run -it --storage-opt size=12m alpine:latest /bin/df -h | grep overlay docker: Error response from daemon: --storage-opt is supported only for overlay over xfs with 'pquota' mount option.
If your storage driver is overlay2 and backing filesystem is extfs (i.e. ext4), then you will get the error response shown above.
In this article I will show how to change the backing filesystem to xfs, which will then allow us to put storage limits on rootfs of the container.
Goal
Changing the backing filesystem is simply a matter of changing the filesystem type of “/var/lib/docker”. If your current “/var/lib/docker” directory is mounted on an ext4 filesystem, then you will need to soft link or mount a directory from an xfs filesystem to this path.
When successful, the output of “docker info” will show a backing filesystem of “xfs”, and we will be able to create a container with disk constraints on its rootfs “/” filesystem.
Prerequisites
For this article, I’m going to assume you have an XFS filesystem already mounted on your host. As a non-boot drive, you need to make sure that it’s entry in /etc/fstab has the “pquota” option so that xfs project quotas are enabled.
For example, the xfs mount and path I use in this article is mounted at “/home”, and here is the entry in /etc/fstab
UUID=.... /home xfs defaults,uquota,pquota 0 0
Before moving on, verify that your xfs mount has project quotas enabled in its options “prjquota”.
$ mount | grep xfs /dev/sda2 on /home type xfs (rw,relatime,attr2,inode64,usrquota,prjquota)
For xfs root partitions this needs to be set in the grub config as “rootflags=uquota,pquota”.
Finally, make sure the xfs format type is valid by checking that ftype option is set to 1.
xfs_info /home | grep ftype
xfs for the docker directory
NOTE: Performing this operation of completely recreating “/var/lib/docker” will give you a fresh docker installation, you will lose all existing images and containers. For images, the recommendation is to save all images and then reload afterwards (1,2). For containers, the ideal situation is they use persistent volumes and can simply be recreated, because export/import only saves the container filesystem and not the metadata required to run.
The first step is to stop the docker service, and move the current directory “/var/lib/docker”.
# stop docker service sudo systemctl stop docker # understand magnitude of size of /var/lib/docker sudo du -sh /var/lib/docker # check ownership and permissions of current /var/lib/docker sudo ls -ld /var/lib/docker # move existing content sudo mv /var/lib/docker /var/lib/docker.bk
On my system, my xfs filestore is mounted at “/home”, so I’m going to create a directory “/home/docker” and then link it to “/var/lib/docker”. Feel free to make adjustments based on your xfs mount point.
# /home is ext4 mount, use directory for new docker backing store mkdir -p /home/docker # match permissions originally on /var/lib/docker sudo chmod 711 /home/docker # link /var/lib/docker to xfs filestore sudo ln -s /home/docker /var/lib/docker # restart docker service sudo systemctl start docker
The Docker backing filesystem should now show ‘xfs’ with d_type set to true.
$ docker info 2>&1 | grep -i "Storage Driver" -A3 Storage Driver: overlay2 Backing Filesystem: xfs Supports d_type: true Native Overlay Diff: true
Spinning up a small alpine image should report back a root filesystem capped at 12Mb.
$ sudo docker run -it --storage-opt size=12m alpine:latest /bin/df -h | grep overlay overlay 12.0M 8.0K 12.0M 0% /
Conversely, bringing up an image without this storage option will show a root filesystem that matches the capacity of the total xfs filestore.
$ sudo docker run -it alpine:latest /bin/df -h | grep overlay overlay 60.5G 102.1M 60.4G 0% /
REFERENCES
docker, configure overlay2 storage driver
github, implementation of XFS quota for overlay2
redhat, history of docker storage drivers
stackoverflow, docker compose with storage-options
NOTES
save/load images is useful for backup/transfer
export/import of containers is not very useful because it creates image where metadata is lost (does not automatically rerun container)
# get rid of all stopped containers and dangling images
sudo docker system prune
# SAVE/LOAD images
sudo docker images
sudo docker run hello-world:latest
sudo docker save hello-world:latest | gzip > hello-world-latest.tar.gz
sudo docker system prune
sudo docker image rm -f hello-world:latest
zcat hello-world-latest.tar.gz | sudo docker load
sudo docker run hello-world:latest
# EXPORT/IMPORT of containers loses metadata
# so you cannot simply re-run
sudo docker ps -a
sudo docker logs -f ${containerId}
# export
sudo docker export ${containerId} | gzip > ${containerId}-imagename.tar.gz
sudo docker stop ${containerId}
sudo docker system prune
# import does not create container, it creates image
zcat ${containerId}-imagename.tar.gz | sudo docker import - restored-imagename