Slimming down your VMs. A guide to thin provisioning with Proxmox
If you’ve been running your Proxmox instance for a while it’s more than possible that you’ve at some point observed that the amount of free storage might not increase as you delete data inside your VMs. In fact, the amount of free storage might not increase at all unless you remove the VM disks all together.
The answer to this problem is called ‘thin provisioning. The concept is simple, instead of allocating the entire VM disk on the underlying storage, thin provisioning ensures allocation of space in a flexible manner, ensuring only the minimum required space is allocated at any given time.
It seems simple in both theory and practice. Simply enable “thin provisioning” in the Proxmox storage configuration UI and you’re good to go right? Well, not so fast.
There are a number of pre-requisites required in order for thin provisioning to actually work. You have to ensure that discarding of blocks work all the way from the VM to the underlying file system.
In this example we will assume you’re running Proxmox 6.3 (based on Debian Buster) and your VM is also running Debian 10 (Buster). The local storage used for the VM will be a ZFS pool running on spinning disks, though everything in this article applies to SSD’s as well.
Before we begin, let’s illustrate the problem.
Single out a VM and run zfs list on your Proxmox host and look at the associated disk. In the following example you’ll notice how the output indicates 15.0 GB being used on the file system. In your case you might even see that your vm disk is fully allocated even though you have plenty of free space left on your VM.
Next, allocate a 4GB file on the VM. dd if=/dev/urandom of=bigfile bs=4096 count=1000000 should do the trick.
Running zfs list again shows that the amount of used space has grown by about 4G as expected. A bit less in this case since I’m utilizing compression on the ZFS pool.
Now, remove the file from the VM. You and notice how the output from zfs list is unchanged. The VM disk is still consuming 18.3 GB of our ZFS pool even though we deleted the file from the VM. In fact, disk space has even increased a bit. Probably due to some other IO activity on the VM that went on in the meantime.
Somehow, we need to get the VM to tell the underlying file system to free up the space for us. The fstrim command is an obvious place to start but unless everything is configured correctly from the VM and and down, it’s not going to do you any good. Let’s verify that and run fstrim in the VM.
In my case, the fstrim command didnt return anything at all. Something’s up.
Lets start from the top. Firstly, we need to ensure that our VM storage configuration is set up correctly.
We need to make it so that fstrim on our VM actually does something. If you run fstrim -av in your VM and nothing happens, chances are the disk configuration is not correct. While the SCSI-controller can be VirtIO-based, the actual VM disk cannot. VirtioBlk received discard support in kernel 5.0 so for Debian Buster we would need to pick SCSCI for our disk device. For existing VM’s you’ll have to edit the VM config file (/etc/pve/nodes/qemu-server) and change to scsi based disks. If your VM is running a newer kernel, this step shouldnt be necessary and Virtio Block based disks can be used.
Next, you need to make sure the the discard option is checked.
Running fstrim -av now should produce some output indicating how much storage has been freed. Great success. Note that the fstrim command in this case is simply used as a mechanism to communicate information about freed space through the hypervisor to the underlying file system.
However, running zfs list on your Proxmox host before and after the fstrim command is run might still reveal that not space has been freed on the host file system. If that’s the case it means that while your VM is discarding blocks, it’s not propagating down the underlying file system. If this is what you’re seeing, read on.
In that case, we need to make some further adjustments. Up until quite recently, Proxmox storage was not thin-provision enabled by default. Navigate to the datacenter view in the web interface, click storage and edit the relevant storage entry. Here you need to ensure that thin-provisioning is actually enabled. If it isnt, you’ll have to migrate all your VM disks in order for thin provisioning to be enabled for them.
After all that is done, thin provisioning should finally work and you can start reclaiming some of that lost storage. For a Proxmox instance that has lived on for a while, that can be quite a sizeable amount of the total storage. Once everything is working you should be able to run zfs list, observe the used storage for the different disks, then run fstrim -av in your VMs and then run zfs list again on the Proxmox host. You should notice a reduction in the amount of used space. Finally, make sure to enable the fstrim.timer service on your VMs. You could add the discardoption to your fstab, but the performance penalty associated with it is most likely not worth it. Also, I’d recommend that you enable lz4 compression on your ZFS pool for even greater storage efficiency. This can be done with a simple one-liner: zfs set compression=lz4 <pool> Run zfs get compression first to check the current compression setting. Note that in order for the compression to take effect the data would have to be re-written.
