Symptoms or Error
There are a variety of errors and symptoms that can be seen when the Control Domain file system becomes full. The errors and symptoms that you see might be different depending on the partition layout of your XenServer host.
Possible errors can include the following errors or alerts might appear in XenCenter:
- File System on Control Domain Full.
- Disk usage for the Control Domain on server 'XenServer' has reached 99.0%. XenServer's performance will be critically affected if this disk becomes full .
- Log files or other non-essential (user created) files should be removed.
- The update precheck stage failed: the server does not have enough space.
- "No space left on the device" error when attempting to apply hotfixes, cumulative updates, or current releases as an update to the XenServer host.
- Performance of the VMs across the pool or host becomes sluggish.
- Host becomes completely unresponsive.
- XenServer console will show error: "The XenAPI connection has timed out , the console will have reduced functionality".
- All the Consoles, for hosts or VM, become unresponsive.
- Error while trying to update hotfixes: "No space left on the device".
- Any command or any process on the host takes too long to execute.
- Host will disconnect from XenCenter and will not connect back
- HA monitor thread stops without warning. Running VMs are no longer protected from host failure.
Command output on a host with the legacy partition layout:
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 4.0G 3.6G 147M 97% /
none 2.0G 40K 2.0G 1% /dev/shm
/opt/xensource/packages/iso/XenCenter.iso
52M 52M 0 100% /var/xen/xc-install
/dev/mapper/XSLocalEXT--6e4fc08e--5459--ccbb--4959--2bccc93076ea-6e4fc08e--5459--ccbb--4959--2bccc93076ea
542G 167G 347G 33% /var/run/sr-mount/6e4fc08e-5459-ccbb-4959-2bccc93076ea
Command output on a host with the new partition layout:
Filesystem Size Used Avail Use% Mounted on
devtmpfs 951M 0 951M 0% /dev
tmpfs 962M 72K 962M 1% /dev/shm
tmpfs 962M 948K 961M 1% /run
tmpfs 962M 0 962M 0% /sys/fs/cgroup
/dev/sda1 18G 17G 1.3G 93% /
xenstore 962M 0 962M 0% /var/lib/xenstored
/dev/loop0 45M 45M 0 100% /var/xen/xc-install
/dev/sda5 3.9G 12M 3.7G 1% /var/logtmpfs 193M 0 193M 0% /run/user/0
If the value of the 'Use%' for the file system mounted on / is greater than 90%, your Control Domain file system is full and you must free up space. You must also free up space in cases where the Use% is less than 90% but you are getting error messages that state that you do not have enough space to complete the action you are attempting (for example, applying an update).
Solution
The causes of these problems and the steps to take can be different depending on the partition layout of your XenServer host. For more information about partition layouts, see https://docs.citrix.com/en-us/xenserver/current-release/install/partition-layout.html.
This section includes steps to take to free up space right now. It also includes practices that you can put in place to avoid the problem occurring in the future.
What partition layout is your host using?
- Legacy partition layout: This partition layout allows a 4 GB partition for the Control Domain. The Control Domain partition is also used for swap and logging.
All XenServer hosts of version 6.5 and earlier use the legacy partition layout. Some XenServer hosts of version 7.0 and later use the legacy partition layout; this is the case when the host was upgraded from XenServer 6.5 or earlier and kept its existing layout, or when the host is a small device that does not have enough space for the new partition layout.
If the output of the df -h command shows the size of the file system mounted on / as 4 GB, your host has the legacy partition layout. - New partition layout: This partition layout allows a dedicated 18 GB partition for the Control Domain. Logs are saved to a separate dedicated partition. XenServer hosts with this partition layout are less likely to experience this problem.
All XenServer hosts of version 7.0 and later have this partition layout by default. However, some hosts can use the legacy partition layout if they were upgraded from XenServer version 6.5 or earlier and kept their existing layout or if they are a small device that does not have enough space for the new partition layout.
If the output of the df -h command shows the size of the file system mounted on / as 18 GB, your host has the new partition layout.
Freeing up space on the Control Domain filesystem now
1. Remove additional logs
Legacy partition layout only: You can clear surplus logs for all XenServer hosts, but it will free space in the Control Domain filesystem only for those hosts that have the legacy partition layout.
- Change directory into the /var/log directory
# cd /var/log - Remove old compressed logs (.gz files). Do not remove *.log files as this will break log rotation and requires a reboot of the host to fix.
# rm *.gz
Refer to the following screen shot for an example command. This command ensures that all the current logs files will be intact and will not be deleted.
2. Remove cached uploaded patch files
# cd /var/patch
2. Remove all the entries except the "/applied" folder
3. Use the following command to remove all the uploaded patches and without deleting the applied folder
# find /var/patch -maxdepth 1 -type f -exec rm -v {} +
# cd /var/update
2. Remove all the entries except the "/applied" folder
3. Use the following command to remove all the uploaded patches and without deleting the applied folder
# find /var/update -maxdepth 1 -type f -exec rm -v {} +
3. Remove the patch-backups
- cd /opt/xensource/patch-backup
- ls
- rm -rf *
4. Remove stunnel entries
1. Change to the tmp directory
# cd /tmp/
2. Remove any old stunnel log entries
Note: If these steps do not reflect the space being cleared, run #service syslog restart and the usage inside the root file system will be cleared.
5. Remove additional files
Check /tmp directory for any rogue files. You can delete everything inside it as XenServer won't store any default files in it.
Check "~" and "/" to see if there are any additional ISO files or any extra packages are present inside the directory. If yes, delete them.
Even after executing all those above steps if the space is still not reclaimed, contact Citrix Technical support.
For more information, see https://support.citrix.com/article/CTX130245.
Problem Cause
Root partition is filled up.
Rogue Entries of patches, updates, or log files.
Additional Resources
CTX215032 - How to limit size of logs in XenServer