Contact Support

Customers who viewed this article also viewed

banner icon

Identify Changes in NetScaler build files with

File Integrity Monitoring

Learn More Watch Video
CTX202821 {{tooltipText}}

File System on Control Domain Full on XenServer

Applicable Products

  • Citrix Hypervisor
  • XenServer

Symptoms or Error

There are a variety of errors and symptoms that can be seen when the Control Domain file system becomes full. The errors and symptoms that you see might be different depending on the partition layout of your XenServer host.
User-added image

Possible errors can include the following errors or alerts might appear in XenCenter:

  • File System on Control Domain Full.
  • Disk usage for the Control Domain on server 'XenServer' has reached 99.0%. XenServer's performance will be critically affected if this disk becomes full .
  • Log files or other non-essential (user created) files should be removed.
  • The update precheck stage failed: the server does not have enough space.
Possible symptoms can include the following:
  • "No space left on the device" error when attempting to apply hotfixes, cumulative updates, or current releases as an update to the XenServer host.
  • Performance of the VMs across the pool or host becomes sluggish.
  • Host becomes completely unresponsive.
  • XenServer console will show error: "The XenAPI connection has timed out , the console will have reduced functionality".
  • All the Consoles, for hosts or VM,  become unresponsive.
  • Error while trying to update hotfixes: "No space left on the device".
  • Any command or any process on the host  takes too long to execute.
  • Host will disconnect from XenCenter and will not connect back
  • HA monitor thread stops without warning. Running VMs are no longer protected from host failure.
To verify that a full file system is the cause of these symptoms, you can run the following command:

Command output on a host with the legacy partition layout:

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2             4.0G  3.6G  147M  97% /
none                  2.0G   40K  2.0G   1% /dev/shm
/opt/xensource/packages/iso/XenCenter.iso
                       52M   52M     0 100% /var/xen/xc-install
/dev/mapper/XSLocalEXT--6e4fc08e--5459--ccbb--4959--2bccc93076ea-6e4fc08e--5459--ccbb--4959--2bccc93076ea
                      542G  167G  347G  33% /var/run/sr-mount/6e4fc08e-5459-ccbb-4959-2bccc93076ea

Command output on a host with the new partition layout:

Filesystem            Size  Used Avail Use% Mounted on
devtmpfs              951M  0    951M    0% /dev
tmpfs                 962M  72K  962M    1% /dev/shm
tmpfs                 962M  948K 961M    1% /run
tmpfs                 962M  0    962M    0% /sys/fs/cgroup
/dev/sda1             18G   17G  1.3G    93% /
xenstore              962M  0    962M    0% /var/lib/xenstored
/dev/loop0            45M   45M  0     100% /var/xen/xc-install
/dev/sda5             3.9G  12M  3.7G    1% /var/log

tmpfs                 193M  0    193M    0% /run/user/0

If the value of the 'Use%' for the file system mounted on / is greater than 90%, your Control Domain file system is full and you must free up space. You must also free up space in cases where the Use% is less than 90% but you are getting error messages that state that you do not have enough space to complete the action you are attempting (for example, applying an update).


Solution

The causes of these problems and the steps to take can be different depending on the partition layout of your XenServer host. For more information about partition layouts, see https://docs.citrix.com/en-us/xenserver/current-release/install/partition-layout.html

This section includes steps to take to free up space right now. It also includes practices that you can put in place to avoid the problem occurring in the future.

What partition layout is your host using?

  • Legacy partition layout: This partition layout allows a 4 GB partition for the Control Domain. The Control Domain partition is also used for swap and logging. 
    All XenServer hosts of version 6.5 and earlier use the legacy partition layout. Some XenServer hosts of version 7.0 and later use the legacy partition layout; this is the case when the host was upgraded from XenServer 6.5 or earlier and kept its existing layout, or when the host is a small device that does not have enough space for the new partition layout.
    If the output of the df -h command shows the size of the file system mounted on / as 4 GB, your host has the legacy partition layout.
  • New partition layout: This partition layout allows a dedicated 18 GB partition for the Control Domain. Logs are saved to a separate dedicated partition. XenServer hosts with this partition layout are less likely to experience this problem.
    All XenServer hosts of version 7.0 and later have this partition layout by default. However, some hosts can use the legacy partition layout if they were upgraded from XenServer version 6.5 or earlier and kept their existing layout or if they are a small device that does not have enough space for the new partition layout.
    If the output of the df -h command shows the size of the file system mounted on / as 18 GB, your host has the new partition layout.

Freeing up space on the Control Domain filesystem now 

1. Remove additional logs

Legacy partition layout only: You can clear surplus logs for all XenServer hosts, but it will free space in the Control Domain filesystem only for those hosts that have the legacy partition layout.

  1. Change directory into the /var/log directory
    # cd /var/log
  2. Remove old compressed logs (.gz files). Do not remove *.log files as this will break log rotation and requires a reboot of the host to fix.
    # rm *.gz

Refer to the following screen shot for an example command. This command ensures that all the current logs files will be intact and will not be deleted.

In the /var/log directory, command rm -rf audit.log.* messages.* SMlog.* xensource.log.* xenstored-access.log.*

2. Remove cached uploaded patch files

For XenServer 7.0 and earlier:
1. Change directory into /var/patch
  # cd /var/patch
2. Remove all the entries except the "/applied" folder 
3. Use the following command to remove all the uploaded patches and without deleting the applied folder
    # find /var/patch -maxdepth 1 -type f -exec rm -v {} +
 
For XenServer 7.1 and later:
1. Change directory into /var/update
   
# cd /var/update
2.
Remove all the entries except the "/applied" folder
3.  Use the following command to remove all the uploaded patches and without deleting the applied folder

  # find /var/update -maxdepth 1 -type f -exec rm -v {} +

3. Remove the patch-backups

  1. cd /opt/xensource/patch-backup
  2. ls
  3. rm -rf *
In the /opt/xensource/patch-backup directory, command rm -rf
 

 

4. Remove stunnel entries

1. Change to the tmp directory
  # cd /tmp/
2. Remove any old stunnel log entries

In the /tmp directory, command rm -rf stunnel0* stunnel8* stunnelf*

Note: If these steps do not reflect the space being cleared, run #service syslog restart and the usage inside the root file system will be cleared.
 

5. Remove additional files 

  • Check /tmp directory for any rogue files. You can delete everything inside it as XenServer won't store any default files in it.

  • Check "~" and "/" to see if there are any additional ISO files or any extra packages are present inside the directory. If yes, delete them.

Even after executing all those above steps if the space is still not reclaimed, contact Citrix Technical support.

For more information, see https://support.citrix.com/article/CTX130245


 

 


Problem Cause

  • Root partition is filled up.

  • Rogue Entries of patches, updates, or log files.


Additional Resources


Disclaimer

The above mentioned sample code is provided to you as is with no representations, warranties or conditions of any kind. You may use, modify and distribute it at your own risk. CITRIX DISCLAIMS ALL WARRANTIES WHATSOEVER, EXPRESS, IMPLIED, WRITTEN, ORAL OR STATUTORY, INCLUDING WITHOUT LIMITATION WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NONINFRINGEMENT. Without limiting the generality of the foregoing, you acknowledge and agree that (a) the sample code may exhibit errors, design flaws or other problems, possibly resulting in loss of data or damage to property; (b) it may not be possible to make the sample code fully functional; and (c) Citrix may, without notice or liability to you, cease to make available the current version and/or any future versions of the sample code. In no event should the code be used to support ultra-hazardous activities, including but not limited to life support or blasting activities. NEITHER CITRIX NOR ITS AFFILIATES OR AGENTS WILL BE LIABLE, UNDER BREACH OF CONTRACT OR ANY OTHER THEORY OF LIABILITY, FOR ANY DAMAGES WHATSOEVER ARISING FROM USE OF THE SAMPLE CODE, INCLUDING WITHOUT LIMITATION DIRECT, SPECIAL, INCIDENTAL, PUNITIVE, CONSEQUENTIAL OR OTHER DAMAGES, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Although the copyright in the code belongs to Citrix, any distribution of the sample code should include only your own standard copyright attribution, and not that of Citrix. You agree to indemnify and defend Citrix against any and all claims arising from your use, modification or distribution of the sample code.