Recently I encountered a alert in vRealize Operations stating that on one of my virtual machines the disks-ace was getting low. It turned out to be the vRealize Log Insight server in my test environment. For those of you that don’t know, vRealize Log Insight is a highly scalable log management solution which can collect all log information in your environment and analyse and present this in intuitive, actionable dashboards and sophisticated analytics.Log Insight

Browsing through the vRealize Log Insight interface I could not find any settings regarding retention times, garbage collection intervals, history settings or any techniques to limit the amount of disk space which is used. The only option I have is to setup an Archive location which ensures long time archival of log data.

So how does vRealize Log Insight manage its internal storage capacity, how to free up disk space, how to extend disk space?

vRealize Log Insight Internal storage management

After the initial installation vRealize Log Insight has the following disk layout.

Volume:

  1. Root file system: 12 GB.
  2. Log Data volume (size depends on deployment):
    1. Extra-Small/Small deployment: 120 GB
    2. Medium/Large: 270 GB
  3. First boot volume: 256 MB.

Log Insight

I deployed an Extra-Small deployment so I have a 120 GB  volume on which the system stores the collected log information. (An observant viewer may have noticed that the vRealize Log Insight System Monitor only reports 93.4 GB of available space. The additional space is reserved for system purposes so that the system can continue to function even if all the space available has filled up.)

Why is the storage space 97% utilized?

vRealize Log Insight stores the collected log data in buckets which are stored locally. New log data will be added to the bucket until the bucket reaches 1 GB. When that happens a new bucket is created and the old bucket will be marked read-only.

The system uses all storage that is presented to it to store these buckets and maintains the space itself. The Log Data volume is designed to fill up to a maximum of 97%.

In normal situations, the system never runs out of disk because every minute it checks if the free space is less than 3%. If the free space on the vRealize Log Insight virtual appliance drops below 3%, old data buckets are retired. However, if the disk is small and log ingestion rate is so high that the free space (3%) is filled out within 1 minute, vRealize Log Insight runs out of disk space. If archiving is enabled, vRealize Log Insight archives the bucket before retiring it. If the free space is filled before the old bucket is archived and retired, the system runs out of disk.

In short, the vRealize Log Insight internal storage management:

  • may fill up the Log Data Volume to a maximum of 97% (under normal operating conditions).
  • will not remove stored events until the maximum capacity (97%) is reached. This means Log Insight will keep all the events as long as it has space for it. So environments with a low log volume may hold years of history, environments with a high log volume may just hold months or even weeks of history.
  • will only report if the amount of data the system can hold at the current ingestion rate is less than the value configured in the Retention Notification Threshold (Administration/Configuration/General). It will not retire data buckets older than the configured threshold.
  • will automatically retire the oldest stored events when retention has been exceeded.

 

caution_smallCaution
  • Shrinking disks on the Log Insight virtual appliance is not supported and might result in data corruption or data loss.
  • Log Insight automatically cleans up the virtual disk. No changes should be made to the OS of the virtual appliance.

How to increase vRealize Log Insight storage space

According to the vRealize Log Insight Configuration Limits, Log Insight 3.0, 3.3 and 3.6 have a maximum of 4TB addressable storage space, on VMDKs with a maximum size of 2TB each. That means you could have two 2TB VMDKs, or four 1TB VMDKs, etc. Once you get to sizes around that point, though, you’ll probably want to scale outward with a larger cluster size instead of adding more disk to existing VMs.

Procedure:

  1. Log in to the vSphere Web Client as a user who has privileges to modify the hardware of virtual machines in the environment.
  2. Shut down the vRealize Log Insight virtual appliance safely.
  3. In the vSphere Client inventory, right-click the vRealize Log Insight virtual appliance and select [Edit Settings].
  4. On the Hardware tab, click [Add].
  5. Select [Hard Disk] and click [Next].
  6. Select [Create a new virtual disk] and click [Next].
    1. Type the disk capacity.vRealize Log Insight supports virtual hard disks of up to 2 TB. If you need more capacity, add more than one virtual hard disk.
    2. Select a disk format.
      • Thick Provision Lazy Zeroed.
      • Thick Provision Eager Zeroed.
      • Thin Provision.
    3. Select a datastore, browse for the datastore location and click [Next].
  7. Accept the default virtual device node and click [Next].
  8. Review the information and click [Finish].
  9. Click [OK] to save your changes and close the dialog box.

 

When you power on the vRealize Log Insight virtual appliance, the virtual machine discovers the new virtual disk and automatically adds it to the Log Data volume.

caution_smallCaution
  • Do not add space to an existing virtual disk as this will not work.
  • Once you add a disk to the virtual appliance, you cannot remove it. Removing disks from the virtual appliance may result in complete data loss.