Identify log consumer in vCenter appliance (KB83070)

Often it happens to me that /storage/log runs out of space for different reasons, here is my way to approach this.
This will show you the 20 biggest sub-directories, in this case you see the

root@vcsa[ /storage/log ]# df -h
Filesystem                                Size  Used Avail Use% Mounted on
devtmpfs                                  9.3G     0  9.3G   0% /dev
tmpfs                                     9.3G  1.8M  9.3G   1% /dev/shm
tmpfs                                     9.3G  1.2M  9.3G   1% /run
tmpfs                                     9.3G     0  9.3G   0% /sys/fs/cgroup
/dev/sda3                                  46G   12G   33G  26% /
tmpfs                                     9.3G  5.6M  9.3G   1% /tmp
/dev/sda2                                 120M   30M   82M  27% /boot
/dev/mapper/vtsdb_vg-vtsdb                 25G   78M   24G   1% /storage/vtsdb
/dev/mapper/vtsdblog_vg-vtsdblog           15G   57M   14G   1% /storage/vtsdblog
/dev/mapper/lifecycle_vg-lifecycle         98G  3.6G   90G   4% /storage/lifecycle
/dev/mapper/core_vg-core                   49G  324M   47G   1% /storage/core
/dev/mapper/updatemgr_vg-updatemgr         98G  8.3G   85G   9% /storage/updatemgr
/dev/mapper/log_vg-log                     20G   19G  709M  97% /storage/log
/dev/mapper/netdump_vg-netdump            985M  2.5M  915M   1% /storage/netdump
/dev/mapper/db_vg-db                      9.8G  1.8G  7.6G  19% /storage/db
/dev/mapper/seat_vg-seat                   25G  8.0G   16G  35% /storage/seat
/dev/mapper/imagebuilder_vg-imagebuilder  9.8G   37M  9.3G   1% /storage/imagebuilder
/dev/mapper/archive_vg-archive             49G   45G  2.4G  95% /storage/archive
/dev/mapper/dblog_vg-dblog                 15G  361M   14G   3% /storage/dblog
/dev/mapper/autodeploy_vg-autodeploy      9.8G  2.2G  7.2G  24% /storage/autodeploy

root@vcsa[ /storage/log ]# du -Sh /storage/log/* | sort -rh | head -20
13G     /storage/log/vmware/vmware-sps
596M    /storage/log/vmware/vmware-updatemgr/vum-server
465M    /storage/log/vmware/vsan-health
390M    /storage/log/vmware
344M    /storage/log/vmware/vpxd
322M    /storage/log/vmware/vsphere-ui/logs
317M    /storage/log/vmware/vmafd
296M    /storage/log/vmware/sso
250M    /storage/log/vmware/analytics
234M    /storage/log/vmware/sca
232M    /storage/log/vmware/vpxd-svcs
181M    /storage/log/vmware/trustmanagement
181M    /storage/log/vmware/eam
158M    /storage/log/vmware/observability
149M    /storage/log/vmware/vmon
130M    /storage/log/vmware/vpxd/drmdump/domain-c26
130M    /storage/log/vmware/lookupsvc
117M    /storage/log/vmware/sso/tomcat
111M    /storage/log/vmware/vapi/endpoint
83M     /storage/log/vmware/pod

In this case vmware-sps is consuming most of the space thus looking in the subdirectory shows the issue.

rm /storage/log/vmware/vmware-sps/sps-access-log.* 

This cleared up 70% of disk space on /storage/log

Leave a Reply