DISK I/O - how to monitor with Zabbix

In the spirit of Zabbix and good monitoring I have decided to share a little something I have found and modified.

In this article I will explain and display how we can go about monitoring DISK I/O.

It should give you a rough idea of what your disks are doing and when it possibly could be a problem.

So here Goes:

First off I will explain a few things on the understanding of what is printed within "/proc/diskstats" for HD activity.

Here is an example:

cat /proc/diskstats
8    0 sda 490 2002 17576 5984 66 17 220 3495 0 9435 9479

I will give you a break down coupled with the explanation from the kernel iostats.txt file for what it is you are looking at:

From left to right:
1 - major number - Used for kernel/driver registration
2 - minor mumber - Used for kernel/driver registration but also maps position on device
3 - device name
4 - reads completed successfully
5 - reads merged
6 - sectors read
7 - time spent reading (ms)
8 - writes completed
9 - writes merged
10 - sectors written
11 - time spent writing (ms)
12 - I/Os currently in progress
13 - time spent doing I/Os (ms)
14 - weighted time spent doing I/Os (ms)

So now you asked yourself. What are all these numbers for and what could it possibly do for me. That my friend is a simple one. In most enterprise level data-centers today, it is not RAM or CPU that kills a server environment. It's DISK I/O. Waiting to write, waiting to read... Waiting... Waiting... Waiting...

To begin our monitoring we add the following User Parameter into your zabbix_agentd.conf of the client/monitored server.

cat << STOP >> /usr/local/etc/zabbix_agentd.conf
#
#
#
### DISK I/O###
UserParameter=custom.vfs.dev.read.ops[*],cat /proc/diskstats | egrep $1 | head -1 | awk '{print $$4}'
UserParameter=custom.vfs.dev.read.ms[*],cat /proc/diskstats | egrep $1 | head -1 | awk '{print $$7}'
UserParameter=custom.vfs.dev.write.ops[*],cat /proc/diskstats | egrep $1 | head -1 | awk '{print $$8}'
UserParameter=custom.vfs.dev.write.ms[*],cat /proc/diskstats | egrep $1 | head -1 | awk '{print $$11}'
UserParameter=custom.vfs.dev.io.active[*],cat /proc/diskstats | egrep $1 | head -1 | awk '{print $$12}'
UserParameter=custom.vfs.dev.io.ms[*],cat /proc/diskstats | egrep $1 | head -1 | awk '{print $$13}'
UserParameter=custom.vfs.dev.read.sectors[*],cat /proc/diskstats | egrep $1 | head -1 | awk '{print $$6}'
UserParameter=custom.vfs.dev.write.sectors[*],cat /proc/diskstats | egrep $1 | head -1 | awk '{print $$10}'
### DISK I/O###
STOP

Coupled with our new user parameters, all we need is to specify a list of disks in our template name that we wish to monitor. I will paste the XML for the template below:

Just copy my code from the template below. Save it as a *.xml. Import it to your Zabbix server and watch the monitoring grow
Server Check IO: See at the bottom of the post, you will need to download it directly :)

Once you have completed this you will need to restart the zabbix_agentd where you have added the new user parameters.

If you have any questions... Feel free to drop me a note :)

On a side note. You can easily clone the disk names once you have imported the template to include any other drives you may have on the system. This can simply be done by cloning an item in zabbix and changing the key value to the corresponding drive name for example: read.ms[sda] can be changed to read.ms[hda] if you have IDE drives.

PS: Some credits to articles on the net I have read.... and also the beauty of LINUX "THE" MAN pages.

PPS: Attached link to template - Zabbix IOSTAT Template