Measuring performance improvements in CentOS 6.2

Recently, we updated some servers from CentOS 5.6 to CentOS 6.2.  Of course, we carefully monitored the performance of the hosts, so we can make a comparison about the performance. The one big change we applied with the application of Cent OS 6 was switching from ext3 to ext4 (although as yet without TRIM enabled.)

Nothing else changed on the hosts – same hardware (Sun X4170’s, in this case), same disks (Intel SSDs in RAID), same workload, same application version.

Different performance. Better performance, which is good – although the reason as to why is unclear.

This first thing that caught our eye was the amount of time the SSDs were busy was significantly less (The load is on the machine with the dark line to start with – then the same load is transferred to the machine with the green line). This is a plot of the amount of the time the SSD was busy:

Looking at the disk operations on the SSDs both systems are doing the same amount of writes – but the physical disk reads have dropped from about 700 per second to 340 per second. (Gotta love SSDs!)

These systems don’t swap, don’t have InnoDB buffer cache misses, and have the same hardware, application and load in both cases – the only variable is Centos 5 to 6.

So while we can’t explain why there should be less disk reads, presumably some changes in the virtual memory system made .. something.. more efficient.

The important thing is that we were measuring it, so we could see it. That way we know about it – and given that it could have gone the other way and made performance worse, it’s essential to be able to know about the impact changes make to systems. (Of course, we saw this change on non-production systems, before we rolled it out.)

Are you tracking the impacts of the changes you make to your infrastructure?