Hardware

This is an "Intel(R) Pentium(R) 4 CPU 3.40GHz" with HT enabled, 1GB of RAM, a SATA controller with a "Maxtor 7Y250M0" attached. The system was running the Linux Kernel version 2.6.20 with GLIBC version 2.3.6.


Software

Tests performed using the March 23, 2007 FIO snapshot, together with GUASI version 0.23:

FIO:   http://brick.kernel.dk/snaps/
GUASI: http://www.xmailserver.org/guasi-lib.html


Notes

To get consistent results, it is better to not use the --runtime FIO option, and use the --size instead to limit the test processing time. This because, for random RW tests, reads and writes may statistically distribute differently over the test time, and chopping it with --runtime may give wrong results. All tests has been run using the Anticipatory I/O Scheduler (AS), but the relative performance did not change when using CFQ, Deadline or Noop.


Results


[iodepth = 100]
>>> fio --name=global --rw=randrw --size=256m --bs=4k --direct=1 
        --ioengine=guasi --name=job1 --iodepth=100 --thread
job1: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=guasi, iodepth=100
Starting 1 thread
Jobs: 1: [m] [99.5% done] [  4165/     0 kb/s] [eta 00m:01s]
job1: (groupid=0, jobs=1): err= 0: pid=26943
  read : io=128MiB, bw=653KiB/s, iops=159, runt=204919msec
    slat (msec): min=    0, max=    3, avg= 0.00, stdev= 0.07
    clat (msec): min=    0, max=37299, avg=378.25, stdev=799.38
    bw (KiB/s) : min=   10, max= 4857, per=126.97%, avg=829.08, stdev=633.74
  write: io=129MiB, bw=818KiB/s, iops=199, runt=164940msec
    slat (msec): min=    0, max=    3, avg= 0.00, stdev= 0.08
    clat (msec): min=    0, max=  645, avg=245.75, stdev=96.78
    bw (KiB/s) : min=    0, max= 2318, per=136.71%, avg=1118.29, stdev=831.44
  cpu          : usr=0.26%, sys=2.70%, ctx=391041
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=99.9%
     lat (msec): 2=0.8%, 4=0.7%, 10=1.5%, 20=2.3%, 50=5.5%, 100=3.5%
     lat (msec): 250=48.6%, 500=19.3%, 750=11.4%, 1000=4.6%, >=2000=1.8%

Run status group 0 (all jobs):
   READ: io=128MiB, aggrb=653KiB/s, minb=653KiB/s, maxb=653KiB/s, mint=204919msec, maxt=204919msec
  WRITE: io=129MiB, aggrb=818KiB/s, minb=818KiB/s, maxb=818KiB/s, mint=164940msec, maxt=164940msec

Disk stats (read/write):
  sda: ios=30520/33054, merge=2155/63, ticks=10343188/157484, in_queue=10500680, util=99.70%



>>> fio --name=global --rw=randrw --size=256m --bs=4k --direct=1 
        --ioengine=libaio --name=job1 --iodepth=100 --thread
job1: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=100
Starting 1 thread
Jobs: 1: [m] [99.0% done] [  3506/     0 kb/s] [eta 00m:02s]
job1: (groupid=0, jobs=1): err= 0: pid=26933
  read : io=128MiB, bw=645KiB/s, iops=157, runt=207316msec
    slat (msec): min=    0, max=  608, avg= 0.15, stdev= 7.37
    clat (msec): min=    0, max= 1945, avg=374.27, stdev=330.52
    bw (KiB/s) : min=   17, max= 5116, per=114.01%, avg=735.38, stdev=653.86
  write: io=129MiB, bw=761KiB/s, iops=185, runt=177050msec
    slat (msec): min=    0, max=  588, avg= 0.19, stdev= 9.08
    clat (msec): min=    0, max= 1347, avg=256.53, stdev=219.55
    bw (KiB/s) : min=    0, max= 2965, per=112.11%, avg=853.16, stdev=732.91
  cpu          : usr=0.11%, sys=1.04%, ctx=77592
  IO depths    : 1=0.0%, 2=0.1%, 4=0.1%, 8=0.3%, 16=0.6%, 32=1.2%, >=64=97.7%
     lat (msec): 2=0.4%, 4=0.3%, 10=2.6%, 20=3.8%, 50=10.8%, 100=11.4%
     lat (msec): 250=22.6%, 500=24.8%, 750=13.3%, 1000=7.1%, >=2000=2.9%

Run status group 0 (all jobs):
   READ: io=128MiB, aggrb=645KiB/s, minb=645KiB/s, maxb=645KiB/s, mint=207316msec, maxt=207316msec
  WRITE: io=129MiB, aggrb=761KiB/s, minb=761KiB/s, maxb=761KiB/s, mint=177050msec, maxt=177050msec

Disk stats (read/write):
  sda: ios=30646/31983, merge=2032/1075, ticks=11528816/8175408, in_queue=19704272, util=99.99%



[iodepth = 1000]
>>> fio --name=global --rw=randrw --size=256m --bs=4k --direct=1 
        --ioengine=guasi --name=job1 --iodepth=1000 --thread
job1: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=guasi, iodepth=1000
Starting 1 thread
Jobs: 1: [m] [100.0% done] [  3275/     0 kb/s] [eta 00m:00s]
job1: (groupid=0, jobs=1): err= 0: pid=25822
  read : io=128MiB, bw=608KiB/s, iops=148, runt=220633msec
    slat (msec): min=    0, max=   41, avg= 0.47, stdev= 3.91
    clat (msec): min=    1, max= 7306, avg=3336.43, stdev=1318.26
    bw (KiB/s) : min=    2, max= 3760, per=110.22%, avg=670.13, stdev=499.45
  write: io=132MiB, bw=647KiB/s, iops=158, runt=213594msec
    slat (msec): min=    0, max=   41, avg= 0.49, stdev= 3.98
    clat (msec): min=    0, max= 6999, avg=3294.01, stdev=1007.12
    bw (KiB/s) : min=    0, max= 2998, per=132.56%, avg=857.67, stdev=794.11
  cpu          : usr=0.24%, sys=2.45%, ctx=434627
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=99.9%
     lat (msec): 2=0.1%, 4=0.0%, 10=0.1%, 20=0.2%, 50=0.2%, 100=0.1%
     lat (msec): 250=0.3%, 500=0.5%, 750=0.5%, 1000=0.6%, >=2000=11.2%

Run status group 0 (all jobs):
   READ: io=128MiB, aggrb=608KiB/s, minb=608KiB/s, maxb=608KiB/s, mint=220633msec, maxt=220633msec
  WRITE: io=132MiB, aggrb=647KiB/s, minb=647KiB/s, maxb=647KiB/s, mint=213594msec, maxt=213594msec

Disk stats (read/write):
  sda: ios=30413/33901, merge=2338/106, ticks=12581280/186340, in_queue=12768264, util=99.66%



>>> fio --name=global --rw=randrw --size=256m --bs=4k --direct=1 
        --ioengine=libaio --name=job1 --iodepth=1000 --thread
job1: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=1000
Starting 1 thread
Jobs: 1: [m] [97.7% done] [   577/   516 kb/s] [eta 00m:05s]
job1: (groupid=0, jobs=1): err= 0: pid=26828
  read : io=128MiB, bw=626KiB/s, iops=153, runt=213413msec
    slat (msec): min=    0, max= 4876, avg=3274.26, stdev=1619.11
    clat (msec): min=    0, max= 6058, avg=613.37, stdev=1333.82
    bw (KiB/s) : min=    0, max= 8126, per=214.47%, avg=1342.61, stdev=1809.33
  write: io=132MiB, bw=652KiB/s, iops=159, runt=212685msec
    slat (msec): min=    0, max= 2724, avg=1550.13, stdev=590.41
    clat (msec): min=    0, max= 4694, avg=469.30, stdev=1153.29
    bw (KiB/s) : min=    0, max= 7128, per=154.74%, avg=1008.93, stdev=1317.10
  cpu          : usr=0.06%, sys=0.72%, ctx=24804
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=99.9%
     lat (msec): 2=75.0%, 4=0.1%, 10=0.2%, 20=0.2%, 50=0.1%, 100=0.1%
     lat (msec): 250=2.4%, 500=2.2%, 750=1.4%, 1000=1.4%, >=2000=5.7%

Run status group 0 (all jobs):
   READ: io=128MiB, aggrb=626KiB/s, minb=626KiB/s, maxb=626KiB/s, mint=213413msec, maxt=213413msec
  WRITE: io=132MiB, aggrb=652KiB/s, minb=652KiB/s, maxb=652KiB/s, mint=212685msec, maxt=212685msec

Disk stats (read/write):
  sda: ios=30215/32234, merge=2448/1707, ticks=19955036/12018636, in_queue=33917852, util=99.94%



[iodepth = 10000]
>>> fio --name=global --rw=randrw --size=256m --bs=4k --direct=1 
        --ioengine=guasi --name=job1 --iodepth=10000 --thread
job1: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=guasi, iodepth=10000
Starting 1 thread
Jobs: 1: [m] [100.0% done] [     0/     0 kb/s] [eta 00m:00s]
job1: (groupid=0, jobs=1): err= 0: pid=21767
  read : io=145MiB, bw=641KiB/s, iops=156, runt=236473msec
    slat (msec): min=    0, max=  230, avg=18.17, stdev=50.51
    clat (msec): min=    1, max=53244, avg=30356.92, stdev=12078.88
    bw (KiB/s) : min=    2, max= 2514, per=91.11%, avg=584.01, stdev=369.46
  write: io=150MiB, bw=677KiB/s, iops=165, runt=232681msec
    slat (msec): min=    0, max=  230, avg=18.53, stdev=50.94
    clat (msec): min=    1, max=53220, avg=32159.18, stdev=10863.48
    bw (KiB/s) : min=    0, max= 3088, per=98.79%, avg=668.82, stdev=669.49
  cpu          : usr=0.20%, sys=2.13%, ctx=321641
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=99.9%
     lat (msec): 2=0.0%, 4=0.0%, 10=0.0%, 20=0.0%, 50=0.0%, 100=0.0%
     lat (msec): 250=0.0%, 500=0.0%, 750=0.0%, 1000=0.0%, >=2000=0.3%

Run status group 0 (all jobs):
   READ: io=145MiB, aggrb=641KiB/s, minb=641KiB/s, maxb=641KiB/s, mint=236473msec, maxt=236473msec
  WRITE: io=150MiB, aggrb=677KiB/s, minb=677KiB/s, maxb=677KiB/s, mint=232681msec, maxt=232681msec

Disk stats (read/write):
  sda: ios=30132/38602, merge=7003/72, ticks=9963460/237800, in_queue=10201296, util=99.11%


>>> fio --name=global --rw=randrw --size=256m --bs=4k --direct=1 
        --ioengine=libaio --name=job1 --iodepth=10000 --thread
job1: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=10000
Starting 1 thread
Jobs: 1: [m] [91.9% done] [     0/     0 kb/s] [eta 00m:21s]]
job1: (groupid=0, jobs=1): err= 0: pid=23794
  read : io=135MiB, bw=663KiB/s, iops=161, runt=213443msec
    slat (msec): min=  405, max=55971, avg=49388.29, stdev=7498.58
    clat (msec): min=    0, max=55981, avg=487.09, stdev=4603.44
    bw (KiB/s) : min=    0, max=78767, per=1604.69%, avg=10639.12, stdev=27546.25
  write: io=138MiB, bw=610KiB/s, iops=149, runt=237610msec
    slat (msec): min=    0, max=55972, avg=17419.86, stdev=9878.98
    clat (msec): min=    0, max=55981, avg=429.89, stdev=4413.43
    bw (KiB/s) : min=    0, max=80638, per=1582.17%, avg=9651.22, stdev=26668.44
  cpu          : usr=0.06%, sys=0.69%, ctx=27038
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=99.9%
     lat (msec): 2=0.4%, 4=97.0%, 10=0.0%, 20=0.5%, 50=0.1%, 100=0.1%
     lat (msec): 250=0.2%, 500=0.4%, 750=0.1%, 1000=0.0%, >=2000=0.0%

Run status group 0 (all jobs):
   READ: io=135MiB, aggrb=663KiB/s, minb=663KiB/s, maxb=663KiB/s, mint=213443msec, maxt=213443msec
  WRITE: io=138MiB, aggrb=610KiB/s, minb=610KiB/s, maxb=610KiB/s, mint=237610msec, maxt=237610msec

Disk stats (read/write):
  sda: ios=33738/28795, merge=907/6695, ticks=24825684/12270488, in_queue=37096692, util=99.57%



Conclusions

Over the multiple tests that I ran using FIO, GUASI and Libaio performance matched each other pretty much consistently. Performance fluctuated over different measures of about +/-10% around their mean value. Typically GUASI resulted only slightly more performing than Libaio, but not that much to sustain a claim (differences were lower than measure fluctuations). These results shows that the GUASI library can be used as generic asynchronous provider, with performance matching the KAIO (Linux Kernel AIO) implementation. Even when pushed to iodepths that are quite big for the test hardware. The big advantage of GUASI, is that it can render asynchronous any system call or whatever userspace function.



Back Home