How to Test Disk IO Speed with FIO (Flexible IO Tester) in Linux / Ubuntu / Fedora

Disk I/O speed is an indicator of how fast the system and read/write to the disk. Usually nvme ssds are the fastest and hard drives and external usb flash drives are the slowest.

There are many commands that can be used to benchmark the performance of storage disks on linux. For e.g. sysbench, dd and fio. In another article we learned how to use the sysbench command to benchmark the io speed of a disk drive. Check it here:

How to Test (Benchmark) Disk IO Speed with Sysbench in Linux / Ubuntu

In this post we shall use a different command line tool called fio (Flexible I/O Tester).

Install

Its available on most distros and should be easy to install

Ubuntu:

sudo apt install fio

Setup your disks

For the sake of this test experiment we shall be using a Seagate One Touch 5TB HDD externally connected via USB 3.2 Gen 1 (5 Gbps) port. We already tested this drive on windows and noted the read write speed for both sequential and random io, using CrystalDiskMark.

In this post we shall be testing the same disk on an Ubuntu 22.10 machine with fio. After installing fio first you need to setup everything correctly for the test.

Mount the drive: Make sure to mount to drive and then navigate to the partition from a terminal so that you can run commands inside that partition of this drive. This is important because fio will create test files directly in that location.

Use the fastest USB port: Make sure to plug the drive into he fastest available usb port that can utilise the full speed that the drive is capable of. Plugging into a slower usb drive will result into a bottleneck and cause the tests to deliver wrong measurements.

Testing with FIO

Finally its time to run the fio command and start testing. Navigate into the drive partition and run some basic commands.

mkdir fiotest
cd fiotest/

Now you should be inside a sub-directory named fiotest. We can run our tests in here and keep everything clean. Basically 2 types of tests would be run, sequential read-write and random read-write.

Sequential IO vs Random IO

In sequential write, a program keeps pushing block after block of data in a sequence without closing the file handle or commiting the data.

OPEN File - WRITE - WRITE - WRITE - WRITE ....... - CLOSE/COMMIT

In random write, a program writes some data, then closes the handle, then again opens the same or another handle, optionally does a seek operationg then writes and then commits and closes. These additional operations of opening, seeking, committing, closing multiple times introduces latency which slows down the overall speed of reading and writing.

OPEN File - WRITE SOME - COMMIT/CLOSE - OPEN File AGAIN (SEEK) - WRITE SOME - COMMIT/CLOSE - ......

On HDDs tracks this would often result in each block getting written to sectors that are physically next to each other in a sequence. But with ssds that is not the case anymore. Due to this the speed difference between sequential and random operation in hdd is huge like around 20-30x, whereas incase of nvme ssds its only about 5-10x.

It should be noted that with sequential operations, the thread count is maintained at 1, since multiple threads ideally should not write to same file (but can read though).

1. Sequential IO

The first type of test we can perform is the sequential write and read operations. This test involves writing large size files in one go without doing a lot of file open or seek operations, thereby delivering high data transfer throughput.

Write test: The simples of the test would involve writing 1 file of 1 GB with 1MB block-size and see what write speed is reported.

The command we shall be using is this:

fio --name=seq_write --numjobs=1 --size=1G --ioengine=libaio --direct=1 --verify=0 --bs=1M --iodepth=8 --rw=write --group_reporting=1

Command parameters:

--name: name for the test
--numjobs: number of jobs. more than 1 job would create parallel threads/processes.
--size: total size of the data to read/write
--ioengine: the read/write engine to use. here its libaio.
--direct: Use non-buffered (direct) IO.
--verify: verify the operation. we are skipping this for speed.
--iodepth: number of queues

To learn more about the options check the man page for fio with man fio command.

Output:

acerlight@acerlight-laptop:/media/acerlight/One Touch/fiotest$ fio --name=seq_write --numjobs=1 --size=1G --ioengine=libaio --direct=1 --verify=0 --bs=1M --iodepth=8 --rw=write --group_reporting=1
seq_write: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
fio-3.30
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=124MiB/s][w=124 IOPS][eta 00m:00s]
seq_write: (groupid=0, jobs=1): err= 0: pid=24356: Wed May 10 11:47:31 2023
  write: IOPS=122, BW=123MiB/s (129MB/s)(1024MiB/8348msec); 0 zone resets
    slat (usec): min=66, max=356, avg=235.02, stdev=63.41
    clat (msec): min=4, max=145, avg=64.91, stdev= 8.28
     lat (msec): min=4, max=145, avg=65.15, stdev= 8.28
    clat percentiles (msec):
     |  1.00th=[   54],  5.00th=[   64], 10.00th=[   64], 20.00th=[   64],
     | 30.00th=[   64], 40.00th=[   64], 50.00th=[   64], 60.00th=[   65],
     | 70.00th=[   65], 80.00th=[   65], 90.00th=[   66], 95.00th=[   74],
     | 99.00th=[   96], 99.50th=[  133], 99.90th=[  142], 99.95th=[  146],
     | 99.99th=[  146]
   bw (  KiB/s): min=110592, max=129024, per=99.97%, avg=125568.00, stdev=4211.70, samples=16
   iops        : min=  108, max=  126, avg=122.62, stdev= 4.11, samples=16
  lat (msec)   : 10=0.20%, 20=0.20%, 50=0.59%, 100=98.05%, 250=0.98%
  cpu          : usr=0.80%, sys=2.55%, ctx=1024, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.2%, 4=0.4%, 8=99.3%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1024,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=123MiB/s (129MB/s), 123MiB/s-123MiB/s (129MB/s-129MB/s), io=1024MiB (1074MB), run=8348-8348msec

Disk stats (read/write):
  sda: ios=0/1995, merge=0/0, ticks=0/125232, in_queue=125231, util=98.83%
acerlight@acerlight-laptop:/media/acerlight/One Touch/fiotest$

Check the second last section

Run status group 0 (all jobs):
  WRITE: bw=123MiB/s (129MB/s), 123MiB/s-123MiB/s (129MB/s-129MB/s), io=1024MiB (1074MB), run=8348-8348msec

The write speed: 123 MiB/s.: This is fairly correct and acceptable performance level for this Seagate One Touch 5 TB HDD.

Read Operation: For testing read operation we shall use a fairly simple test. A single 1GB file will be read in 1 Job, with 1MB block-size and queue depth = 1. This is the closest we can get to a single file read operation in sequential mode of operation.

The command will be:

fio --name=seq_read --numjobs=1  --size=1G  --ioengine=libaio --direct=1 --verify=0 --bs=1M --iodepth=1 --rw=read --group_reporting=1

Output:

acerlight@acerlight-laptop:/media/acerlight/One Touch/fiotest$ fio --name=seq_read --numjobs=1  --size=1G  --ioengine=libaio --direct=1 --verify=0 --bs=1M --iodepth=1 --rw=read --group_reporting=1seq_read: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.30
Starting 1 process
seq_read: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=105MiB/s][r=105 IOPS][eta 00m:00s]
seq_read: (groupid=0, jobs=1): err= 0: pid=25089: Wed May 10 12:46:36 2023
  read: IOPS=113, BW=113MiB/s (119MB/s)(1024MiB/9033msec)
    slat (usec): min=55, max=567, avg=193.56, stdev=55.66
    clat (msec): min=2, max=120, avg= 8.62, stdev= 6.45
     lat (msec): min=3, max=121, avg= 8.81, stdev= 6.46
    clat percentiles (msec):
     |  1.00th=[    7],  5.00th=[    8], 10.00th=[    8], 20.00th=[    8],
     | 30.00th=[    8], 40.00th=[    9], 50.00th=[    9], 60.00th=[    9],
     | 70.00th=[    9], 80.00th=[    9], 90.00th=[    9], 95.00th=[    9],
     | 99.00th=[   28], 99.50th=[   36], 99.90th=[  107], 99.95th=[  122],
     | 99.99th=[  122]
   bw (  KiB/s): min=30720, max=126976, per=100.00%, avg=116394.67, stdev=24518.21, samples=18
   iops        : min=   30, max=  124, avg=113.67, stdev=23.94, samples=18
  lat (msec)   : 4=0.10%, 10=97.85%, 20=0.29%, 50=1.37%, 100=0.20%
  lat (msec)   : 250=0.20%
  cpu          : usr=0.33%, sys=2.33%, ctx=1026, majf=0, minf=267
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=113MiB/s (119MB/s), 113MiB/s-113MiB/s (119MB/s-119MB/s), io=1024MiB (1074MB), run=9033-9033msec

Disk stats (read/write):
  sda: ios=2043/0, merge=0/0, ticks=12929/0, in_queue=12928, util=99.02%
acerlight@acerlight-laptop:/media/acerlight/One Touch/fiotest$

The sequential read speed: 113 MiB/s. This is pretty decent as the external hdd is rated to have speed close to that value.

2. Random I/O Operations

Next is the random io operations. These involve writing large number of small sized file which cause a lot of open/close/seek operations thereby emulating real life latency conditions.

Random Write:

The command we shall be using for random write test:

fio --name=seq_write --numjobs=5 --size=100M --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=64 --rw=randwrite --group_reporting=1

acerlight@acerlight-laptop:/media/acerlight/One Touch/fiotest$ fio --name=seq_write --numjobs=5 --size=100M --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=64 --rw=randwrite --group_reporting=1
seq_write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.30
Starting 5 processes
Jobs: 2 (f=0): [_(1),f(1),_(2),f(1)][100.0%][w=504KiB/s][w=126 IOPS][eta 00m:00s]
seq_write: (groupid=0, jobs=5): err= 0: pid=24732: Wed May 10 12:28:39 2023
  write: IOPS=125, BW=503KiB/s (515kB/s)(500MiB/1017900msec); 0 zone resets
    slat (nsec): min=1039, max=3491.2M, avg=37939273.87, stdev=174151951.81
    clat (msec): min=60, max=15622, avg=2394.80, stdev=3130.37
     lat (msec): min=60, max=15767, avg=2432.74, stdev=3179.65
    clat percentiles (msec):
     |  1.00th=[  121],  5.00th=[  153], 10.00th=[  174], 20.00th=[  201],
     | 30.00th=[  226], 40.00th=[  253], 50.00th=[  288], 60.00th=[  359],
     | 70.00th=[ 4329], 80.00th=[ 5805], 90.00th=[ 7282], 95.00th=[ 8423],
     | 99.00th=[10671], 99.50th=[11476], 99.90th=[13489], 99.95th=[14026],
     | 99.99th=[14966]
   bw (  KiB/s): min=   40, max= 8264, per=100.00%, avg=702.05, stdev=285.30, samples=7318
   iops        : min=   10, max= 2066, avg=175.51, stdev=71.32, samples=7318
  lat (msec)   : 100=0.23%, 250=38.70%, 500=24.47%, 750=0.75%, 1000=0.02%
  lat (msec)   : 2000=0.46%, >=2000=35.38%
  cpu          : usr=0.01%, sys=0.04%, ctx=27417, majf=0, minf=62
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,128000,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=503KiB/s (515kB/s), 503KiB/s-503KiB/s (515kB/s-515kB/s), io=500MiB (524MB), run=1017900-1017900msec

Disk stats (read/write):
  sda: ios=0/126764, merge=0/1238, ticks=0/57905153, in_queue=57905153, util=99.85%
acerlight@acerlight-laptop:/media/acerlight/One Touch/fiotest$

The random write speed: 515 KB/s. This turns out to be very low because if the specific operation parameters chosen in this test. Real life write operations vary a lot and so will the actual speed of writing data.

Cleanup test files

FIO creates files for each test which look somewhat like this:

acerlight@acerlight-laptop:/media/acerlight/One Touch/fiotest$ ls -la
total 3555840
drwxr-xr-x  2 acerlight acerlight     262144 May 10 12:46 .
drwxr-xr-x 13 acerlight acerlight     262144 May 10 11:38 ..
-rwxr-xr-x  1 acerlight acerlight 1073741824 May 10 12:44 read_throughput.0.0
-rwxr-xr-x  1 acerlight acerlight 1073741824 May 10 12:46 seq_read.0.0
-rwxr-xr-x  1 acerlight acerlight 1073741824 May 10 12:25 seq_write.0.0
-rwxr-xr-x  1 acerlight acerlight  104857600 May 10 12:28 seq_write.1.0
-rwxr-xr-x  1 acerlight acerlight  104857600 May 10 12:28 seq_write.2.0
-rwxr-xr-x  1 acerlight acerlight  104857600 May 10 12:28 seq_write.3.0
-rwxr-xr-x  1 acerlight acerlight  104857600 May 10 12:28 seq_write.4.0
acerlight@acerlight-laptop:/media/acerlight/One Touch/fiotest$

They can be cleared easily with the rm * command.