{{tag> benchmark i/o festplatte raid}}
====== Überprüfen der Disk-I/O ======
Leitfrage: Festplatten, Festplatten-RAIDs, Software-RAIDs oder Hardware-RAIDs - was braucht man für welchen Zweck?
===== Frage: Hardware oder Software-RAID =====
Neben der Fragen der Performance, zählt für mich noch das Argument der Wiederherstellung im Disasterfall: Wenn der Hardware-Controller ausfällt braucht man (meines Wissens nach) einen identischen, zumindest gleicher Bauart und neuer RAID-Controller, damit auf dem neuen RAID-Controller die Plattenzusammenstellung wieder erkannt wird.
Beim Software-RAID muss man nur einen neuen Computer/Mainboard finden und alle relevanten Platten einen Anschluss finden. Und man würde natürlich auf SATA statt SAS setzen, was es billiger macht und man flexibler ist.
==== Test zwischen Hard- und Software-RAID 5 ====
Ich teste auf einem System zwei RAID5: einmal 3 Festplatten auf einem Hardware-RAID, einmal 3 Festplatten in einem Software-RAID
=== Testbedingungen ===
^ Eigenschaft ^ Software-RAID ^ Hardware-RAID ^
| Art | RAID5 | RAID5 |
| controller | mdadm (DISKs als JBOD von 3ware durchgereicht) | 3ware 9650SE |
| Anzahl HDDs | 3 | 3 |
| Typ HDDs | 3x7200 RPM Barracuda-ES, baugleich | 3x7200RPM Barracuda-ES, 2 baugleich, 1 kleiner und älter |
| Zeit | morgens 7 Uhr, nahezu kein Traffic | morgens 7 Uhr, nahezu kein Traffic |
| load | im laufenden Betrieb, sonst kein Zugriff | im laufenden Betrieb, Zugriff von 5 VMs |
=== Ergebnisse hdparm ===
== Software-RAID ==
# hdparm -t -T /dev/md/storage
/dev/md/storage:
Timing cached reads: 17692 MB in 2.00 seconds = 8854.64 MB/sec
Timing buffered disk reads: 772 MB in 3.01 seconds = 256.84 MB/sec
== Hardware-RAID ==
# hdparm -t -T /dev/sda
/dev/sda:
Timing cached reads: 17066 MB in 2.00 seconds = 8540.70 MB/sec
Timing buffered disk reads: 586 MB in 3.00 seconds = 195.14 MB/sec
=== Ergebnisse fio ===
# aptitude install fio
== Konfiguration fio ==
Diese Konfiguration wurde einfach von [[http://www.linux.com/learn/tutorials/442451-inspecting-disk-io-performance-with-fio|hier]] übernommen, ohne nachzudenken, ob das I/O dem des linuxmuster.net Server entspricht oder nicht.
Das ''directory'' muss für die jeweilige Konfiguration in einem Verzeichnis auf dem Dateisystem des zu testenden Systems liegen.
# fio --output four-threads-randio.fio.log four-threads-randio.fio
# fio --output 3ware-four-threads-randio.fio.log 3ware-four-threads-randio.fio
++++ Konfiguration aufklappen |
# cat four-threads-randio.fio
; Four threads, two query, two writers.
[global]
rw=randread
size=256m
directory=./fio-testing-data
ioengine=libaio
iodepth=4
invalidate=1
direct=1
[bgwriter]
rw=randwrite
iodepth=32
[queryA]
iodepth=1
ioengine=mmap
direct=0
thinktime=3
[queryB]
iodepth=1
ioengine=mmap
direct=0
thinktime=5
[bgupdater]
rw=randrw
iodepth=16
thinktime=40
size=32m
++++
== Software-RAID ==
++++ Gesamtes Ergebnis-LOG aufklappen |
# cat four-threads-randio.fio.log
bgwriter: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
queryA: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=mmap, iodepth=1
queryB: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=mmap, iodepth=1
bgupdater: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=16
fio 1.59
Starting 4 processes
bgwriter: (groupid=0, jobs=1): err= 0: pid=27437
write: io=262144KB, bw=2263.4KB/s, iops=565 , runt=115822msec
slat (usec): min=3 , max=118225 , avg=39.25, stdev=1290.65
clat (usec): min=156 , max=1515.7K, avg=56502.06, stdev=61108.65
lat (usec): min=163 , max=1515.7K, avg=56541.80, stdev=61120.94
bw (KB/s) : min= 0, max= 4359, per=83.34%, avg=2002.68, stdev=811.04
cpu : usr=0.84%, sys=3.49%, ctx=63684, majf=0, minf=20
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued r/w/d: total=0/65536/0, short=0/0/0
lat (usec): 250=0.07%, 500=0.73%, 750=0.62%, 1000=0.63%
lat (msec): 2=2.62%, 4=4.60%, 10=6.84%, 20=8.96%, 50=25.22%
lat (msec): 100=37.08%, 250=12.07%, 500=0.21%, 750=0.17%, 1000=0.09%
lat (msec): 2000=0.08%
queryA: (groupid=0, jobs=1): err= 0: pid=27438
read : io=262144KB, bw=643708 B/s, iops=157 , runt=417014msec
clat (usec): min=176 , max=200425 , avg=6287.15, stdev=11708.44
lat (usec): min=176 , max=200425 , avg=6287.50, stdev=11708.43
bw (KB/s) : min= 42, max= 3548, per=49.00%, avg=635.08, stdev=363.95
cpu : usr=0.03%, sys=0.98%, ctx=131137, majf=65536, minf=30
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w/d: total=65536/0/0, short=0/0/0
lat (usec): 250=0.93%, 500=17.14%, 750=5.39%, 1000=0.86%
lat (msec): 2=6.15%, 4=15.67%, 10=46.72%, 20=4.41%, 50=0.99%
lat (msec): 100=1.34%, 250=0.38%
queryB: (groupid=0, jobs=1): err= 0: pid=27439
read : io=262144KB, bw=646142 B/s, iops=157 , runt=415443msec
clat (usec): min=160 , max=271656 , avg=6261.13, stdev=11846.33
lat (usec): min=160 , max=271657 , avg=6261.49, stdev=11846.33
bw (KB/s) : min= 37, max= 2801, per=49.59%, avg=642.66, stdev=363.18
cpu : usr=0.02%, sys=1.00%, ctx=131139, majf=65536, minf=29
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w/d: total=65536/0/0, short=0/0/0
lat (usec): 250=0.75%, 500=17.31%, 750=5.46%, 1000=1.05%
lat (msec): 2=6.20%, 4=15.78%, 10=46.44%, 20=4.26%, 50=1.01%
lat (msec): 100=1.32%, 250=0.41%, 500=0.01%
bgupdater: (groupid=0, jobs=1): err= 0: pid=27440
read : io=16548KB, bw=772410 B/s, iops=188 , runt= 21938msec
slat (usec): min=6 , max=62538 , avg=30.38, stdev=972.10
clat (usec): min=150 , max=248703 , avg=34749.50, stdev=34677.60
lat (usec): min=166 , max=248728 , avg=34780.21, stdev=34677.13
bw (KB/s) : min= 0, max= 1344, per=53.27%, avg=690.34, stdev=253.09
write: io=16220KB, bw=757100 B/s, iops=184 , runt= 21938msec
slat (usec): min=5 , max=67015 , avg=59.93, stdev=1725.22
clat (usec): min=164 , max=700093 , avg=50753.48, stdev=45899.73
lat (usec): min=178 , max=700105 , avg=50813.75, stdev=45948.04
bw (KB/s) : min= 541, max= 1648, per=30.77%, avg=739.43, stdev=174.52
cpu : usr=0.51%, sys=1.51%, ctx=14239, majf=0, minf=23
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w/d: total=4137/4055/0, short=0/0/0
lat (usec): 250=0.24%, 500=1.71%, 750=1.28%, 1000=1.27%
lat (msec): 2=6.07%, 4=8.07%, 10=12.06%, 20=9.80%, 50=20.98%
lat (msec): 100=29.98%, 250=8.41%, 500=0.09%, 750=0.04%
Run status group 0 (all jobs):
READ: io=540836KB, aggrb=1296KB/s, minb=643KB/s, maxb=772KB/s, mint=21938msec, maxt=417014msec
WRITE: io=278364KB, aggrb=2403KB/s, minb=757KB/s, maxb=2317KB/s, mint=21938msec, maxt=115822msec
Disk stats (read/write):
md127: ios=135165/69672, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=68188/46436, aggrmerge=9/32, aggrticks=1253285/593384, aggrin_queue=1846674, aggrutil=66.60%
sdb: ios=68242/46501, merge=0/43, ticks=1260148/592240, in_queue=1852372, util=66.60%
sdd: ios=68168/46420, merge=0/41, ticks=1250500/590996, in_queue=1841512, util=66.05%
sde: ios=68155/46387, merge=28/12, ticks=1249208/596916, in_queue=1846140, util=66.47%
++++
== Hardware-RAID ==
++++ Gesamtes Ergebnis-LOG aufklappen |
# cat 3ware-four-threads-randio.fio.log
bgwriter: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
queryA: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=mmap, iodepth=1
queryB: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=mmap, iodepth=1
bgupdater: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=16
fio 1.59
Starting 4 processes
bgwriter: (groupid=0, jobs=1): err= 0: pid=4990
write: io=262144KB, bw=563087 B/s, iops=137 , runt=476721msec
slat (usec): min=5 , max=3312.1K, avg=682.18, stdev=42345.18
clat (usec): min=65 , max=8143.4K, avg=231980.82, stdev=861214.70
lat (usec): min=77 , max=8143.4K, avg=232663.24, stdev=862096.66
bw (KB/s) : min= 1, max= 3411, per=103.74%, avg=605.85, stdev=659.98
cpu : usr=0.02%, sys=0.26%, ctx=60906, majf=0, minf=20
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued r/w/d: total=0/65536/0, short=0/0/0
lat (usec): 100=0.01%, 500=0.37%, 750=0.32%, 1000=0.32%
lat (msec): 2=1.32%, 4=85.57%, 10=2.38%, 20=0.72%, 50=0.22%
lat (msec): 100=0.40%, 250=0.52%, 500=0.30%, 750=0.24%, 1000=0.15%
lat (msec): 2000=0.68%, >=2000=6.47%
queryA: (groupid=0, jobs=1): err= 0: pid=4991
read : io=262144KB, bw=315841 B/s, iops=77 , runt=849905msec
clat (usec): min=65 , max=1146.6K, avg=12892.38, stdev=38584.22
lat (usec): min=65 , max=1146.6K, avg=12892.73, stdev=38584.22
bw (KB/s) : min= 5, max= 2142, per=52.53%, avg=334.10, stdev=293.16
cpu : usr=0.01%, sys=0.49%, ctx=131093, majf=65536, minf=30
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w/d: total=65536/0/0, short=0/0/0
lat (usec): 100=0.74%, 250=5.76%, 500=7.66%, 750=2.81%, 1000=0.64%
lat (msec): 2=2.88%, 4=12.95%, 10=45.56%, 20=13.32%, 50=3.72%
lat (msec): 100=1.65%, 250=1.63%, 500=0.60%, 750=0.08%, 1000=0.01%
lat (msec): 2000=0.01%
queryB: (groupid=0, jobs=1): err= 0: pid=4992
read : io=262144KB, bw=316713 B/s, iops=77 , runt=847565msec
clat (usec): min=62 , max=1453.9K, avg=12854.74, stdev=40260.59
lat (usec): min=62 , max=1453.9K, avg=12855.10, stdev=40260.59
bw (KB/s) : min= 3, max= 1204, per=53.04%, avg=337.32, stdev=302.66
cpu : usr=0.01%, sys=0.48%, ctx=131109, majf=65536, minf=29
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w/d: total=65536/0/0, short=0/0/0
lat (usec): 100=0.96%, 250=7.71%, 500=7.86%, 750=2.65%, 1000=0.63%
lat (msec): 2=2.80%, 4=12.47%, 10=44.34%, 20=13.24%, 50=3.38%
lat (msec): 100=1.58%, 250=1.69%, 500=0.59%, 750=0.08%, 1000=0.02%
lat (msec): 2000=0.01%
bgupdater: (groupid=0, jobs=1): err= 0: pid=4993
read : io=16296KB, bw=81036 B/s, iops=19 , runt=205921msec
slat (usec): min=7 , max=3201.3K, avg=4739.56, stdev=104796.65
clat (usec): min=2 , max=2752.5K, avg=98666.26, stdev=215133.50
lat (usec): min=78 , max=3201.3K, avg=103406.16, stdev=237364.57
bw (KB/s) : min= 0, max= 5928, per=29.74%, avg=189.14, stdev=623.76
write: io=16472KB, bw=81911 B/s, iops=19 , runt=205921msec
slat (usec): min=8 , max=3011.1K, avg=1733.96, stdev=63925.62
clat (usec): min=65 , max=8140.2K, avg=695273.75, stdev=1395565.63
lat (usec): min=74 , max=8140.2K, avg=697008.03, stdev=1396181.47
bw (KB/s) : min= 0, max= 6248, per=34.52%, avg=201.62, stdev=754.72
cpu : usr=0.02%, sys=0.15%, ctx=13742, majf=0, minf=23
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w/d: total=4074/4118/0, short=0/0/0
lat (usec): 4=0.06%, 100=7.52%, 250=18.26%, 500=7.17%, 750=1.97%
lat (usec): 1000=1.00%
lat (msec): 2=2.25%, 4=8.94%, 10=6.49%, 20=5.70%, 50=7.40%
lat (msec): 100=6.25%, 250=8.96%, 500=4.14%, 750=1.32%, 1000=0.67%
lat (msec): 2000=2.94%, >=2000=8.97%
Run status group 0 (all jobs):
READ: io=540584KB, aggrb=636KB/s, minb=81KB/s, maxb=316KB/s, mint=205921msec, maxt=849905msec
WRITE: io=278616KB, aggrb=584KB/s, minb=81KB/s, maxb=563KB/s, mint=205921msec, maxt=476721msec
Disk stats (read/write):
dm-2: ios=135457/71309, merge=0/0, ticks=2052000/18297432, in_queue=20349460, util=100.00%, aggrios=146894/83854, aggrmerge=33536/40266, aggrticks=2219656/22239272, aggrin_queue=24461388, aggrutil=100.00%
sda: ios=146894/83854, merge=33536/40266, ticks=2219656/22239272, in_queue=24461388, util=100.00%
++++
== Unterschiede anzeigen ==
Links: Hardware-RAID, Rechts: Software-RAID
++++ Unterschiede komplett aufklappen |
diff -y *.log | less -S
bgwriter: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libai bgwriter: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libai
queryA: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=mmap, io queryA: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=mmap, io
queryB: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=mmap, io queryB: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=mmap, io
bgupdater: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, bgupdater: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio,
fio 1.59 fio 1.59
Starting 4 processes Starting 4 processes
bgwriter: (groupid=0, jobs=1): err= 0: pid=4990 | bgwriter: (groupid=0, jobs=1): err= 0: pid=27437
write: io=262144KB, bw=563087 B/s, iops=137 , runt=476721ms | write: io=262144KB, bw=2263.4KB/s, iops=565 , runt=115822ms
slat (usec): min=5 , max=3312.1K, avg=682.18, stdev=42345 | slat (usec): min=3 , max=118225 , avg=39.25, stdev=1290.6
clat (usec): min=65 , max=8143.4K, avg=231980.82, stdev=8 | clat (usec): min=156 , max=1515.7K, avg=56502.06, stdev=6
lat (usec): min=77 , max=8143.4K, avg=232663.24, stdev=8 | lat (usec): min=163 , max=1515.7K, avg=56541.80, stdev=6
bw (KB/s) : min= 1, max= 3411, per=103.74%, avg=605.85 | bw (KB/s) : min= 0, max= 4359, per=83.34%, avg=2002.68
cpu : usr=0.02%, sys=0.26%, ctx=60906, majf=0, min | cpu : usr=0.84%, sys=3.49%, ctx=63684, majf=0, min
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32= IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%,
issued r/w/d: total=0/65536/0, short=0/0/0 issued r/w/d: total=0/65536/0, short=0/0/0
lat (usec): 100=0.01%, 500=0.37%, 750=0.32%, 1000=0.32% | lat (usec): 250=0.07%, 500=0.73%, 750=0.62%, 1000=0.63%
lat (msec): 2=1.32%, 4=85.57%, 10=2.38%, 20=0.72%, 50=0. | lat (msec): 2=2.62%, 4=4.60%, 10=6.84%, 20=8.96%, 50=25.
lat (msec): 100=0.40%, 250=0.52%, 500=0.30%, 750=0.24%, | lat (msec): 100=37.08%, 250=12.07%, 500=0.21%, 750=0.17%
lat (msec): 2000=0.68%, >=2000=6.47% | lat (msec): 2000=0.08%
queryA: (groupid=0, jobs=1): err= 0: pid=4991 | queryA: (groupid=0, jobs=1): err= 0: pid=27438
read : io=262144KB, bw=315841 B/s, iops=77 , runt=849905mse | read : io=262144KB, bw=643708 B/s, iops=157 , runt=417014ms
clat (usec): min=65 , max=1146.6K, avg=12892.38, stdev=38 | clat (usec): min=176 , max=200425 , avg=6287.15, stdev=11
lat (usec): min=65 , max=1146.6K, avg=12892.73, stdev=38 | lat (usec): min=176 , max=200425 , avg=6287.50, stdev=11
bw (KB/s) : min= 5, max= 2142, per=52.53%, avg=334.10, | bw (KB/s) : min= 42, max= 3548, per=49.00%, avg=635.08,
cpu : usr=0.01%, sys=0.49%, ctx=131093, majf=65536 | cpu : usr=0.03%, sys=0.98%, ctx=131137, majf=65536
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 3 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 3
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
issued r/w/d: total=65536/0/0, short=0/0/0 issued r/w/d: total=65536/0/0, short=0/0/0
lat (usec): 100=0.74%, 250=5.76%, 500=7.66%, 750=2.81%, | lat (usec): 250=0.93%, 500=17.14%, 750=5.39%, 1000=0.86%
lat (msec): 2=2.88%, 4=12.95%, 10=45.56%, 20=13.32%, 50= | lat (msec): 2=6.15%, 4=15.67%, 10=46.72%, 20=4.41%, 50=0
lat (msec): 100=1.65%, 250=1.63%, 500=0.60%, 750=0.08%, | lat (msec): 100=1.34%, 250=0.38%
lat (msec): 2000=0.01% | queryB: (groupid=0, jobs=1): err= 0: pid=27439
queryB: (groupid=0, jobs=1): err= 0: pid=4992 | read : io=262144KB, bw=646142 B/s, iops=157 , runt=415443ms
read : io=262144KB, bw=316713 B/s, iops=77 , runt=847565mse | clat (usec): min=160 , max=271656 , avg=6261.13, stdev=11
clat (usec): min=62 , max=1453.9K, avg=12854.74, stdev=40 | lat (usec): min=160 , max=271657 , avg=6261.49, stdev=11
lat (usec): min=62 , max=1453.9K, avg=12855.10, stdev=40 | bw (KB/s) : min= 37, max= 2801, per=49.59%, avg=642.66,
bw (KB/s) : min= 3, max= 1204, per=53.04%, avg=337.32, | cpu : usr=0.02%, sys=1.00%, ctx=131139, majf=65536
cpu : usr=0.01%, sys=0.48%, ctx=131109, majf=65536 <
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 3 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 3
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
issued r/w/d: total=65536/0/0, short=0/0/0 issued r/w/d: total=65536/0/0, short=0/0/0
lat (usec): 100=0.96%, 250=7.71%, 500=7.86%, 750=2.65%, | lat (usec): 250=0.75%, 500=17.31%, 750=5.46%, 1000=1.05%
lat (msec): 2=2.80%, 4=12.47%, 10=44.34%, 20=13.24%, 50= | lat (msec): 2=6.20%, 4=15.78%, 10=46.44%, 20=4.26%, 50=1
lat (msec): 100=1.58%, 250=1.69%, 500=0.59%, 750=0.08%, | lat (msec): 100=1.32%, 250=0.41%, 500=0.01%
lat (msec): 2000=0.01% | bgupdater: (groupid=0, jobs=1): err= 0: pid=27440
bgupdater: (groupid=0, jobs=1): err= 0: pid=4993 | read : io=16548KB, bw=772410 B/s, iops=188 , runt= 21938mse
read : io=16296KB, bw=81036 B/s, iops=19 , runt=205921msec | slat (usec): min=6 , max=62538 , avg=30.38, stdev=972.10
slat (usec): min=7 , max=3201.3K, avg=4739.56, stdev=1047 | clat (usec): min=150 , max=248703 , avg=34749.50, stdev=3
clat (usec): min=2 , max=2752.5K, avg=98666.26, stdev=215 | lat (usec): min=166 , max=248728 , avg=34780.21, stdev=3
lat (usec): min=78 , max=3201.3K, avg=103406.16, stdev=2 | bw (KB/s) : min= 0, max= 1344, per=53.27%, avg=690.34,
bw (KB/s) : min= 0, max= 5928, per=29.74%, avg=189.14, | write: io=16220KB, bw=757100 B/s, iops=184 , runt= 21938mse
write: io=16472KB, bw=81911 B/s, iops=19 , runt=205921msec | slat (usec): min=5 , max=67015 , avg=59.93, stdev=1725.22
slat (usec): min=8 , max=3011.1K, avg=1733.96, stdev=6392 | clat (usec): min=164 , max=700093 , avg=50753.48, stdev=4
clat (usec): min=65 , max=8140.2K, avg=695273.75, stdev=1 | lat (usec): min=178 , max=700105 , avg=50813.75, stdev=4
lat (usec): min=74 , max=8140.2K, avg=697008.03, stdev=1 | bw (KB/s) : min= 541, max= 1648, per=30.77%, avg=739.43,
bw (KB/s) : min= 0, max= 6248, per=34.52%, avg=201.62, | cpu : usr=0.51%, sys=1.51%, ctx=14239, majf=0, min
cpu : usr=0.02%, sys=0.15%, ctx=13742, majf=0, min <
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%,
issued r/w/d: total=4074/4118/0, short=0/0/0 | issued r/w/d: total=4137/4055/0, short=0/0/0
lat (usec): 4=0.06%, 100=7.52%, 250=18.26%, 500=7.17%, 7 | lat (usec): 250=0.24%, 500=1.71%, 750=1.28%, 1000=1.27%
lat (usec): 1000=1.00% | lat (msec): 2=6.07%, 4=8.07%, 10=12.06%, 20=9.80%, 50=20
lat (msec): 2=2.25%, 4=8.94%, 10=6.49%, 20=5.70%, 50=7.4 | lat (msec): 100=29.98%, 250=8.41%, 500=0.09%, 750=0.04%
lat (msec): 100=6.25%, 250=8.96%, 500=4.14%, 750=1.32%, <
lat (msec): 2000=2.94%, >=2000=8.97% <
Run status group 0 (all jobs): Run status group 0 (all jobs):
READ: io=540584KB, aggrb=636KB/s, minb=81KB/s, maxb=316KB/ | READ: io=540836KB, aggrb=1296KB/s, minb=643KB/s, maxb=772K
WRITE: io=278616KB, aggrb=584KB/s, minb=81KB/s, maxb=563KB/ | WRITE: io=278364KB, aggrb=2403KB/s, minb=757KB/s, maxb=2317
Disk stats (read/write): Disk stats (read/write):
dm-2: ios=135457/71309, merge=0/0, ticks=2052000/18297432, | md127: ios=135165/69672, merge=0/0, ticks=0/0, in_queue=0,
sda: ios=146894/83854, merge=33536/40266, ticks=2219656/2 | sdb: ios=68242/46501, merge=0/43, ticks=1260148/592240, i
> sdd: ios=68168/46420, merge=0/41, ticks=1250500/590996, i
> sde: ios=68155/46387, merge=28/12, ticks=1249208/596916,
++++
=== Fazit aus der Sicht des Software-RAIDs ===
* Schreibprozesse haben höhere Latenz, kleineren maximalen Schreibdurchsatz, aber größeren Durchsatz im Schnitt
* Leseprozesse haben ähnliche Latenz, deutlich kleineren maximalen Lesedurchsatz, aber größeren Durchsatz im Schnitt
* Updateprozsse haben ähnliche Latenz, kleineren maximalen Durchsatz, größeren Durchsatz im Schnitt
Man kann auch andere ERgebnisse herauslesen, wenn man nur die Durchschnittswerte liest. Dann hat auch bei der Latenz das Software-RAID die Nase vorn.
Die CPU-Last ist beim Software-RAID immer höher:
* Schreiben: 3,5% vs. 0,3%
* Lesen: 1% vs. 0,5%
* update: 1,5% vs. 0,15%