Hier werden die Unterschiede zwischen zwei Versionen angezeigt.
— | anwenderwiki:server:disk-i-o-testing [2016/03/08 10:52] (aktuell) – angelegt - Externe Bearbeitung 127.0.0.1 | ||
---|---|---|---|
Zeile 1: | Zeile 1: | ||
+ | {{tag> benchmark i/o festplatte raid}} | ||
+ | ====== Überprüfen der Disk-I/O ====== | ||
+ | |||
+ | Leitfrage: Festplatten, | ||
+ | |||
+ | ===== Frage: Hardware oder Software-RAID ===== | ||
+ | |||
+ | Neben der Fragen der Performance, | ||
+ | Beim Software-RAID muss man nur einen neuen Computer/ | ||
+ | |||
+ | ==== Test zwischen Hard- und Software-RAID 5 ==== | ||
+ | Ich teste auf einem System zwei RAID5: einmal 3 Festplatten auf einem Hardware-RAID, | ||
+ | |||
+ | === Testbedingungen === | ||
+ | |||
+ | ^ Eigenschaft | ||
+ | | Art | RAID5 | RAID5 | | ||
+ | | controller | ||
+ | | Anzahl HDDs | 3 | 3 | | ||
+ | | Typ HDDs | 3x7200 RPM Barracuda-ES, | ||
+ | | Zeit | morgens 7 Uhr, nahezu kein Traffic | ||
+ | | load | im laufenden Betrieb, sonst kein Zugriff | ||
+ | |||
+ | |||
+ | === Ergebnisse hdparm === | ||
+ | |||
+ | == Software-RAID == | ||
+ | < | ||
+ | # hdparm -t -T / | ||
+ | |||
+ | / | ||
+ | | ||
+ | | ||
+ | </ | ||
+ | |||
+ | == Hardware-RAID == | ||
+ | |||
+ | < | ||
+ | # hdparm -t -T /dev/sda | ||
+ | |||
+ | /dev/sda: | ||
+ | | ||
+ | | ||
+ | </ | ||
+ | |||
+ | |||
+ | === Ergebnisse fio === | ||
+ | < | ||
+ | == Konfiguration fio == | ||
+ | Diese Konfiguration wurde einfach von [[http:// | ||
+ | |||
+ | Das '' | ||
+ | |||
+ | < | ||
+ | # fio --output four-threads-randio.fio.log four-threads-randio.fio | ||
+ | # fio --output 3ware-four-threads-randio.fio.log 3ware-four-threads-randio.fio | ||
+ | </ | ||
+ | |||
+ | ++++ Konfiguration aufklappen | | ||
+ | < | ||
+ | # cat four-threads-randio.fio | ||
+ | ; Four threads, two query, two writers. | ||
+ | |||
+ | [global] | ||
+ | rw=randread | ||
+ | size=256m | ||
+ | directory=./ | ||
+ | ioengine=libaio | ||
+ | iodepth=4 | ||
+ | invalidate=1 | ||
+ | direct=1 | ||
+ | |||
+ | [bgwriter] | ||
+ | rw=randwrite | ||
+ | iodepth=32 | ||
+ | |||
+ | [queryA] | ||
+ | iodepth=1 | ||
+ | ioengine=mmap | ||
+ | direct=0 | ||
+ | thinktime=3 | ||
+ | |||
+ | [queryB] | ||
+ | iodepth=1 | ||
+ | ioengine=mmap | ||
+ | direct=0 | ||
+ | thinktime=5 | ||
+ | |||
+ | [bgupdater] | ||
+ | rw=randrw | ||
+ | iodepth=16 | ||
+ | thinktime=40 | ||
+ | size=32m | ||
+ | </ | ||
+ | ++++ | ||
+ | |||
+ | |||
+ | == Software-RAID == | ||
+ | |||
+ | ++++ Gesamtes Ergebnis-LOG aufklappen | | ||
+ | < | ||
+ | |||
+ | # cat four-threads-randio.fio.log | ||
+ | |||
+ | bgwriter: (g=0): rw=randwrite, | ||
+ | queryA: (g=0): rw=randread, | ||
+ | queryB: (g=0): rw=randread, | ||
+ | bgupdater: (g=0): rw=randrw, bs=4K-4K/ | ||
+ | fio 1.59 | ||
+ | Starting 4 processes | ||
+ | |||
+ | bgwriter: (groupid=0, jobs=1): err= 0: pid=27437 | ||
+ | write: io=262144KB, | ||
+ | slat (usec): min=3 , max=118225 , avg=39.25, stdev=1290.65 | ||
+ | clat (usec): min=156 , max=1515.7K, | ||
+ | lat (usec): min=163 , max=1515.7K, | ||
+ | bw (KB/s) : min= 0, max= 4359, per=83.34%, avg=2002.68, | ||
+ | cpu : usr=0.84%, sys=3.49%, ctx=63684, majf=0, minf=20 | ||
+ | IO depths | ||
+ | | ||
+ | | ||
+ | | ||
+ | lat (usec): 250=0.07%, 500=0.73%, 750=0.62%, 1000=0.63% | ||
+ | lat (msec): 2=2.62%, 4=4.60%, 10=6.84%, 20=8.96%, 50=25.22% | ||
+ | lat (msec): 100=37.08%, 250=12.07%, 500=0.21%, 750=0.17%, 1000=0.09% | ||
+ | lat (msec): 2000=0.08% | ||
+ | queryA: (groupid=0, jobs=1): err= 0: pid=27438 | ||
+ | read : io=262144KB, | ||
+ | clat (usec): min=176 , max=200425 , avg=6287.15, | ||
+ | lat (usec): min=176 , max=200425 , avg=6287.50, | ||
+ | bw (KB/s) : min= 42, max= 3548, per=49.00%, avg=635.08, stdev=363.95 | ||
+ | cpu : usr=0.03%, sys=0.98%, ctx=131137, majf=65536, minf=30 | ||
+ | IO depths | ||
+ | | ||
+ | | ||
+ | | ||
+ | lat (usec): 250=0.93%, 500=17.14%, 750=5.39%, 1000=0.86% | ||
+ | lat (msec): 2=6.15%, 4=15.67%, 10=46.72%, 20=4.41%, 50=0.99% | ||
+ | lat (msec): 100=1.34%, 250=0.38% | ||
+ | queryB: (groupid=0, jobs=1): err= 0: pid=27439 | ||
+ | read : io=262144KB, | ||
+ | clat (usec): min=160 , max=271656 , avg=6261.13, | ||
+ | lat (usec): min=160 , max=271657 , avg=6261.49, | ||
+ | bw (KB/s) : min= 37, max= 2801, per=49.59%, avg=642.66, stdev=363.18 | ||
+ | cpu : usr=0.02%, sys=1.00%, ctx=131139, majf=65536, minf=29 | ||
+ | IO depths | ||
+ | | ||
+ | | ||
+ | | ||
+ | lat (usec): 250=0.75%, 500=17.31%, 750=5.46%, 1000=1.05% | ||
+ | lat (msec): 2=6.20%, 4=15.78%, 10=46.44%, 20=4.26%, 50=1.01% | ||
+ | lat (msec): 100=1.32%, 250=0.41%, 500=0.01% | ||
+ | bgupdater: (groupid=0, jobs=1): err= 0: pid=27440 | ||
+ | read : io=16548KB, bw=772410 B/s, iops=188 , runt= 21938msec | ||
+ | slat (usec): min=6 , max=62538 , avg=30.38, stdev=972.10 | ||
+ | clat (usec): min=150 , max=248703 , avg=34749.50, | ||
+ | lat (usec): min=166 , max=248728 , avg=34780.21, | ||
+ | bw (KB/s) : min= 0, max= 1344, per=53.27%, avg=690.34, stdev=253.09 | ||
+ | write: io=16220KB, bw=757100 B/s, iops=184 , runt= 21938msec | ||
+ | slat (usec): min=5 , max=67015 , avg=59.93, stdev=1725.22 | ||
+ | clat (usec): min=164 , max=700093 , avg=50753.48, | ||
+ | lat (usec): min=178 , max=700105 , avg=50813.75, | ||
+ | bw (KB/s) : min= 541, max= 1648, per=30.77%, avg=739.43, stdev=174.52 | ||
+ | cpu : usr=0.51%, sys=1.51%, ctx=14239, majf=0, minf=23 | ||
+ | IO depths | ||
+ | | ||
+ | | ||
+ | | ||
+ | lat (usec): 250=0.24%, 500=1.71%, 750=1.28%, 1000=1.27% | ||
+ | lat (msec): 2=6.07%, 4=8.07%, 10=12.06%, 20=9.80%, 50=20.98% | ||
+ | lat (msec): 100=29.98%, 250=8.41%, 500=0.09%, 750=0.04% | ||
+ | |||
+ | Run status group 0 (all jobs): | ||
+ | READ: io=540836KB, | ||
+ | WRITE: io=278364KB, | ||
+ | |||
+ | Disk stats (read/ | ||
+ | md127: ios=135165/ | ||
+ | sdb: ios=68242/ | ||
+ | sdd: ios=68168/ | ||
+ | sde: ios=68155/ | ||
+ | |||
+ | </ | ||
+ | ++++ | ||
+ | |||
+ | == Hardware-RAID == | ||
+ | |||
+ | ++++ Gesamtes Ergebnis-LOG aufklappen | | ||
+ | < | ||
+ | # cat 3ware-four-threads-randio.fio.log | ||
+ | |||
+ | bgwriter: (g=0): rw=randwrite, | ||
+ | queryA: (g=0): rw=randread, | ||
+ | queryB: (g=0): rw=randread, | ||
+ | bgupdater: (g=0): rw=randrw, bs=4K-4K/ | ||
+ | fio 1.59 | ||
+ | Starting 4 processes | ||
+ | |||
+ | bgwriter: (groupid=0, jobs=1): err= 0: pid=4990 | ||
+ | write: io=262144KB, | ||
+ | slat (usec): min=5 , max=3312.1K, | ||
+ | clat (usec): min=65 , max=8143.4K, | ||
+ | lat (usec): min=77 , max=8143.4K, | ||
+ | bw (KB/s) : min= 1, max= 3411, per=103.74%, | ||
+ | cpu : usr=0.02%, sys=0.26%, ctx=60906, majf=0, minf=20 | ||
+ | IO depths | ||
+ | | ||
+ | | ||
+ | | ||
+ | lat (usec): 100=0.01%, 500=0.37%, 750=0.32%, 1000=0.32% | ||
+ | lat (msec): 2=1.32%, 4=85.57%, 10=2.38%, 20=0.72%, 50=0.22% | ||
+ | lat (msec): 100=0.40%, 250=0.52%, 500=0.30%, 750=0.24%, 1000=0.15% | ||
+ | lat (msec): 2000=0.68%, > | ||
+ | queryA: (groupid=0, jobs=1): err= 0: pid=4991 | ||
+ | read : io=262144KB, | ||
+ | clat (usec): min=65 , max=1146.6K, | ||
+ | lat (usec): min=65 , max=1146.6K, | ||
+ | bw (KB/s) : min= 5, max= 2142, per=52.53%, avg=334.10, stdev=293.16 | ||
+ | cpu : usr=0.01%, sys=0.49%, ctx=131093, majf=65536, minf=30 | ||
+ | IO depths | ||
+ | | ||
+ | | ||
+ | | ||
+ | lat (usec): 100=0.74%, 250=5.76%, 500=7.66%, 750=2.81%, 1000=0.64% | ||
+ | lat (msec): 2=2.88%, 4=12.95%, 10=45.56%, 20=13.32%, 50=3.72% | ||
+ | lat (msec): 100=1.65%, 250=1.63%, 500=0.60%, 750=0.08%, 1000=0.01% | ||
+ | lat (msec): 2000=0.01% | ||
+ | queryB: (groupid=0, jobs=1): err= 0: pid=4992 | ||
+ | read : io=262144KB, | ||
+ | clat (usec): min=62 , max=1453.9K, | ||
+ | lat (usec): min=62 , max=1453.9K, | ||
+ | bw (KB/s) : min= 3, max= 1204, per=53.04%, avg=337.32, stdev=302.66 | ||
+ | cpu : usr=0.01%, sys=0.48%, ctx=131109, majf=65536, minf=29 | ||
+ | IO depths | ||
+ | | ||
+ | | ||
+ | | ||
+ | lat (usec): 100=0.96%, 250=7.71%, 500=7.86%, 750=2.65%, 1000=0.63% | ||
+ | lat (msec): 2=2.80%, 4=12.47%, 10=44.34%, 20=13.24%, 50=3.38% | ||
+ | lat (msec): 100=1.58%, 250=1.69%, 500=0.59%, 750=0.08%, 1000=0.02% | ||
+ | lat (msec): 2000=0.01% | ||
+ | bgupdater: (groupid=0, jobs=1): err= 0: pid=4993 | ||
+ | read : io=16296KB, bw=81036 B/s, iops=19 , runt=205921msec | ||
+ | slat (usec): min=7 , max=3201.3K, | ||
+ | clat (usec): min=2 , max=2752.5K, | ||
+ | lat (usec): min=78 , max=3201.3K, | ||
+ | bw (KB/s) : min= 0, max= 5928, per=29.74%, avg=189.14, stdev=623.76 | ||
+ | write: io=16472KB, bw=81911 B/s, iops=19 , runt=205921msec | ||
+ | slat (usec): min=8 , max=3011.1K, | ||
+ | clat (usec): min=65 , max=8140.2K, | ||
+ | lat (usec): min=74 , max=8140.2K, | ||
+ | bw (KB/s) : min= 0, max= 6248, per=34.52%, avg=201.62, stdev=754.72 | ||
+ | cpu : usr=0.02%, sys=0.15%, ctx=13742, majf=0, minf=23 | ||
+ | IO depths | ||
+ | | ||
+ | | ||
+ | | ||
+ | lat (usec): 4=0.06%, 100=7.52%, 250=18.26%, 500=7.17%, 750=1.97% | ||
+ | lat (usec): 1000=1.00% | ||
+ | lat (msec): 2=2.25%, 4=8.94%, 10=6.49%, 20=5.70%, 50=7.40% | ||
+ | lat (msec): 100=6.25%, 250=8.96%, 500=4.14%, 750=1.32%, 1000=0.67% | ||
+ | lat (msec): 2000=2.94%, > | ||
+ | |||
+ | Run status group 0 (all jobs): | ||
+ | READ: io=540584KB, | ||
+ | WRITE: io=278616KB, | ||
+ | |||
+ | Disk stats (read/ | ||
+ | dm-2: ios=135457/ | ||
+ | sda: ios=146894/ | ||
+ | |||
+ | </ | ||
+ | ++++ | ||
+ | |||
+ | == Unterschiede anzeigen == | ||
+ | |||
+ | Links: Hardware-RAID, | ||
+ | ++++ Unterschiede komplett aufklappen | | ||
+ | < | ||
+ | diff -y *.log | less -S | ||
+ | bgwriter: (g=0): rw=randwrite, | ||
+ | queryA: (g=0): rw=randread, | ||
+ | queryB: (g=0): rw=randread, | ||
+ | bgupdater: (g=0): rw=randrw, bs=4K-4K/ | ||
+ | fio 1.59 fio 1.59 | ||
+ | Starting 4 processes | ||
+ | |||
+ | bgwriter: (groupid=0, jobs=1): err= 0: pid=4990 | ||
+ | write: io=262144KB, | ||
+ | slat (usec): min=5 , max=3312.1K, | ||
+ | clat (usec): min=65 , max=8143.4K, | ||
+ | lat (usec): min=77 , max=8143.4K, | ||
+ | bw (KB/s) : min= 1, max= 3411, per=103.74%, | ||
+ | cpu : usr=0.02%, sys=0.26%, ctx=60906, majf=0, min | | ||
+ | IO depths | ||
+ | | ||
+ | | ||
+ | | ||
+ | lat (usec): 100=0.01%, 500=0.37%, 750=0.32%, 1000=0.32% | ||
+ | lat (msec): 2=1.32%, 4=85.57%, 10=2.38%, 20=0.72%, 50=0. | lat (msec): 2=2.62%, 4=4.60%, 10=6.84%, 20=8.96%, 50=25. | ||
+ | lat (msec): 100=0.40%, 250=0.52%, 500=0.30%, 750=0.24%, | ||
+ | lat (msec): 2000=0.68%, > | ||
+ | queryA: (groupid=0, jobs=1): err= 0: pid=4991 | ||
+ | read : io=262144KB, | ||
+ | clat (usec): min=65 , max=1146.6K, | ||
+ | lat (usec): min=65 , max=1146.6K, | ||
+ | bw (KB/s) : min= 5, max= 2142, per=52.53%, avg=334.10, | bw (KB/s) : min= 42, max= 3548, per=49.00%, avg=635.08, | ||
+ | cpu : usr=0.01%, sys=0.49%, ctx=131093, majf=65536 | | ||
+ | IO depths | ||
+ | | ||
+ | | ||
+ | | ||
+ | lat (usec): 100=0.74%, 250=5.76%, 500=7.66%, 750=2.81%, | ||
+ | lat (msec): 2=2.88%, 4=12.95%, 10=45.56%, 20=13.32%, 50= | lat (msec): 2=6.15%, 4=15.67%, 10=46.72%, 20=4.41%, 50=0 | ||
+ | lat (msec): 100=1.65%, 250=1.63%, 500=0.60%, 750=0.08%, | ||
+ | lat (msec): 2000=0.01% | ||
+ | queryB: (groupid=0, jobs=1): err= 0: pid=4992 | ||
+ | read : io=262144KB, | ||
+ | clat (usec): min=62 , max=1453.9K, | ||
+ | lat (usec): min=62 , max=1453.9K, | ||
+ | bw (KB/s) : min= 3, max= 1204, per=53.04%, avg=337.32, | | ||
+ | cpu : usr=0.01%, sys=0.48%, ctx=131109, majf=65536 < | ||
+ | IO depths | ||
+ | | ||
+ | | ||
+ | | ||
+ | lat (usec): 100=0.96%, 250=7.71%, 500=7.86%, 750=2.65%, | ||
+ | lat (msec): 2=2.80%, 4=12.47%, 10=44.34%, 20=13.24%, 50= | lat (msec): 2=6.20%, 4=15.78%, 10=46.44%, 20=4.26%, 50=1 | ||
+ | lat (msec): 100=1.58%, 250=1.69%, 500=0.59%, 750=0.08%, | ||
+ | lat (msec): 2000=0.01% | ||
+ | bgupdater: (groupid=0, jobs=1): err= 0: pid=4993 | ||
+ | read : io=16296KB, bw=81036 B/s, iops=19 , runt=205921msec | ||
+ | slat (usec): min=7 , max=3201.3K, | ||
+ | clat (usec): min=2 , max=2752.5K, | ||
+ | lat (usec): min=78 , max=3201.3K, | ||
+ | bw (KB/s) : min= 0, max= 5928, per=29.74%, avg=189.14, | | ||
+ | write: io=16472KB, bw=81911 B/s, iops=19 , runt=205921msec | ||
+ | slat (usec): min=8 , max=3011.1K, | ||
+ | clat (usec): min=65 , max=8140.2K, | ||
+ | lat (usec): min=74 , max=8140.2K, | ||
+ | bw (KB/s) : min= 0, max= 6248, per=34.52%, avg=201.62, | | ||
+ | cpu : usr=0.02%, sys=0.15%, ctx=13742, majf=0, min < | ||
+ | IO depths | ||
+ | | ||
+ | | ||
+ | | ||
+ | lat (usec): 4=0.06%, 100=7.52%, 250=18.26%, 500=7.17%, 7 | lat (usec): 250=0.24%, 500=1.71%, 750=1.28%, 1000=1.27% | ||
+ | lat (usec): 1000=1.00% | ||
+ | lat (msec): 2=2.25%, 4=8.94%, 10=6.49%, 20=5.70%, 50=7.4 | lat (msec): 100=29.98%, 250=8.41%, 500=0.09%, 750=0.04% | ||
+ | lat (msec): 100=6.25%, 250=8.96%, 500=4.14%, 750=1.32%, | ||
+ | lat (msec): 2000=2.94%, > | ||
+ | |||
+ | Run status group 0 (all jobs): | ||
+ | READ: io=540584KB, | ||
+ | WRITE: io=278616KB, | ||
+ | |||
+ | Disk stats (read/ | ||
+ | dm-2: ios=135457/ | ||
+ | sda: ios=146894/ | ||
+ | > sdd: ios=68168/ | ||
+ | > sde: ios=68155/ | ||
+ | </ | ||
+ | ++++ | ||
+ | |||
+ | === Fazit aus der Sicht des Software-RAIDs === | ||
+ | |||
+ | * Schreibprozesse haben höhere Latenz, kleineren maximalen Schreibdurchsatz, | ||
+ | * Leseprozesse haben ähnliche Latenz, deutlich kleineren maximalen Lesedurchsatz, | ||
+ | * Updateprozsse haben ähnliche Latenz, kleineren maximalen Durchsatz, größeren Durchsatz im Schnitt | ||
+ | |||
+ | Man kann auch andere ERgebnisse herauslesen, | ||
+ | |||
+ | Die CPU-Last ist beim Software-RAID immer höher: | ||
+ | |||
+ | * Schreiben: 3,5% vs. 0,3% | ||
+ | * Lesen: 1% vs. 0,5% | ||
+ | * update: 1,5% vs. 0,15% |