Gábor Samu
Gábor Samu
Creator of this blog.
Mar 5, 2021 27 min read

Fun with an AArch64 NAS

thumbnail for this post

If you’re anything like me, managing the data that is produced by our modern lifestyles is a chore. I’m the designated archival person in the family and as such I’m always looking for better ways to manage the huge volumes of data, from family photos and video to all of my music which I’ve digitized from my huge personal CD library.

My NAS over approximately the last 4 years has been my Turris Omnia router, which I’ve equipped with both an external USB disk and an internal mSATA drive. The Turris Omnia for those who aren’t familiar with it, is a high performance router that runs open source software based on OpenWrt. It’s served me very well without ever missing a beatiand will continue as my home router for the foreseeable future. Although a NAS enclosure is available for the Turris Omnia, I never purchased one. That’s because the Turris Omnia NAS option only has 2 drive bays. About a week ago, I started to ponder building a NAS which would take me to the next level of capacity and performance.

As luck would have it, some acquaintances have also been looking at looking at modernizing their NAS setups. So this fact has also provide some motivation. Most are looking at turnkey NAS solutions through, of which many shapes and sizes exist. When it comes to IT, I’m never one to take the easy path. I like to push the envelope and try to build using non-X86 architectures whenever possible. And it’s no different when it comes to a NAS for my household.

A device which recently caught my attention is the Traverse Ten64, which is a high-performance Arm based networking platform. Certainly the fact that it’s Arm based got my attention. Furthermore, it looks quite similar to the Turris Omnia but more powerful. What really piqued my interest was the writeup Building a NAS with Ten64 and Rockstor. Certainly I’ve come across Rockstor before, which is a well known NAS and cloud solution. But I was not aware of the beta Rockstor 4 recipe for aarch64. This really got me thinking! As the Traverse Ten64 is not shipping yet, how could I try out Rockstor on aarch64? Of course, there was the teasing of my acquaintances that I should just get a brand X NAS and be done with it. With that the (NAS) gauntlet was laid down.

Lurking in my basement is a very well used SolidRun MACCHIATObin system. It ticked a number of important boxes: it’s Arm (aarch64) based, has 3 SATA ports, doesn’t consume tons of power and is quiet. It struck me that this would be a great platform to try out Rockstor on aarch64.

The plan

Much like what was described in the Building a NAS with Ten64 and Rockstor article, I decided to run Rockstor on my MACCHIATObin system under KVM. This would prevent me from having to change the host OS to OpenSUSE (from Fedora). The Rockstor 4 ARM64EFI installation recipe is based on OpenSUSE. As a first step, I had to make certain that it was possible to use KVM on MACCHIATObin. After a bit of reading I had my answer and proceeded to get OpenSUSE Leap 15.2 running under KVM with minimal fuss.

Next, I followed the detailed instructions at the Rockstor site for setting up the version 4 beta on top of an existing OpenSUSE 15.2 installation. For this, I followed instructions from the following pages:

https://forum.rockstor.com/t/rockstor-4-installer-recipe-call-for-beta-testers/7237

https://github.com/rockstor/rockstor-installer/blob/master/README.md

https://forum.rockstor.com/t/rockstor-4-pi-build-first-observations/7367/2

For storage, I relied on spinning rust. I attached 2 x 4TB Seagate IronWolf SATA drives to the MACCHIATObin. Overall the installation of Rockstar went very smoothly and with no real hiccups to speak of. At the end of the installation I simply pointed my web browser to the Rockstor node and logged in. Via the Rockstor UI, I parepared the 2 Seagate disks and created a storage pool configured with RAID1 (mirroring). Finally, I defined NFS, SMB and SFTP shares against the storage pool in order to start using the system - in anger.

Below are a few photos of what the rig looks like.

For performance testing, I reliped upon some tried and trusted tools: iperf, dd and iozone. I NFS mounted the Rockstor server from a dual socket POWER9 system running CentOS Stream 8. And of course, my switch in this case connecting it all together is the Turris Omnia (Gigabit Ethernet throughout)

Some data from the sample runs follows:

A quick bandwidth test with iperf

a. Client side output

]$ ./iperf3 -c 192.168.1.YYY
Connecting to host 192.168.1.YYY, port 5201
[  4] local 192.168.1.XXX port 33738 connected to 192.168.1.YYY port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   114 MBytes   959 Mbits/sec    0    444 KBytes
[  4]   1.00-2.00   sec   112 MBytes   942 Mbits/sec    0    465 KBytes
[  4]   2.00-3.00   sec   112 MBytes   941 Mbits/sec    0    488 KBytes
[  4]   3.00-4.00   sec   113 MBytes   948 Mbits/sec    0    667 KBytes
[  4]   4.00-5.00   sec   112 MBytes   942 Mbits/sec    0    700 KBytes
[  4]   5.00-6.00   sec   112 MBytes   940 Mbits/sec    0    732 KBytes
[  4]   6.00-7.00   sec   112 MBytes   937 Mbits/sec    0    882 KBytes
[  4]   7.00-8.00   sec   112 MBytes   941 Mbits/sec    0    926 KBytes
[  4]   8.00-9.00   sec   112 MBytes   941 Mbits/sec    0    970 KBytes
[  4]   9.00-10.00  sec   112 MBytes   941 Mbits/sec    0   1017 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1.10 GBytes   943 Mbits/sec    0             sender
[  4]   0.00-10.00  sec  1.09 GBytes   940 Mbits/sec                  receiver

iperf Done.

b. Server side output

 # ./iperf3 -s
warning: this system does not seem to support IPv6 - trying IPv4
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.1.XXX, port 33736
[  5] local 192.168.1.YYY port 5201 connected to 192.168.1.XXX port 33738
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   108 MBytes   901 Mbits/sec
[  5]   1.00-2.00   sec   112 MBytes   938 Mbits/sec
[  5]   2.00-3.00   sec   112 MBytes   941 Mbits/sec
[  5]   3.00-4.00   sec   112 MBytes   938 Mbits/sec
[  5]   4.00-5.00   sec   112 MBytes   941 Mbits/sec
[  5]   5.00-6.00   sec   112 MBytes   941 Mbits/sec
[  5]   6.00-7.00   sec   112 MBytes   936 Mbits/sec
[  5]   7.00-8.00   sec   112 MBytes   941 Mbits/sec
[  5]   8.00-9.00   sec   112 MBytes   941 Mbits/sec
[  5]   9.00-10.00  sec   112 MBytes   941 Mbits/sec
[  5]  10.00-10.05  sec  4.96 MBytes   941 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.05  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.05  sec  1.09 GBytes   936 Mbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

Write 8GB to mountpoint with dd

$ time dd if=/dev/zero of=/mnt/raktar/test_dd bs=8K count=1024000
1024000+0 records in
1024000+0 records out
8388608000 bytes (8.4 GB, 7.8 GiB) copied, 76.8675 s, 109 MB/s

real	1m16.872s
user	0m0.925s
sys	0m2.584s

Write with 4GB to mountpoint with dd

$ time dd if=/dev/zero of=/mnt/raktar/test_dd bs=4K count=1024000
1024000+0 records in
1024000+0 records out
4194304000 bytes (4.2 GB, 3.9 GiB) copied, 39.5062 s, 106 MB/s

real	0m39.509s
user	0m0.815s
sys	0m2.093s

Thrashing the spinning rust with iozone

$ time ./iozone -a -R -c -f /mnt/raktar/testfile
	Iozone: Performance Test of File I/O
	        Version $Revision: 3.489 $
		Compiled for 64 bit mode.
		Build: linux-powerpc64 

	Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
	             Al Slater, Scott Rhine, Mike Wisner, Ken Goss
	             Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
	             Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
	             Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
	             Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
	             Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
	             Vangel Bojaxhi, Ben England, Vikentsi Lapa,
	             Alexey Skidanov, Sudhir Kumar.

	Run began: Sat Mar  6 15:06:52 2021

	Auto Mode
	Excel chart generation enabled
	Include close in write timing
	Command line used: ./iozone -a -R -c -f /mnt/raktar/testfile
	Output is in kBytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 1024 kBytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride                                    
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
              64       4      766      512   161629   160374   177677     1239   161629       417     59867     1201      512   163699   170672
              64       8     1318      665   162019   168847   165823     1612   111531       473    112892     1028      551   102074   103891
              64      16     1425      558    87901   104742   100620     1271   101764       557    102229     1556      550   177324   168002
              64      32      601      602   171108   180788   175355     1272   101113       563    111300     1224      604    91684    94960
              64      64     1480      601   163301   172982   173429     1551    94692       565    105941     1542      548   103053   181276
             128       4     3441     1191   297797   295665   302495     3197   188520      1236    186555     2446     1337   347758   199668
             128       8     2800      392   193899   342216   268999     1559   302495      1123    198708     3049     1101   199372   288205
             128      16     1246     1224   219148   333293   338761     2072   222971      1123    211966     3126     1201   326403   350711
             128      32     3066     1192   200939   356533   340264     3225   207141      1348    202913     2506     1324   228961   349797
             128      64     2307     1192   231529   344190   334331     3208   229353      1112    226164     3069     1198   242839   320748
             128     128     1021     1103   330828   343091   329812     3123   227796      1127    224932     3033     1198   343091   259633
             256       4     5550     2182   561637   580461   552960     5128   381597      2449    323630     3618     2407   367241   587127
             256       8     2831     2038   375328   582983   621448     4151   585845      2472    641497     6170     1371   603630   609455
             256      16     5321     2419   735537   750970   561343     6295   707430      2225    624339     6049     2382   632429   688381
             256      32     4422     2200   658411   668246   718796     6395   688381      2413    595594     4987     2400   773140   807442
             256      64     6770     2399   785010   800221   686620     5007   717355      2429    609455     4842      995   654798   738065
             256     128     6178     2180   663292   654399   684432     5100   658008      2247    668246     5947     2047   635047   678807
             256     256     4612     2035   670332   748875   644965     6361   727070      2051    810490     7763     2052   686182   697776
             512       4     8478     4432  1155924  1160923   911184    10157   966113      2508    936209     5425     3468   982467   994294
             512       8    12369     4387  1099128  1169140  1135150    10039  1073309      3805   1053821     9847     4367  1140576  1152822
             512      16    10894     4782  1355045  1380303  1166599    10031  1276891      4449   1227269     7313     4324  1383861  1518938
             512      32     7773     3488  1248678  1329871  1308798    12665  1391032      5389   1199171     7036     4065  1422357  1471074
             512      64    10716     4810  1375881  1458090  1276891     9979  1286067      4868   1216149     7310     3990  1537424  1528669
             512     128    13507     4776  1417662  1429934  1242896     8310  1347393      4543   1115688     7989     3484  1127995  1289930
             512     256     5408     4314  1369738  1382969  1296159    12632  1333173      4149   1458090     7946     4821  1245780  1368865
             512     512     5106     3765  1163439  1225169  1296159    10144  1387437      4428   1273861     9715     3742  1355045  1413929
            1024       4    11637     8251  1600327  1609925  1340164    14150  1326505      8366   1220575    13853     7632  1434138  1594978
            1024       8    13016     8906  1874871  1875690  1762539    14240  1633188      8954   1695065    16636     7555  1790460  2004366
            1024      16    15004     8720  2102484  2120127  1976691    12715  1968537      7704   1614767    19160     6546  2052253  2206161
            1024      32    14118     9301  2236023  2360124  2321847    13136  2106609      9533   2169388    19593     8130  2198257  2338280
            1024      64    15127     7499  2326879  2409105  2155236    14981  2286009      8050   2391665    19566     9624  2120127  2516377
            1024     128    13110     8832  2360124  2409105  2503178    14925  2265510      8879   2321847    15910     7608  2165014  2478620
            1024     256    14024     9611  2432298  2450338  2485792    12700  2286009      8327   2316837    15960     8814  2514903  2546213
            1024     512    15271     8842  1783768  2169388  2290886    12649  2337008      5856   2003431    12456     8124  2573677  2632033
            1024    1024    14946     9055  2592318  2751762  2767722    13950  2426801      8293   2503178    13786     8807  2516377  2111788
            2048       4    20893    14967  2214011  2240576  1747411    22968  1739978     16843   1762469    27893    13966  2124220  2194777
            2048       8    23866    15022  2752473  2592166  2386884    22881  2359349     16556   2434922    20044    14155  2760434  2809182
            2048      16    15582    16221  3346546  3384788  3061485    22716  2938934     16490   3074635    28567    12449  3374151  3229534
            2048      32    22680    14979  3777778  3841986  3717289    25448  3561619     18210   3334853    27811    16144  3901313  3929871
            2048      64    21076    15019  3999400  3922692  3806235    20393  3749743     15463   3799501    32083    14991  4038890  4240255
            2048     128    19838    16169  4229816  4250747  4164199    16931  3864455     17908   4162181    22159    15187  4248644  4453499
            2048     256    12745     9084  4259178  4196751  4293237    20211  3812993     17887   4054139    28122    13249  4154130  4441984
            2048     512    26016    15202  4267642  4229816  4453499    24891  4154130     16492   4444282    28216    15017  4188565  4405533
            2048    1024    23457    19229  3917325  4130162  4154130    20590  4196751     11737   3650932    24566    15136  2482779  2659582
            2048    2048    16566    16249  2494314  2635106  2653011    23273  1933883     15579   2176427    25411    16236  2133188  1906837
            4096       4    38048    24611  2298370  2162402  1788674    35265  1640426     32323   1618939    40241    24720  2269226  2391139
            4096       8    39802    26380  3503374  3519162  3327622    37629  2294686     34059   2233238    46115    19740  2363506  2585847
            4096      16    23628    27804  2970339  3013586  2756362    35645  3821310     33360   3817065    45585    28058  4149921  4280194
            4096      32    38047    26565  3290024  3543113  3117487    38176  5761565     32992   5818150    44147    22494  4934183  6847827
            4096      64    34621    26347   114155  5007531  4791076    33686  4270618     33444   3706707    45574    26252  3558525  3621537
            4096     128    29910    27839  3533639  3580776  3586757    38303  3489144     40310   4735608    37164    18835  3717133  3680502
            4096     256    33432    26663  4934183  5101205  5424983    40428  3531460     22529   4906002    40634    20792  3343814  3433360
            4096     512    32295    20977  5883908  5424983  5468151    37088  3664018     27316   5332372    36376    15273  5217395  5558378
            4096    1024    40430    20977  3682869  3385325  3647681    41777  3771811     31407   3792628    36666    27891  4157956  3706707
            4096    2048    21802    26450  3486312  3911800  3719547    38135  4982841     33183   5375755    36398    23427  4904602  5020702
            4096    4096    32822    26575  5382492  5190597  5263753    37140  3404782     27528   4965558    39529    26352 14427096 14271297
            8192       4    41208    34571  2528346  2205119  1642685    37514  1790147     61064   1945315    49482    29458  3030545  3136499
            8192       8    45073    31839  4171488  4498640  3852936    49560  3365696     63898   3840876    42399    31969  3944043  4164914
            8192      16    47033    29557  4500997  4373801  3995873    42362  3912159     63290   4067778    26672    44661   111474  5271672
            8192      32    53750    34202  2863357  5287898  2123356    30104  4662231     62863   5000831    58402    30695  6480761  6809564
            8192      64    42802    34921  5901904  5971654  5602613    37601  5313247     67584   5723946    40963    38040  5712526  5871647
            8192     128    43970    34463  5817959  5988306  6071906    46729  5454094     67699   6140266    42889    37885  5785631  5975808
            8192     256    43887    33123  5910025  5940679  6054786    43865  5745001     68143   5988306    43016    29500  5814021  6196743
            8192     512    41983    34669  5944791  5940679  6239504    44618  5696426     66238   6947247    38716    31906  6145757  5997714
            8192    1024    39435    37870  6399884  5830794  6067617    41948  6094523     62955   7937463    40642    28386  6924845  7735533
            8192    2048    39275    25258  6072979  7427838  7440706    43609  6374949     35618   8743368    40711    31933  5336352  5142283
            8192    4096    46473    28679  4946832  5126937  4876623    33948  5097274     47802   8086916    39041    37522  5010310  4876623
            8192    8192    40918    29668  4998648  5056765  4979812    40625  5088216     29293   5688881    51283    33280  4929091  4710810
           16384       4    58065    45957  3120803  3071013  2228850    55784  2116542    103626   2184496    57936    42028  2900803  3032391
           16384       8    57669    47322  4914218  4948188  2688155    51293  3717820    131949   3654745    51302    43187  4729877  4761338
           16384      16    52989    44720  5923681  5845080  4901250    60184  4996397    139152   5012066    58686    47250  5811959  5964297
           16384      32    59565    55140  7817198  7717122  6889995    41780  6452650    125794   6537361    60210    39198  8233027  7481050
           16384      64    48498    47273  7724930  7853828  7350615    59972  8312700    147326   8484089    60358    47194  7765083  7827884
           16384     128    52363    46150  8257760  7933620  7692072    51845  7877236    126376   9277883    58286    50090  7926299  8270680
           16384     256    57480    47199  9199633  8225143  8003847    53723  7406867    151085   8459025    54447    36649  9729770  9786581
           16384     512    56439    58500  8533606  8240925  8673628    53771    94848     49413   9267872    60466    51694  8623560  8351087
           16384    1024    56339    45925  7182365  8291637  8291637    67496 10304615     85962  19068585    46184    53250  8351087  8659420
           16384    2048    69858    50186 11009550 10290727 10272267    55174  8168437     91274  12859651    45328    45915  8286637  8467363
           16384    4096    57811    48674  8919040  8696680  8692280    67235  9334595     81668  10659461    60520    47228  8359214  8346016
           16384    8192    54791    39197  8240925  9984224  9056567    58229  7628034     63197   8440323    56615    46231  7460745  7904418
           16384   16384    52309    48775  7239873  7606082  7692072    59469  6798654     47149   7010184    66211    50302  7135381  7331011
           32768      64    68452    63438 10767866 11145983 10587860    66188  8553152    196836  10262914    70797    63662 10445431 12477655
           32768     128    74330    56587 10937536 11203222 11976409    68123 10425622    270919  12794771    69441    63307 10560200 11665355
           32768     256    73302    59555 10796626 11473510 12383220    65037 10701629    164278  12277028    68121    58698 10958466 12068958
           32768     512    72445    60667 11631788 12359834 13567777    72396 10879527    171307  12724879    69853    61717  9849567 11800570
           32768    1024    71159    67179  9858045 10186091 11123431    65485 11148695    275460  12961285    69705    61881 10151482 10709134
           32768    2048    73047    60915 11582774 12755585 13128427    72514 11222432    219878  20532259    68973    62478 10807663 12337643
           32768    4096    68780    64220 10252961 10835782 11594500    62992 10227021    132334  18702216    69636    63082  9683705 12305608
           32768    8192    70705    63127  9825626 10875222 11311097    70012 10246845    118346  10694135    64358    64404  9902793 12516290
           32768   16384    72368    59947  9375862  9728947  9881434    67853  8419541     89947   8839196    60092    58554   112629  9533907
           65536      64    82509    79055 10401087 11421453 11023864    77645  9652016    440101  10654675    77948    77407 10149535 11630216
           65536     128    84080    78150 10842965 12055622 12293394    82366 10710305    535691  12327576    79992    77452 11395884 13320434
           65536     256    80498    72047 10597577 12559643 12197385    80466 11233809    290491  11908856    81132    76306 10447339 12629467
           65536     512    81152    74496 10827162 12076278 13295950    75815 11061126    460655  12509344    78097    76545 11636124 14444598
           65536    1024    83013    71312 11366197 12867728 13396389    80359 11770154    247265  12900943    81124    76239 11403448 13896193
           65536    2048    83236    76272   119806 11270196 10942676    82650 11677648    247548  13306893    82129    78141 12022929 13579712
           65536    4096    82357    76272 11948197 12920955 14283968    82385 11696531    258841  22230669    79791    78366 12113531 14039564
           65536    8192    85060    77279 13008396 13655947 14105847    78461 12654468    222259  14807195    78973    77165 11548130 12850283
           65536   16384    82554    77219 11929012 11872336 11831964    79076 11770154    176494  12284603    81136    76313 11644504 11700514
          131072      64    88544    83386 11835991 12145325 11670659    87840 11391587    419979  11686040    88820    82872 11811578 12481740
          131072     128    88630    81787 12860445 13091354 13302644    86476 12881840    467063  13230293    88445    79831 12765186 13511230
          131072     256    90497    84498 12881840 13012339 12985292    87772 13108523    414247  13319081    89824    83386 12600753 13263809
          131072     512    88596    82448 12769930 12958357 13536514    85905 13670482    416191  14118093    83436    84523 13137656 13840848
          131072    1024    89407    78835 13194095 13263809 13855499    84908 13827619    442800  13946536    82538    78850 13371887 14115193
          131072    2048    88707    84477 13946536 14042363 14785216    86420 14674318    457513  14565072    86827    83928 12639283 13414303
          131072    4096    89182    83355 13417250 13516877 13986987    85565 13788430    497857  14016943    83531    83394 13105710 13740526
          131072    8192    88133    82921 13332647 13313597 13712081    86749 13703194    415595   1591452    90140    82009 13259651 13324246
          131072   16384    88919    83384 12569350 12383894 12445289    87123 12239423    273752  12745062    87518    81834 11830642 11918875
          262144      64    92046    89238 12412246 12322798 11471432    90962 11729024    910557  11846678    92929    85955 12535646 12852148
          262144     128    93163    87728 13213567 13137311 12926184    90851 12826360    900170  13110368    90907    83813   106392 11100705
          262144     256    89530    87436 13288465 13291356 13391725    89797 12979437    924856  13410345    91811    88369 13264418 13639078
          262144     512    88940    83024 13982505 13753728 14127128    91268 13858259    863287  13773196    91523    87486 13691728 14047711
          262144    1024    94344    87151 14090737 13934836 14349852    91634 14453412    884871  14068741    87858    87861 14205245 14639112
          262144    2048    93323    83784 14630931 14471865 15098659    89320 14816258    837400  14496670    91384    87290 14246105 14644962
          262144    4096    91123    85736 14333390 14121866 14464820    86524 14602367    831004  14610711    92440    87452 14302812 14770479
          262144    8192    93882    88322 14379692 14047711 14591902    89510 14455122    681831  14176855    89511    87001 13901010 14244628
          262144   16384    89497    85428 12697054 12364092 12420940    91501 12849594    568459  13704015    90625    80203 12796951 12740160
          524288      64    94358    91054 13009600 12589457 11779147    91311 11947459   1255265  11876481    89799    87753 12691250 12857393
          524288     128    92317    88896 13905388 13609686 13317259    92937 13277137   1300817  13449123    92321    86529 13869692 14155027
          524288     256    95697    90554 13770592 13504875 13470626    92289 13213552   1065322  13137375    93384    92184 13450851 13637369
          524288     512    92883    88220 14063505 13708355 13953121    92841 14061617   1341596  14121485    91142    90098 14090720 14200640
          524288    1024    95296    89887 14497129 14125204 14520774    91158 14661640   1315825  14669954    93129    88311 14578534 14875572
          524288    2048    95723    88312 14628576 14222223 14656851    90120 14689160   1008900  14199907    93524    90571 14814641 15050689
          524288    4096    96008    89645 14907236 14518761 15025186    93152 14806661   1239205  14519144    92925    90417 14841838 15099157
          524288    8192    94085    90818 14516748 14098760 14655484    93151 14690436    888697  14513490    88415    88908 14614285 14856578
          524288   16384    93354    90226 12920169 12536489 12795299    92156 12896679    798495  12860025    92265    89190 12914478 13190646

iozone test complete.
Excel output is below:

"Writer report"
        "4"  "8"  "16"  "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"64"   766  1318  1425  601  1480 
"128"   3441  2800  1246  3066  2307  1021 
"256"   5550  2831  5321  4422  6770  6178  4612 
"512"   8478  12369  10894  7773  10716  13507  5408  5106 
"1024"   11637  13016  15004  14118  15127  13110  14024  15271  14946 
"2048"   20893  23866  15582  22680  21076  19838  12745  26016  23457  16566 
"4096"   38048  39802  23628  38047  34621  29910  33432  32295  40430  21802  32822 
"8192"   41208  45073  47033  53750  42802  43970  43887  41983  39435  39275  46473  40918 
"16384"   58065  57669  52989  59565  48498  52363  57480  56439  56339  69858  57811  54791  52309 
"32768"   0  0  0  0  68452  74330  73302  72445  71159  73047  68780  70705  72368 
"65536"   0  0  0  0  82509  84080  80498  81152  83013  83236  82357  85060  82554 
"131072"   0  0  0  0  88544  88630  90497  88596  89407  88707  89182  88133  88919 
"262144"   0  0  0  0  92046  93163  89530  88940  94344  93323  91123  93882  89497 
"524288"   0  0  0  0  94358  92317  95697  92883  95296  95723  96008  94085  93354 

"Re-writer report"
        "4"  "8"  "16"  "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"64"   512  665  558  602  601 
"128"   1191  392  1224  1192  1192  1103 
"256"   2182  2038  2419  2200  2399  2180  2035 
"512"   4432  4387  4782  3488  4810  4776  4314  3765 
"1024"   8251  8906  8720  9301  7499  8832  9611  8842  9055 
"2048"   14967  15022  16221  14979  15019  16169  9084  15202  19229  16249 
"4096"   24611  26380  27804  26565  26347  27839  26663  20977  20977  26450  26575 
"8192"   34571  31839  29557  34202  34921  34463  33123  34669  37870  25258  28679  29668 
"16384"   45957  47322  44720  55140  47273  46150  47199  58500  45925  50186  48674  39197  48775 
"32768"   0  0  0  0  63438  56587  59555  60667  67179  60915  64220  63127  59947 
"65536"   0  0  0  0  79055  78150  72047  74496  71312  76272  76272  77279  77219 
"131072"   0  0  0  0  83386  81787  84498  82448  78835  84477  83355  82921  83384 
"262144"   0  0  0  0  89238  87728  87436  83024  87151  83784  85736  88322  85428 
"524288"   0  0  0  0  91054  88896  90554  88220  89887  88312  89645  90818  90226 

"Reader report"
        "4"  "8"  "16"  "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"64"   161629  162019  87901  171108  163301 
"128"   297797  193899  219148  200939  231529  330828 
"256"   561637  375328  735537  658411  785010  663292  670332 
"512"   1155924  1099128  1355045  1248678  1375881  1417662  1369738  1163439 
"1024"   1600327  1874871  2102484  2236023  2326879  2360124  2432298  1783768  2592318 
"2048"   2214011  2752473  3346546  3777778  3999400  4229816  4259178  4267642  3917325  2494314 
"4096"   2298370  3503374  2970339  3290024  114155  3533639  4934183  5883908  3682869  3486312  5382492 
"8192"   2528346  4171488  4500997  2863357  5901904  5817959  5910025  5944791  6399884  6072979  4946832  4998648 
"16384"   3120803  4914218  5923681  7817198  7724930  8257760  9199633  8533606  7182365  11009550  8919040  8240925  7239873 
"32768"   0  0  0  0  10767866  10937536  10796626  11631788  9858045  11582774  10252961  9825626  9375862 
"65536"   0  0  0  0  10401087  10842965  10597577  10827162  11366197  119806  11948197  13008396  11929012 
"131072"   0  0  0  0  11835991  12860445  12881840  12769930  13194095  13946536  13417250  13332647  12569350 
"262144"   0  0  0  0  12412246  13213567  13288465  13982505  14090737  14630931  14333390  14379692  12697054 
"524288"   0  0  0  0  13009600  13905388  13770592  14063505  14497129  14628576  14907236  14516748  12920169 

"Re-Reader report"
        "4"  "8"  "16"  "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"64"   160374  168847  104742  180788  172982 
"128"   295665  342216  333293  356533  344190  343091 
"256"   580461  582983  750970  668246  800221  654399  748875 
"512"   1160923  1169140  1380303  1329871  1458090  1429934  1382969  1225169 
"1024"   1609925  1875690  2120127  2360124  2409105  2409105  2450338  2169388  2751762 
"2048"   2240576  2592166  3384788  3841986  3922692  4250747  4196751  4229816  4130162  2635106 
"4096"   2162402  3519162  3013586  3543113  5007531  3580776  5101205  5424983  3385325  3911800  5190597 
"8192"   2205119  4498640  4373801  5287898  5971654  5988306  5940679  5940679  5830794  7427838  5126937  5056765 
"16384"   3071013  4948188  5845080  7717122  7853828  7933620  8225143  8240925  8291637  10290727  8696680  9984224  7606082 
"32768"   0  0  0  0  11145983  11203222  11473510  12359834  10186091  12755585  10835782  10875222  9728947 
"65536"   0  0  0  0  11421453  12055622  12559643  12076278  12867728  11270196  12920955  13655947  11872336 
"131072"   0  0  0  0  12145325  13091354  13012339  12958357  13263809  14042363  13516877  13313597  12383894 
"262144"   0  0  0  0  12322798  13137311  13291356  13753728  13934836  14471865  14121866  14047711  12364092 
"524288"   0  0  0  0  12589457  13609686  13504875  13708355  14125204  14222223  14518761  14098760  12536489 

"Random read report"
        "4"  "8"  "16"  "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"64"   177677  165823  100620  175355  173429 
"128"   302495  268999  338761  340264  334331  329812 
"256"   552960  621448  561343  718796  686620  684432  644965 
"512"   911184  1135150  1166599  1308798  1276891  1242896  1296159  1296159 
"1024"   1340164  1762539  1976691  2321847  2155236  2503178  2485792  2290886  2767722 
"2048"   1747411  2386884  3061485  3717289  3806235  4164199  4293237  4453499  4154130  2653011 
"4096"   1788674  3327622  2756362  3117487  4791076  3586757  5424983  5468151  3647681  3719547  5263753 
"8192"   1642685  3852936  3995873  2123356  5602613  6071906  6054786  6239504  6067617  7440706  4876623  4979812 
"16384"   2228850  2688155  4901250  6889995  7350615  7692072  8003847  8673628  8291637  10272267  8692280  9056567  7692072 
"32768"   0  0  0  0  10587860  11976409  12383220  13567777  11123431  13128427  11594500  11311097  9881434 
"65536"   0  0  0  0  11023864  12293394  12197385  13295950  13396389  10942676  14283968  14105847  11831964 
"131072"   0  0  0  0  11670659  13302644  12985292  13536514  13855499  14785216  13986987  13712081  12445289 
"262144"   0  0  0  0  11471432  12926184  13391725  14127128  14349852  15098659  14464820  14591902  12420940 
"524288"   0  0  0  0  11779147  13317259  13470626  13953121  14520774  14656851  15025186  14655484  12795299 

"Random write report"
        "4"  "8"  "16"  "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"64"   1239  1612  1271  1272  1551 
"128"   3197  1559  2072  3225  3208  3123 
"256"   5128  4151  6295  6395  5007  5100  6361 
"512"   10157  10039  10031  12665  9979  8310  12632  10144 
"1024"   14150  14240  12715  13136  14981  14925  12700  12649  13950 
"2048"   22968  22881  22716  25448  20393  16931  20211  24891  20590  23273 
"4096"   35265  37629  35645  38176  33686  38303  40428  37088  41777  38135  37140 
"8192"   37514  49560  42362  30104  37601  46729  43865  44618  41948  43609  33948  40625 
"16384"   55784  51293  60184  41780  59972  51845  53723  53771  67496  55174  67235  58229  59469 
"32768"   0  0  0  0  66188  68123  65037  72396  65485  72514  62992  70012  67853 
"65536"   0  0  0  0  77645  82366  80466  75815  80359  82650  82385  78461  79076 
"131072"   0  0  0  0  87840  86476  87772  85905  84908  86420  85565  86749  87123 
"262144"   0  0  0  0  90962  90851  89797  91268  91634  89320  86524  89510  91501 
"524288"   0  0  0  0  91311  92937  92289  92841  91158  90120  93152  93151  92156 

"Backward read report"
        "4"  "8"  "16"  "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"64"   161629  111531  101764  101113  94692 
"128"   188520  302495  222971  207141  229353  227796 
"256"   381597  585845  707430  688381  717355  658008  727070 
"512"   966113  1073309  1276891  1391032  1286067  1347393  1333173  1387437 
"1024"   1326505  1633188  1968537  2106609  2286009  2265510  2286009  2337008  2426801 
"2048"   1739978  2359349  2938934  3561619  3749743  3864455  3812993  4154130  4196751  1933883 
"4096"   1640426  2294686  3821310  5761565  4270618  3489144  3531460  3664018  3771811  4982841  3404782 
"8192"   1790147  3365696  3912159  4662231  5313247  5454094  5745001  5696426  6094523  6374949  5097274  5088216 
"16384"   2116542  3717820  4996397  6452650  8312700  7877236  7406867  94848  10304615  8168437  9334595  7628034  6798654 
"32768"   0  0  0  0  8553152  10425622  10701629  10879527  11148695  11222432  10227021  10246845  8419541 
"65536"   0  0  0  0  9652016  10710305  11233809  11061126  11770154  11677648  11696531  12654468  11770154 
"131072"   0  0  0  0  11391587  12881840  13108523  13670482  13827619  14674318  13788430  13703194  12239423 
"262144"   0  0  0  0  11729024  12826360  12979437  13858259  14453412  14816258  14602367  14455122  12849594 
"524288"   0  0  0  0  11947459  13277137  13213552  14061617  14661640  14689160  14806661  14690436  12896679 

"Record rewrite report"
        "4"  "8"  "16"  "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"64"   417  473  557  563  565 
"128"   1236  1123  1123  1348  1112  1127 
"256"   2449  2472  2225  2413  2429  2247  2051 
"512"   2508  3805  4449  5389  4868  4543  4149  4428 
"1024"   8366  8954  7704  9533  8050  8879  8327  5856  8293 
"2048"   16843  16556  16490  18210  15463  17908  17887  16492  11737  15579 
"4096"   32323  34059  33360  32992  33444  40310  22529  27316  31407  33183  27528 
"8192"   61064  63898  63290  62863  67584  67699  68143  66238  62955  35618  47802  29293 
"16384"   103626  131949  139152  125794  147326  126376  151085  49413  85962  91274  81668  63197  47149 
"32768"   0  0  0  0  196836  270919  164278  171307  275460  219878  132334  118346  89947 
"65536"   0  0  0  0  440101  535691  290491  460655  247265  247548  258841  222259  176494 
"131072"   0  0  0  0  419979  467063  414247  416191  442800  457513  497857  415595  273752 
"262144"   0  0  0  0  910557  900170  924856  863287  884871  837400  831004  681831  568459 
"524288"   0  0  0  0  1255265  1300817  1065322  1341596  1315825  1008900  1239205  888697  798495 

"Stride read report"
        "4"  "8"  "16"  "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"64"   59867  112892  102229  111300  105941 
"128"   186555  198708  211966  202913  226164  224932 
"256"   323630  641497  624339  595594  609455  668246  810490 
"512"   936209  1053821  1227269  1199171  1216149  1115688  1458090  1273861 
"1024"   1220575  1695065  1614767  2169388  2391665  2321847  2316837  2003431  2503178 
"2048"   1762469  2434922  3074635  3334853  3799501  4162181  4054139  4444282  3650932  2176427 
"4096"   1618939  2233238  3817065  5818150  3706707  4735608  4906002  5332372  3792628  5375755  4965558 
"8192"   1945315  3840876  4067778  5000831  5723946  6140266  5988306  6947247  7937463  8743368  8086916  5688881 
"16384"   2184496  3654745  5012066  6537361  8484089  9277883  8459025  9267872  19068585  12859651  10659461  8440323  7010184 
"32768"   0  0  0  0  10262914  12794771  12277028  12724879  12961285  20532259  18702216  10694135  8839196 
"65536"   0  0  0  0  10654675  12327576  11908856  12509344  12900943  13306893  22230669  14807195  12284603 
"131072"   0  0  0  0  11686040  13230293  13319081  14118093  13946536  14565072  14016943  1591452  12745062 
"262144"   0  0  0  0  11846678  13110368  13410345  13773196  14068741  14496670  14610711  14176855  13704015 
"524288"   0  0  0  0  11876481  13449123  13137375  14121485  14669954  14199907  14519144  14513490  12860025 

"Fwrite report"
        "4"  "8"  "16"  "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"64"   1201  1028  1556  1224  1542 
"128"   2446  3049  3126  2506  3069  3033 
"256"   3618  6170  6049  4987  4842  5947  7763 
"512"   5425  9847  7313  7036  7310  7989  7946  9715 
"1024"   13853  16636  19160  19593  19566  15910  15960  12456  13786 
"2048"   27893  20044  28567  27811  32083  22159  28122  28216  24566  25411 
"4096"   40241  46115  45585  44147  45574  37164  40634  36376  36666  36398  39529 
"8192"   49482  42399  26672  58402  40963  42889  43016  38716  40642  40711  39041  51283 
"16384"   57936  51302  58686  60210  60358  58286  54447  60466  46184  45328  60520  56615  66211 
"32768"   0  0  0  0  70797  69441  68121  69853  69705  68973  69636  64358  60092 
"65536"   0  0  0  0  77948  79992  81132  78097  81124  82129  79791  78973  81136 
"131072"   0  0  0  0  88820  88445  89824  83436  82538  86827  83531  90140  87518 
"262144"   0  0  0  0  92929  90907  91811  91523  87858  91384  92440  89511  90625 
"524288"   0  0  0  0  89799  92321  93384  91142  93129  93524  92925  88415  92265 

"Re-Fwrite report"
        "4"  "8"  "16"  "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"64"   512  551  550  604  548 
"128"   1337  1101  1201  1324  1198  1198 
"256"   2407  1371  2382  2400  995  2047  2052 
"512"   3468  4367  4324  4065  3990  3484  4821  3742 
"1024"   7632  7555  6546  8130  9624  7608  8814  8124  8807 
"2048"   13966  14155  12449  16144  14991  15187  13249  15017  15136  16236 
"4096"   24720  19740  28058  22494  26252  18835  20792  15273  27891  23427  26352 
"8192"   29458  31969  44661  30695  38040  37885  29500  31906  28386  31933  37522  33280 
"16384"   42028  43187  47250  39198  47194  50090  36649  51694  53250  45915  47228  46231  50302 
"32768"   0  0  0  0  63662  63307  58698  61717  61881  62478  63082  64404  58554 
"65536"   0  0  0  0  77407  77452  76306  76545  76239  78141  78366  77165  76313 
"131072"   0  0  0  0  82872  79831  83386  84523  78850  83928  83394  82009  81834 
"262144"   0  0  0  0  85955  83813  88369  87486  87861  87290  87452  87001  80203 
"524288"   0  0  0  0  87753  86529  92184  90098  88311  90571  90417  88908  89190 

"Fread report"
        "4"  "8"  "16"  "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"64"   163699  102074  177324  91684  103053 
"128"   347758  199372  326403  228961  242839  343091 
"256"   367241  603630  632429  773140  654798  635047  686182 
"512"   982467  1140576  1383861  1422357  1537424  1127995  1245780  1355045 
"1024"   1434138  1790460  2052253  2198257  2120127  2165014  2514903  2573677  2516377 
"2048"   2124220  2760434  3374151  3901313  4038890  4248644  4154130  4188565  2482779  2133188 
"4096"   2269226  2363506  4149921  4934183  3558525  3717133  3343814  5217395  4157956  4904602  14427096 
"8192"   3030545  3944043  111474  6480761  5712526  5785631  5814021  6145757  6924845  5336352  5010310  4929091 
"16384"   2900803  4729877  5811959  8233027  7765083  7926299  9729770  8623560  8351087  8286637  8359214  7460745  7135381 
"32768"   0  0  0  0  10445431  10560200  10958466  9849567  10151482  10807663  9683705  9902793  112629 
"65536"   0  0  0  0  10149535  11395884  10447339  11636124  11403448  12022929  12113531  11548130  11644504 
"131072"   0  0  0  0  11811578  12765186  12600753  13137656  13371887  12639283  13105710  13259651  11830642 
"262144"   0  0  0  0  12535646  106392  13264418  13691728  14205245  14246105  14302812  13901010  12796951 
"524288"   0  0  0  0  12691250  13869692  13450851  14090720  14578534  14814641  14841838  14614285  12914478 

"Re-Fread report"
        "4"  "8"  "16"  "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"64"   170672  103891  168002  94960  181276 
"128"   199668  288205  350711  349797  320748  259633 
"256"   587127  609455  688381  807442  738065  678807  697776 
"512"   994294  1152822  1518938  1471074  1528669  1289930  1368865  1413929 
"1024"   1594978  2004366  2206161  2338280  2516377  2478620  2546213  2632033  2111788 
"2048"   2194777  2809182  3229534  3929871  4240255  4453499  4441984  4405533  2659582  1906837 
"4096"   2391139  2585847  4280194  6847827  3621537  3680502  3433360  5558378  3706707  5020702  14271297 
"8192"   3136499  4164914  5271672  6809564  5871647  5975808  6196743  5997714  7735533  5142283  4876623  4710810 
"16384"   3032391  4761338  5964297  7481050  7827884  8270680  9786581  8351087  8659420  8467363  8346016  7904418  7331011 
"32768"   0  0  0  0  12477655  11665355  12068958  11800570  10709134  12337643  12305608  12516290  9533907 
"65536"   0  0  0  0  11630216  13320434  12629467  14444598  13896193  13579712  14039564  12850283  11700514 
"131072"   0  0  0  0  12481740  13511230  13263809  13840848  14115193  13414303  13740526  13324246  11918875 
"262144"   0  0  0  0  12852148  11100705  13639078  14047711  14639112  14644962  14770479  14244628  12740160 
"524288"   0  0  0  0  12857393  14155027  13637369  14200640  14875572  15050689  15099157  14856578  13190646 

real	10m25.537s
user	0m1.316s
sys	0m14.994s