Well that post is getting longer and longer and here are some additional notes:
- SATA drives can plug directly into a SAS backplane. Wow that is pretty cool the so if you have an SAS motherboard, you just need SAS cables to the backplane
Storage with SSDs
The future of course is moving to PCI Express direct connect for drives. There isn’t a reason really to get to this and a good experiment for this project. Amazon now has a sea of adapters that let you connect the M.2 directly into PCI Express. This works pretty well for these huge server boards with 7 slots of PCI Express and should be more reliable that having connectors and then having:
- SATA bottleneck (6Gbps for SATA 3.0). Moving any faster would be a big problem for SATA. That by the way is about 560MBps effective according to Anandtech. When we built a RAID0 configuration with two SATA SSD drives, we in fact got to a real 1GBps of transfer on our Aquarium PC, so this actually can happen at least at peak rates.
- SAS bottleneck (12Gbps). Can you believe we are talking about 12Gbps to a drive being a bottleneck
- PCI Express 3.0. There lots of math, but basically each lane is 1GBps of bandwidth so roughly speaking just an x1 PCI Express is 8Gbps . Wow. So an x4 is 4GBps or an amazing 32Gbps with effectively 1.56GBps of bandwidth.
- SATAe. This is an upcoming standard that basically stuffs PCi Express lanes into a SATA cable, so that you don’t need a PCI Express card adapter. Of course the cables will be unreliable, but its a neat idea. New motherboards like the ASUS Z87 are getting updated with SATAe drivers
- NVMe. There will also be a move from the ACHI software stack to something called NVMe which you need to build into the SATA SSDs (the SF3700 will have this in the consumer world shortly so look out for those. This will increase the number of IOPs because the latency is much shorter for accessing data. The soon to be shipping Intel PC3500 is a consumer grade PCI Express system (no adapter needed) with NVMe priced at $600 for 400GB
- M.2 B key vs. M key. Current SSds are either B-key or M-keyed. B-keyed means they support x2 PCI and M means x4. So, you want to get x4 as you get double the bandwidth out of them. Only a few SSDs are M-keyed like the new Samsung XP941 ($500 for 500GB from Newegg) which is a 1GBps SSD so needs x4. Most SSDs are 500MBps so can live with x2. Note that if you have an adapter that has say a pair of SSDs, then you need to double the number of lanes to sustain peak performance. This can make some sense as a Crucial M550 for instance is one third the cost per bit ($250 for 500GB
So what kinds of cards can you get to adapt if you just want to get the advantages of PCI Express now?
- PCI M.2 to PCI Express card. This is a $25 adapter and it is an x2 so it has a 750GBps bandwidth maximum and works with B-key SSds
- Lycom DT-120 PCI M.2 to PCI Express x4. $24 and this only supports M-key which means that it is for PCI M.2 cards which support PCI Express 4 lane whereas B-key only supports 2 lane. This would be a good choice for a monster boot drive that needs 1GBps boot times
- Micro SATA m.2 to PCI Express x4. This has two B-keyed slots so a good match for say a pair of Crucial M550s. It doubles the storage density effectively. This gives you double the density so say 1TB and if you RAID-0 them a cheap way to get 1GBps theoretically. Although some folks say that RAID-0 doesn’t work as effectively in the real world and you are better off with two logical drives and easier time access the data concurrently.
Net, net, you want SSDs in the future with SATAe and NVMe support the thing seems to be:
- to wait for the Intel PC3500 to ship and get 400GB of glory and incredible boot performance with x4 and NVMe. $600
- If you can’t wait and want super high bandwidth (but higher latency without NVMe), then get a pair of Crucial and a dual m2. to PCI Express x4. This is less than half the price of PC3500 at $600 for 1TB
- If you want blazing performance right now (but without NVMe), then its the Samsung XP941 on the Lycom x4 for $550 for 500GB.
