Ok last post on the file server topic. Vlad pointed out that there are two storage layers here. That is near line SAS plus a PC3500 when it’s available for high speed ssd.
So with 20 slots in a Norco 4220 you could easily add a caching Sata ssd layer as well. So you end up with 32-256GB DDR4, 2-4GBps NVMe SSD, 800MBps Sata ssd and then the main 140MBps hard disk layer.
So so what is a good caching ssd that zfs can use?
There are lots of consumer SSD reviews but not many enterprise SSD roundup but  Anandtech shows the Micron 500DC Is a good example. About $600 for 500MB. Seems expensive but they are 40% over-provisioned.
Tom’s IT Pro has a good overview including SAS drives and the pricing is such that I wonder if we should really recommend a SSD cache layer between the Intel PC3500 PCIExpress system and hard drives. The pricing is about the same as the NVMe layer and performance is about 4x less.