Rich's Tongfamily (Richtong.net add-on)

OK, now that we have our disk arrays in order, I was curious to see what performance is like. First some basics, with hard disks, performance is a little hard to understand. The short answer is that two things are going on, The first is, that you have the data under the disk head and you are blazing away reading it, how fast can it go? So when I’m using my disks, I’m seeing speeds on the order of 300MBps and with my current copy it is running at 200MBps, so the question is, is this a good or a bad number.

At first glance, it looks terrible compared with a typical SSD which can run at 3GB to 7GBps.

What is the maximum throughput for a single 20TB drive? 272MBps

The easy way to understand this is that a hard disk is spinning at probably 7200 rpm and there are typically on a 20TB Seagate EXOS hard drive, 10 platters, so you have 2TB per platter and then on each platter, there is typically. So if you have no “seeking” to different locations, you can get 272MBps out of something like this is how the math works:

1. 2700rpm / 60 second/min = 45 revolutions/second and on each revolution
2. 18TB, because 20TB drives are 20TiB where 1K is 1000 bytes not 1024 in a true KB, marketing that was started by Dell in the day, so 18TB/10 = 1.8TB/platter
3. 20 heads against 10 platters as they have two sides each or 900MB/platter side
4. So the 272MBps/20 heads is 13.6MBps/head
5. So with 45 revolutions per second, each track has 13.6MBps/45rev/s is 302Kb/track
6. Put another way there are 4,882,956,288 sectors/X20, so that means 244,147,814 sectors/heads on dual platters for 4K bytes per logical sector.

So if this is true, you would expect with RAID6 on a 6-drive array, you would get 4 effective disks (the other two are parity), so this should get you to 1GBps. In truth, you don’t get this because you can’t just read sequentially for very long. The fact is that in a short while, you have to “seek” that is physically move the head to another location.

I have never gotten anything close to that kind of sustained throughput. What this means

What matters more is IOPS at 166 IOPS/drive for 4K random writes

So the real limit is IOPS or I/O Operations per second which is all about how many different places can a drive get to in a second. For the Seagate 4K the best you can get with a high queue depth (that is a lot to keep you busy) is 1,000 IOPS write for 8 drives in RAID-6 (so 6 data drives at 166 IOPS write) for random 4K reads/write. If you are doing sequentially reads and writes it’s much higher more like 8,000 IOPS

Now the other confounding factor is caching, each drive typically has a 256MB cache and there is also caching in the operating system itself so it’s hard to figure out what should you get.

What’s all then about Synology Utilization?

One measure that Synology has is something called Utilization. Their definition is that means how long is there an operation active on a disk system. This gets to something called Queue Depth, which means how many outstanding I/O operations can be enqueued into a hard disk system. So an 80% utilization means that for any interval, say one second, 80% of that second, there was an active disk operation.

Reducing disk utilization with Memory Compression

OK, this is a surprising thing, if you get rid of Memory Compression, this improves disk utilization, and this improves performance. I think this is because it is pretty CPU-intensive to compress memory in real-time.

Real-world 12TB copy from 16TB RAID-6 with File Station

Here are the figures I see in a real-world copy of 12TB from a 16TB array onto a brand new 20TB RAID 6 array:

• 200 MBps (which seems way less than maximum given these are mainly 5-10GB data files, although this could be because the data on the 16TB drive is fragmented)
• 700 IOPS read and 500 IOPS for writes (which seems ok if it is mainly random writes, but way too slow if has lots of sequential storage)
• 60% disk utilization which also seems curious because this is a massive copy so you would expect much close to 100%
• 80% CPU and this is the puzzle, that’s pretty high
• 9% memory utilization which means memory is not the problem.

The clue here is the CPU utilization, this is a 10-year-old Atom processor vs a Ryzen V1500. So, the net is that it looks like the throughput and the IOPS are pretty low compared to what is possible. I’m betting right now that there are two problems:

1. Fragmented 16TB RAID 6 array. I didn’t pay much attention to creation so imagine it is pretty fragmented
2. Filestation copy is CPU intensive because it is use level. There could be things that are faster.

Next step: Try Synology Replication or rsync

So the next question is to see if Synology Replication is faster, I was just using a simple File Station copy and should try rsync. I have some good hope for Snapshot replication since this is block level copies so theoretically fragmentation shouldn’t matter and there is less for the CPU to do. So will report on that next.