Tong Family

Well, I spent an entire two weeks figuring this out. The background is that I got a new Synology RS2423+ which is a pretty awesome piece of hardware. It is a 2U rackmount with 12 drive bays. I got it all running and then I hit the first problem.

I’ve been using Synology Snapshot Replication. This is a cool service, if you use btrfs as your underlying storage system, you can have it take snapshots, this is a copy-on-write operating system, so you can get a huge number of views of your data (like Time Machine works on MacOS). And they have a service that lets you replicate those so you can keep different servers in sync.

I’ve been using this successfully between my Synology DS2413+ and my DS1812+, but when I installed the new RS2423+, I discovered three problems and the resulting tips and tricks:

Physical Installation of the Synology RS2423+ needs rear support

In trying to install this, I found that it is a 19″ rack mount, but I have a 23″ EIA rack, so I got the usual rabbit ears adapters that I used with our UniFi UDM Pro and Pro 48 switches, but this particular rack doesn’t have simple rabbit ears, it has a special assembly that allows quick release. Instead, I just put a shelf underneath and it is not using the rabbit ears until I figure out how to replace them.

The issue is that I have a two-post 23″ EIA rack and this is a big heavy server thingy and it supports the more standard four-post 19″ computer rack. So if you just mount it on the two posts, the back of the system will just hang down. There doesn’t seem to be an easy way to convert this two-post system into a four-post frame, so the solution instead is to use a shelf that I have that supports the back.

If you do have a four-post system, then you should buy the optional RTX-02 rail system which let’s you just pull the thing out when you need it.

Synology Replication from DSM 7 to DSM 6 needs an insecure connection or you get “Inbound Connection Failed”

The DS1812+ is no longer supported. It only has DSM 6.2 and won’t get any more updates. The big reason to get the RS2423+ is to get a little more time.

But now I’m running DSM 7 against DSM 6 with Synology Snapshot Replication. This is kind of a cool feature that’s a little hard to understand, but Synology by default uses a file system called btrfs. One of its features is that it is a “copy-on-write” operating system. So that when you create or modify a new file, it is very smart, instead of copying the file, it just looks and sees what blocks are the same and puts a pointer in for them instead. So a copy is lightning fast, more like a hard link than a copy.

One of the implications of this is that taking a snapshot of the file system daily is trivial, you just do a copy of the whole thing and stick that into the “#snapshot” directory. Then, when you make changes, the old blocks are still referred to by references in that directory, so you in effect have a history of all the files without a lot of overhead. If you delete all the files, you don’t delete them, you still have copies for years and years, but it is a very nice way to keep old copies around.

Synology uses this feature to make it easy to keep two shared folders in sync. When you do a daily backup instead of having to copy everything, you can just send the blocks that have changed. The first replication takes a long time, but then the next ones are very fast as they are just deltas.

At first, it all worked fine, but two days ago, suddenly the older replication reported an “Inbound connection error” and there is no discussion about what this is. I tried the diagnostic and fix procedure which is to turn off the firewalls and then check to make sure that Port 5566 is visible.

Debugging Replication

You are supposed to have Port 5566 and 5001 open which you can do with the Netcat utility. When doing a nc -zv _server_ 5566 it reports all those ports open. This is true even with the firewalls on, so that wasn’t the problem and you want to leave them on.

One other thing I did was to change the default administrative ports from 5000, 5001 and 22 to new ones, but I ultimately decided to turn them back to the defaults. This is more of a security hole since these are well known, but it prevents me from making a mistake as DSM Snapshot Replication assumes the default ports for the DSM admin user interface at 5000 and 5001.

Well, it turns out that there is some sort of problem with the certificates, so if you switch to non-encrypted, you are using port 5000 and it all works fine. I don’t feel great about using an unencrypted thing, but I do know that for admin, it always redirects from the insecure 5000 to the 5001 and it seems to work.

I think this is an issue with the way certificates are handled. You can get a Let’s Encrypt certification that will do a Dynamic DNS connection to your server, but this doesn’t have the internal Bonjour names which look like “server.local” and this means that you get these invalid certification messages. For whatever reason, this isn’t a problem going from DSM 7 to 7 or from DSM 6 to DSM 6, but becomes a problem otherwise.

You have to wait for an entire replication to see your data and only runs at 10-20MBps

The second problem is that this is a 13TB replication and I seem to remember that with DSM 6, you would start seeing files pretty much immediately. This doesn’t seem to be the case here when you create a replication, I had to wait 10 days for the replication to complete. The way that you can check to see how things are going is to go to the Synology DSM portal and choose Synology Snapshot > Replication and then click on the replication and then info. This tells you the state of the replication.

The thing is actually pretty smart, you can set it up so that you can replicate from one shared folder to up to three other local or remote servers. It is confusing how to do this because there is no additional user interface, you basically run create as many times as you want and it hooks the replications together. There is no obvious user interface for this, but the Info button in the Synology Snapshot Replication application is your friend. Note that one thing you can do (which I don’t need) is to chain replications, so for instance, a common use case is in the branch office, where you can setup replication over a VPN or even the public internet to say a regional office and from there replicate to corporate. Pretty cool.

The replication is only going at 10-20MBps which seems pretty slow given the multiple RAID10 drives and because one drive has a 10Gbe and the other is a bonded 2GBe, but if you are doing a long replication, don’t give up hope!

If you cancel a Replication, the miracle is that BTRFS figures out the orphaned one is still there and uses it

In the course of debugging this, I killed the replication from DSM 7 to DSM 6 and it is not easy to figure out how to restart it because well, you can change the name of a destination, it picks it for you. You basically create another replication, but beware, it won’t pick up the old files, so if you are replicating from a Shared Folder named, “Personal” and you create it on the destination server, you will get automatically a read-only folder also called “Personal” which is convenient.

But if you kill the replication, then when you try to create a new one, it will start again with “Personal_1” which is not great. The solution is to choose Control Panel > Shared Folders and rename the old folder to say “Personal-old”, then when you create it you get the right name.

I thought I would now have to wait weeks for things to sync, but it turns out that the BTRFS file system is so smart. It detects that I already have blocks stored in “Personal0old” and when it starts the snapshot replication to the new “Personal” it happens in the flash noting all the old blocks. So whatever you do, don’t delete “Personal-old” before you start this. Once the replication is complete, you can delete Personal-old though because at that point the old blocks are kept because there are still references to them from the new replication. Very handy!