OK, I've been running Tongfamily.com on Digital Ocean for the last five years and it has been fairly trouble-free (particularly when I added the 2GB swapfile) to prevent out-of-memory problems, but the last two days it has been crashing. It turns out that my simple installation with a 25GB disk has been running out of memory because of all the images that I've been using in posts.
So a few things that I fixed because of deferred maintenance (eg me not paying attention):
- There is a new web-based console that requires installing some Ubuntu packages. The problem is that the old web console was disabled, so you have to have SSH access to the machine or use their recovery console to install it. It is one of those pipes from some random URL into bash which is a little scary, but it does work.
- When I looked into this and i saw that
/var/logwas taking up 10GB, I saw a journal file was taking up over 10% of the disk space, but a quick
journalctl --vacuum-size=1Gcommand solved that and I capped the journaling at 1GB this game to breathing room. The other big file was in
auth.logwhich is the authorization but most of it was 300MB in the main log, so I left it.
- The problem was in
/var/www/htmlwhich is where WordPress installs itself, let me say that
du -shfor summarizing the disk size in human-readable format if very useful as is
df -sto look at which volumes are full. And I found that the WordPress installation was nearly 15GB with most of it being in
/var/www/html/uploadswhich is where the images for the blog post tops go.
- The problem is that Digital Ocean only lets you upgrade the size of the droplet one way, but really storage should be separate for the asset files. You do want to leave the main installation on the droplet for speed. There are a few choices here, you can use Digital Ocean Block Volume storage for $2.50/month or you can use the new Digital Ocean S3 compatible bucket storage called Digital Ocean Spaces. The nice thing is that 80% of my space is taken up with image uploads, so either would work, but the S3 does have a CDN built in and sets me up for static hosting someday, so I thought I would try that.
- This requires that you create a new one Digital Ocean Spaces of these, get the Access Token and the Secret Key from the API, and then you can use s3cmd to access it or you can use the Sync to Spaces WordPress plugin. Then you enter that information into Spaces Sync.
- You can use
s3cmdto access any S3-compatible storage system which is pretty nice, you just run a config command and you can connect directly to the S3.
- Note that I did try to use a third-party system that is S3 independent and even set up their image processing cloud pieces. But when I turn it on, everything basically stopped. So I uninstalled that.
- This seems to only work for new Images and videos that are uploaded, but this post is how I'm going to try it. The main problem was that I got the images to upload including the thumbnails, but they would not display properly, but it seems to work fine if you do dual storage (which kind of defeats the purpose.
Digital Ocean Spaces deprecated and Using Blocks
The first thing to notice is that buried in the documentation is that there are two modes that you can run, the Spaces can have the images, or both the server and the Spaces can. One big problem is that the plugin is now deprecated, so it isn't clear how to do this in the long term, so I just installed the Block Device for now
- The DigitalOcean Spaces Sync first needs the Connection Settings which are the Spaces Key, the Secret Token, then the container name, and finally the endpoint. So for instance, if the Spaces location is https://foo.nyc1.digitaloceanspace.com, the foo is the Spaces Container (confusing, this is not a docker container but just an S3 bucket equivalent) and the endpoint is
- The confusing this is what the Full URL-path to files is. Basically, if you use something like a full URL path like
https://tongfamily.com/wp-content/uploadsthen it is mirrored, but if you use a storage public domain then it is only stored in cloud storage. Confused yet? I'm confused. It needs something called the
storage public domainI'm very unclear on what this means, but it looks like you can somehow set a DNS CNAME to some location.
- The solution appears to be that I can get the mirrored thing to work by putting in the wp-content/uploads URL above, but can't figure out how not to use the local server.
- So I just created and a Block device and then edited /etc/fstab with the right entry to first create a
uploads.new, then copy all the uploads over and then finally move that and then remount the fstab at
uploadsand everything is good. The major commands are to first create a mount point with
sudo mkdir /var/www/html/wp-content/uploads.new, looking the
/dev/disk/by-idmount point since you should not use
/dev/sdbfor instance since these points can change on boot then editing /etc/fstab and adding the line
sudo mount -o defaults,nofail,discard,noatime /dev/disk/by-id/scsi-example /var/www/html/wp-content/uploads.new.At this point, you have the original uploads, so it is easy to
rsync -av /var/www/html/wp-content/uploads /var/www/html/wp-content/uploads.newwhich will copy the files to the mount device, then you can
mv uploads uploads.newand edit the fstab to use the regular uploads. Here the
findmnt --verify --verboseis very useful as it parses the /etc/fstab. And if it works then
mount -areads the table and mounts it.
- This method works, but when creating a new post, the deprecated plugin doesn't work, so time to just remove the whole thing and delete it
Trying and Failing with Imgix
OK, before I settled on the Digital Ocean Space Sync plugin, I tried a third party one called Ingix, which seemed to work but unfortunately hung the WordPress installation. It turns out that the free version doesn't allow migration of images, you have to pay for that, so I ended up uninstalling it for both those reasons, but here is how to install if you want to try yourself:
- Installing the Ingix and having it secure means adding a security access private key so that there is a digital signature when you download the images
- The installation of the sync was pretty easy, but man the options were really daunting, the Digital Ocean thing in comparison is much simpler, but Ingix does allow image hosting for up to 1,000 original images, so I might look at that some day.