Large uploads to iCloud… continued

OK, I’m trying to see if iCloud Photo Library is a good place to store things. So I tried uploading 6K (that’s actually a small percentage) of my photos up there. What I found is that Mac Photos hangs after a while. It is showing strange messages in the console like cloudphotod sending spurious data in the console. So here’s a decoder ring:

TL;dr

If this happens to you, try rebooting and so forth, but what worked for me is to CTRL-CMD Click on Mac Photos and then rebuild the Photo Library. Seems that it gets corrupt.
Turn off iCloud Photo Sharing and My Photostream, I don’t know if this makes a difference, but it removes one process from the mix. Just first to the upload and then you can make the sharing stuff work.
I also turned off my VPN and DNSCrypt. I don’t know if that makes a difference, but since it is about network access, I did do that. When the VPN is on, I was getting 40KBps, but when off, the speed went up dramatically. Not sure why.
When you are uploading, it can take days, right now an upload of 8K photos (I know that’s a lot) is taking about 1K photos a day at a data rate of 700-800KBps. The uplink supports up to 50MBps, so this looks like the throttle rate on the iCloud side initially. After 7,000 photos, it is down to 300KBps and I’m getting about 400 photos loaded per day. So you have to patient. The thing definitely goes faster sometimes and slower others, but I can’t figure out how to force it to go fast.
There is much folklore about whether you should have Photos up and running or whether it should be in the background. I’m finding that it doesn’t seem to make much difference. When the window is visible you do get a console message noting it and when it is closed, another console message, but the data rates seem about the same. When you close it, it does seem to reduce the data rate and it looks very bouncy. When fully open the rate seems to be about 300 photos a day (not very fast), will now try it closed for 24 hours and report back but quick look shows that the data rate seems to fall in half and seems more variable.
The photos counts seems to vary a lot by whether you look at iCloud.com, the Photos library doing the upload or an iPhone. Right now, there appears to be about a 12 hour lag between upload and the time it propagates into iCloud.com and another 12 hour lag before it appears in an iPhone. Hopefully the photo counts will equalize. One thing to check is whether all photos you imported are copied into the Photo Library or merely referenced. Referenced items do not appear in the iCloud photo library, you have to select the copy option in Photos/Preferences
Also download the free Photos deduplicator since it looks like about 1-5% of the photos do not get into their proper location. You will also probably need an EXIF editor as well because the photos are probably mislabeled particularly if they come from social media.
When you are doing this, click on Keep Organization on import so that you get the directory tree and it is easier to find photos
If none of this works, then a direct upload to iCloud via the web browser is another way to go but I haven’t resorted to that yet.
The import definitely misses items. So you want to import again and see what is “new”. It also gets confused by short videos, it seems to duplicate those over and over. Just make sure to use the Keep Folder organization selected because then you can see easily what was copied and then merge albums.

Decoder Ring

There are bunch of not documented processes around photos:

Photo Agent. Not quite sure what this does, doesn’t seem to run much
photoanalysisd. This is the long running routine that does face and object recognition on your photos. This is what Facebook is doing to all your photos (more on how not to post there later). But it let’s Siri answer questions like find pumpkins in photos. It runs forever and can be a resource hog. Also not clear if this result is shared in iCloud Photo Library.
cloudphotosd. This is the process that downloads Shared Albums and My Photostream. Apple is super confusing, but they have one storage place for your photos normally in ~/Pictures/Photos Library.photoslibrary but they also store it in another place `~/Library/Containers/com.apple.cloudphotosd/Data/Library/Application` which caches all your shared albums. Cloudphotosd is the daemon that does this downloading. One implication is that if you want to minimize disk space, you need to symlink this to a different drive. Deleting this folder by the way is the way to reset shared albums, if you have problems. I often have shared sync problems. Also if you turn off shared features, then cloudphotosd does not seem to run, so I stopped these processes just to try to isolate the upload issues.
photolibraryd. I’m not really sure what this process is doing. Some people think it controls the My Photo Stream, but others think that is what cloudphotosd does. When you do a CMD-OPT click on Photos and it is rebuilding the library, photolibraryd does really run heavily, this implies that it is the core daemon that is managing the photos. The rebuild by the way takes a long time, with 10K photos that take up to 120GB of data, it took 30 minutes to process.
nsurlsessiond. This is reportedly the process that does the upload. I didn’t see it active much, normally when doing an upload, it is photolibraryd.
dasd. Now this does not stand for the old name for a hard disk (Direct Access Storage Array), but is the dispatcher for jobs, if you search for Photos in the console, you will see it getting dispatched from here.
filecoordinationd. This handles multiple processes accessing the same file.
Console is your friend. This is the place where messages appear, you can also write a specific command line just to see photo related problems. Note that it is nsurlsessiond that seems to do the actual download work, but the message fly back pretty quickly. When I first hung the upload, cloudphotosd was throwing errors every few seconds saying spurious upload so that is one reason I took it out.