OK, some really strange things trying to get a public repo working. Many times when you are trying to recreate a machine learning work is that you need to have the data along with the actual code.
It's unfortunately completely non-standard how this is handled. I had thought that I could just turn Git LFS on and then the data files could just live there. However, it turns out this doesn't work. To prevent abuse, you cannot turn on Git LFS when you fork a repo.
As an aside, to figure this out, you have to turn on tracing and it is clearer what is going on.
GIT_TRACE=1 GIT_TRANSFER_TRACE=1 GIT_CURL_VERBOSE=1 git push
So this means that you have to deploy some other tricks to attach data to machine learning source code. There really should be a better way to do this.