Ok, this area of Netlify is super confusing, but basically, there is the concept of a deployment key and this appears to work if you have a private submodule but not all the time. GitHub has this limit where you can only use a single SSH key per user or per deployment key in a repository. And for some reason, this worked with one repo but not another. You cannot use the same deployment key for every repo so you have to generate lots of deployment keys. I think if you have 4 submodules then you need 4 deployment keys which is a pain to administer.
The documentation is incredibly hard to read, but basically, the simplest answer is to create a “netlify user” that is just for netlify integration into GitHub. That user just has read permissions on your repo. Then you create if you have say four different repos each that need to have netlify, then in each, you create a deployment key in the Netlify UI. This basically generates a private key inside Netlify that you never see.
Then you take the public key from there and go to the dummy user’s GitHub account and go to security and add all four keys and then everything should work.
Also from a cost optimization point of view, it is nice to make that netlify@tongfamily.com as an example of what you might name the sole team member for Netlify. Anyway, it’s hard to figure it out.
Of course, it would be nicer if you could just disable submodule pulls from the Netlify workflow, then you wouldn’t have to worry about this key problem. But I have not found a way to do that.