OK, I had this problem with my personal repo where I had copied a bunch of courses that meant I was dragging around a lot of git lfs content. The solution was to recreate the repo as a new "richtong" repo, but this meant that all the organizations where I'm working needed major surgery to make it work:

1. First is that if you have an old submodule, and want to reuse it's name as of December 2021, you have to do a git rm rich which will remove it. But it will still have a copy of all the module information in ./.git/modules/<path to rich/rich so you need to do a manual rm -rf to get rid of it.
2. Then you need to gh repo delete not that this is a very precipitous action, so you might want to instead just rename the repo gh repo rename so you still have the old one. Note that I'm using brew install gh here which is the github command line and very handy.
3. Now you just fork the repo with gh repo fork git@github.com:richtong/rich --org <the new org> which will do the fork properly
4. Now add it as a submodule with git submodule add git@github.com:<the new org>/rich and all is good!

1. A quick check is with find . -type l to look at our symlinks and see which ones might be missing. And see which point outside the repo.