More living with Windows: weirdness with colon sin file names and git-lfs and OpenSSH v8 authentication
Well, here are some more notes about living with Windows. I do have to say, what pain to get this running, but the path of just using git-bash seems to work with a few caveats that are really annoying, but here are some tips, but I was getting a very obscure error message. It was really hard to figure out what was wrong, but tl;dr two deep bugs:
- You cannot have certain characters like a colon (:) in a file name, git gets messed up because Windows doesn’t work.
- There is a deep bug the Windows version of OpenSSH (v7.7) and you have to replace it with the Choco installation of V8.0 which involves much Powershell hackery to start the right SSHD agents. You do not want the native Windows installation
- PowerShell is a very strange product, it is completely different from the shell sh scripts and derivatives and the cmd.exe of old windows, I did a lot of GGoogling to figure it out.
- If you want to use ssh links like firstname.lastname@example.org:richtong/src instead of https, then you have to do some special magic. The first is that you have to start OpenSSH authentication on the Windows side of the house. So in Powershell side of the house, you need to make sure it is installed and running.
- Then you have to make sure that the magic shell variable GIT_SSH is set properly and that you have pushed the OpenSSH version of ssh up high enough. Annoyingly, the default puts /usr/bin up way ahead of the Windows path names and there is no easy way to untangle that. It makes sense because most of the time you *do not* want the Windows executable, but the one supplied by MINGW64.
- If you are debugging
export GIT_TRACE=1so that you can see what commands are being executed.
- Also for the git submodule, you can add the
--progressflag so you can see what is happening
- The first thing discovered is that there are more restrictions on Windows file names than Linux/Mac ones. Specifially the ext4 and APFS systems vs NTFS.
- If you are using git lfs, then the first git submodule is doing to take a LOOOONGG time on Windows like it is hung, but when you are running the trace command and progress, you can see that git-lfs gets to update times 100% and then stops.
- And if you try to stop it and remove it, then some process hangs and you get a
resource busymessage. I think what is happening is that the git lfs implementation is downloading things but you can’t see it.
- So a reboot is the only way that I can find to solve this
- Then you if you are using Git LFS, you need to know that Windows NTFS is way more restrictive on file names that Linux/Mac, so that means that if a repo have an offending file name then really bad things happen. For me, this was a hang in
git submodule updatebecause git LFS has no idea how to deal with a long file name and gets trapped.
The only solution appears to be that you have to make your git repos safe for Windows and there are lots of rules here:
- On a Mac, you can’t have a colon (:) in older file system with the new APFS you can use that but not the slash, Linux, you can’t have a forward slash (/), but on Windows, these are also not allowed,
- There are some reserved names in Windows from the early days which are primarily device names, like /dev/ names in Linux/Mac so CON, PRN, AUX, NUL, COM[1-9] an LPT[1-9]
- The maximum file name length of a file name is 256 and path length is 32K bytes
As an example, the following name is illegal on Windows but works fine on Linux/Ma
course/deep1/data/Week 4/Deep Neural Network Application: Image Classification
But I think this is actually a problem with git lfs on Windows.
The next problem has to do with git lfs. This is a really strange one, but the command is hanging for some reason. Looks like an interaction with the Open SSH client which is noted in Git LFS Issue #4285 where git lfs hangs on clone (I’ve had that) and it hangs on push (Issue #3779). And the core issue is still open in Win32-OpenSSH which does not like stdin to be a null which is true in version 7.x.
I actually tried to make this work with the ssh that is included with MINGW64 and it does not interact properly with the git which is living in windows land. This makes sense because there are really two world here real windows and the Linux system. You can see this by doing a
which git and then a
which git-lfs and you can see git-lfs lives in MINGW64 while git lives in and it fails in PowerShell because ssh.exe does not like redirection from /dev/null
c:\Windows\System32\OpenSSH\ssh.exe -- 'email@example.com git-lfs-authenticate richtong/docker upload' </dev/null 2>/dev/null
And this finally led me to the right Github issue which has to od with this authentication and it is a bug in the OpenSSH side of the house.
There are few fixes for this. YOu cna actually change the core.sshCommand with beecause cat does not mind /dev/null as input. Also some has said that installing openssh with
scoop install openssh fixes this by bumping up openssh. There is also a git-with-openssh there which is interesting and make sure you unset GIT_SSH in your profile and GIT_SSH_COMMAND
git config --global core.sshCommand 'cat - | /C/Windows/System32/OpenSSH/ssh'
There are a sea of OpenSSH out there for Windows, so a
choco install OpenSSH installs version 7 but j
scoop install openssh does not seem to work it reports an error
The confusing thing to install a newer version of OpenSSH, is that you first have to uninstall the Windows version and then install the Chocolaty version
# In the Powershell and hold your nose on the syntax Get-Windows Capability -Online | ? Name -like 'OpenSSH*' Remove-Windows-Capability -Online -Name OpenSSH.Client~~~~0.0.1.0 Remove-Windows-Capability -Online -Name OpenSSH.Serverj~~~~0.0.1.0 # Now to get the later version 8.x from Choco choco install openssh -parmas ""/SSHServerFeature /KeyBasedAuthenticationFeature"" -y # Note that this dumps ssh into a different directory # so in git-bash you need # In powershell, this will not be on the path you can see this with Get-Command ssh # and you check the path with [System.Environment]::GetEnvironmentVariable('Path') # or the shorter form $ENV:PATH
Now Choco puts the client in a different dedicated directory and the problem is that, unlike, Linux, the standard directories are always ahead of the user added ones which you do not want, so while Choco does add
C:\Windows\Porgram Files\OpenSSH-Win64 it is too far back.
There is no ~/.bash_profile in Windows, so what you have to do is to set the registry (I’m holding my nose as to how verbose and different PowerShell is from Linux)
# echo oldpath=$PATH becomes $oldpath = (Get-ItemProperty -Path 'Registry::HKEY_LOCAL_MACHONE\System\CurrentControlSet\Control\Session Manager\Environment" -Name PATH).Path # or you can use the simpler syntax $oldpath = $env:path # this is the same as PATH="/dnew:$PATH" $newpath = "c:\Windows Program Files\OpenSSH-Win64;$oldpath" # simpler syntax but this is not permanent $env:path = "C:\Program Files\OpenSSH-Win64;$env:path" # Need to set it here to make it work alway and make sure you are in Administrator RunAs mode Set-ItemProperty -Path 'Registry::HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Environment' -Name PATH -Value $env:path
One thing you notice in looking at the Windows path is that it is filled with duplicates. Things are not checking ahead of time and really you want the standard things to be last and not first
($env:path).split(";") > foo.txt # then use vi or something to clean it up and then this rejoins it # because Get Content returns a string with spaces and the -join # tells the string to use semicolon as the delimiter $env:path = (Get-Content foo.txt) -join ";" # now commit to registry Set-ItemProperty -Path 'Registry::HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Environment' -Name PATH -value $env:path
Note that this is way down in the session manager, so you will need actually reboot the machine to make it work.
Now that you have this installed, you need to install the daemons
c:\Program Files\OpenSSH-Win64\install-sshd.ps1 Set-Service SSHD -StartupType Automatic Set-Service SSH-Agent -StartupType Automatic