Thursday, February 27, 2020

Windows Server: OpenSSH service does not start

I was trying to run OpenSSH server on a Windows Server and got the error

Windows could not start the sshd service on Local Computer.

Error 1067: the process terminated unexpectedly.


Here is a screenshot of the error from the OpenSSH Github project.


Error 1067 is not an error message specific to OpenSSH, but it is a generic error message from the Windows Services subsystem, when the executable for the service exit with an error. You will  to find out about the error message from OpenSSH's logs.

OpenSSH has various level of verbosity for its logs. Due to the fact that a server could have hundreds if not thousands of connection attempts per day, logging is either turned off or at a low level of verbosity. Refer to OpenSSH for Windows' logging section for more information on logging configuration.

File Permissions

SSH users in Linux might have come across this error message

Permission denied (publickey).


Often times, the password and public key used is correct, yet the error message still appears. Unfortunately the debug messages from ssh does not indicate one of the cause of this error - the file permission for "~/.ssh" as well as the private key permissions, for example "~/.ssh/id_rsa"

Some SSH files are critical for securing the identity of the client (user) or the server. SSH may refuse to connect, or SSH daemon refuse to start up if the permissions are too open.


User Keys

The sshd manual explains file permissions requirements for the following user key files


~/.ssh/
    This directory is the default location for all user-specific configuration and authentication information. There is no general requirement to keep the entire contents of this directory secret, but the recommended permissions are read/write/execute for the user, and not accessible by others.

~/.ssh/authorized_keys
    Lists the public keys (DSA, ECDSA, Ed25519, RSA) that can be used for logging in as this user. The format of this file is described above. The content of the file is not highly sensitive, but the recommended permissions are read/write for the user, and not accessible by others.

    If this file, the ~/.ssh directory, or the user's home directory are writable by other users, then the file could be modified or replaced by unauthorized users. In this case, sshd will not allow it to be used unless the StrictModes option has been set to “no”.


Server Host Key

The SSH daemon also has host keys that uniquely and securely identifies the server. If the server is rebuilt or physically swapped, this is when you will see the remote host identification warning

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
51:82:00:1c:7e:6f:ac:ac:de:f1:53:08:1c:7d:55:68.
Please contact your system administrator.
Add correct host key in /home/hanxuel/.ssh/known_hosts to get rid of this message.
Offending RSA key in /home/hanxuel/.ssh/known_hosts:12
RSA host key for 10.1.16.158 has changed and you have requested strict checking.
Host key verification failed.


The host key is critical for the SSH server's security!

On UNIX-like operating systems, the host key files need to be owned and readable by root only. From the sshd manual:

/etc/ssh/ssh_host_ecdsa_key
/etc/ssh/ssh_host_ed25519_key
/etc/ssh/ssh_host_rsa_key
    These files contain the private parts of the host keys. These files should only be owned by root, readable only by root, and not accessible to others. Note that sshd does not start if these files are group/world-accessible.

/etc/ssh/ssh_host_ecdsa_key.pub
/etc/ssh/ssh_host_ed25519_key.pub   
/etc/ssh/ssh_host_rsa_key.pub
    These files contain the public parts of the host keys. These files should be world-readable but writable only by root. Their contents should match the respective private parts. These files are not really used for anything; they are provided for the convenience of the user so their contents can be copied to known hosts files. These files are created using ssh-keygen(1).


Fixing OpenSSH Windows File Permissions

How does the UNIX permissions map to Windows permission? This is where Windows permissions gets a bit tricky:
  • UNIX root user generally maps to the System account and Administrator Group
  • UNIX root also maps to Administrators and members of Administrator Group
  • But when you log on to Windows, even if you are in Administrator Group, you do not have elevated permissions - you are a regular user
  • There is files/folders permission inheritance, where unexpected users and groups have access 
This Wiki article from OpenSSH for Windows Github project explains in more detail.

File permission problems for OpenSSH on Windows is common enough that the project released two PowerShell scripts to check and fix file permissions:
  • FixUserFilePermissions.ps1
  • FixHostFilePermissions.ps1

Fix User Permissions

This script will fix errors when you are using ssh.exe to connect to a remote server. Generally speaking, they do not affect the sshd.exe daemon.

Running the script will fix the permissions of any private key found in your %USERPROFILE%\.ssh directory

PS C:\Windows\OpenSSH> .\FixUserFilePermissions.ps1
  [*] C:\Users\hanxuel\.ssh\id_rsa
      looks good

  [*] C:\Users\hanxuel\.ssh\id_rsa.pub
      looks good

   Done.


Fix Server Permissions

The other script will fix errors with the sshd.exe daemon.


PS C:\Windows\OpenSSH> .\FixHostFilePermissions.ps1
  [*] C:\ProgramData\ssh\sshd_config

Need to remove the inheritance before repair the rules.
Shall I remove the inheritace?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): y
Inheritance is removed from 'C:\ProgramData\ssh\sshd_config'.

'BUILTIN\Users' should not have access to 'C:\ProgramData\ssh\sshd_config'..
Shall I remove this access?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): y
'BUILTIN\Users' has no more access to 'C:\ProgramData\ssh\sshd_config'.
      Repaired permissions

  [*] C:\ProgramData\ssh\ssh_host_dsa_key

Current owner: 'CORP\hanxuel'. 'NT AUTHORITY\SYSTEM' should own 'C:\ProgramData\ssh\ssh_host_dsa_key'.
Shall I set the file owner?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): y

'CORP\hanxuel' should not have access to 'C:\ProgramData\ssh\ssh_host_dsa_key'..
Shall I remove this access?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): y
'CORP\hanxuel' has no more access to 'C:\ProgramData\ssh\ssh_host_dsa_key'.
      Repaired permissions

  [*] C:\ProgramData\ssh\ssh_host_dsa_key.pub

Current owner: 'CORP\hanxuel'. 'NT AUTHORITY\SYSTEM' should own 'C:\ProgramData\ssh\ssh_host_dsa_key.pub'.
Shall I set the file owner?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): y

'CORP\hanxuel' has the following access to 'C:\ProgramData\ssh\ssh_host_dsa_key.pub': 'Modify, Synchronize'.
Shall I make it Read only?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): y
'CORP\hanxuel' now has Read access to 'C:\ProgramData\ssh\ssh_host_dsa_key.pub'.
      Repaired permissions

  [*] C:\ProgramData\ssh\ssh_host_ecdsa_key

Current owner: 'CORP\hanxuel'. 'NT AUTHORITY\SYSTEM' should own 'C:\ProgramData\ssh\ssh_host_ecdsa_key'.
Shall I set the file owner?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): y

'CORP\hanxuel' should not have access to 'C:\ProgramData\ssh\ssh_host_ecdsa_key'..
Shall I remove this access?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): y
'CORP\hanxuel' has no more access to 'C:\ProgramData\ssh\ssh_host_ecdsa_key'.
      Repaired permissions

  [*] C:\ProgramData\ssh\ssh_host_ecdsa_key.pub

Current owner: 'CORP\hanxuel'. 'NT AUTHORITY\SYSTEM' should own 'C:\ProgramData\ssh\ssh_host_ecdsa_key.pub'.
Shall I set the file owner?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): y

'CORP\hanxuel' has the following access to 'C:\ProgramData\ssh\ssh_host_ecdsa_key.pub': 'Modify, Synchronize'.
Shall I make it Read only?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): y
'CORP\hanxuel' now has Read access to 'C:\ProgramData\ssh\ssh_host_ecdsa_key.pub'.
      Repaired permissions

  [*] C:\ProgramData\ssh\ssh_host_ed25519_key

Current owner: 'CORP\hanxuel'. 'NT AUTHORITY\SYSTEM' should own 'C:\ProgramData\ssh\ssh_host_ed25519_key'.
Shall I set the file owner?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): y

'CORP\hanxuel' should not have access to 'C:\ProgramData\ssh\ssh_host_ed25519_key'..
Shall I remove this access?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): y
'CORP\hanxuel' has no more access to 'C:\ProgramData\ssh\ssh_host_ed25519_key'.
      Repaired permissions

  [*] C:\ProgramData\ssh\ssh_host_ed25519_key.pub

Current owner: 'CORP\hanxuel'. 'NT AUTHORITY\SYSTEM' should own 'C:\ProgramData\ssh\ssh_host_ed25519_key.pub'.
Shall I set the file owner?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): y

'CORP\hanxuel' has the following access to 'C:\ProgramData\ssh\ssh_host_ed25519_key.pub': 'Modify, Synchronize'.
Shall I make it Read only?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): y
'CORP\hanxuel' now has Read access to 'C:\ProgramData\ssh\ssh_host_ed25519_key.pub'.
      Repaired permissions

  [*] C:\ProgramData\ssh\ssh_host_rsa_key

Current owner: 'CORP\hanxuel'. 'NT AUTHORITY\SYSTEM' should own 'C:\ProgramData\ssh\ssh_host_rsa_key'.
Shall I set the file owner?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): y

'CORP\hanxuel' should not have access to 'C:\ProgramData\ssh\ssh_host_rsa_key'..
Shall I remove this access?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): y
'CORP\hanxuel' has no more access to 'C:\ProgramData\ssh\ssh_host_rsa_key'.
      Repaired permissions

  [*] C:\ProgramData\ssh\ssh_host_rsa_key.pub

Current owner: 'CORP\hanxuel'. 'NT AUTHORITY\SYSTEM' should own 'C:\ProgramData\ssh\ssh_host_rsa_key.pub'.
Shall I set the file owner?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): y

'CORP\hanxuel' has the following access to 'C:\ProgramData\ssh\ssh_host_rsa_key.pub': 'Modify, Synchronize'.
Shall I make it Read only?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): y
'CORP\hanxuel' now has Read access to 'C:\ProgramData\ssh\ssh_host_rsa_key.pub'.
      Repaired permissions

  [*] C:\Users\hanxuel\.ssh\authorized_keys
      looks good

   Done.



Try restarting the OpenSSH Windows service again and it should now start up correctly.







No comments:

Post a Comment