Skip to main content

SSH Hardening

In this article I'm going to sum up all the information I've gathered of securing an ssh server and client, basically it's going to be a copy-paste of the sources as an all in one document reference.

On the server side we are going to cover:

  • Fwknop
  • Update ssh
  • SSHD config
  • Key generation
  • Protect files and directories
  • Perform compile-time configuration
  • Disable sftp
  • Restrict users to sftp and don't allow them to ssh
  • Only allow one command
  • Allow more than one command
  • Batch job authentication
  • Chrooting
  • Traffic analysis resistance
  • Keyserver key rotation
  • ssh-copy-id
  • Multi-Factor authentication
  • Remove the r-Commands
  • Use subsystems
  • Hardware security
  • Change port number
  • Hide SSH version
  • Flip Feng Shui

It’s probably a good idea to test the changes. ssh -v will print the selected algorithms and also makes problems easier to spot. Be extremely careful when configuring SSH on a remote host. Always keep an active session.

You also have the option to execute sshd with the flag -t which will start the server, test the configuration and close

On the client side we are going to cover:

  • SSH client configuration
  • SSH Jumping
  • SSH-Agent
  • Use multiple keys
  • Protection of the keys
  • ssh-addhost

Server side

Fwknop

First of all I'll suggest to use Single Packet Authorization to protect your ports instead of the well known Port Knocking.

It doesn't matter if your sshd version is vulnerable to some bug, first Mallory must arrive to that port, so you securely reduce the threat spectrum by a lot.

To see how to configure Fwknop follow my other post

Update ssh

Yes, I know it's obvious but it has to be said. Some older versions have known security holes taht are easily exploited. Always run the latest stable version, and apply updates or patches in a timely manner

SSHD config

Let's continue tweaking /etc/sshd/sshd_config

For the TL;DR people here is the result:

# Package generated configuration file
# See the sshd_config(5) manpage for details

#---------------
# General config
#---------------

# Disable protocol 1
Protocol 2

# What ports, IPs and protocols we listen for
Port 22
Port 443

# Use these options to restrict which interfaces/protocols sshd will bind to
#ListenAddress ::
#ListenAddress 0.0.0.0
#ListenAddress 127.0.0.1:22


# Authentication:
LoginGraceTime 30
PermitRootLogin no
StrictModes yes

# Don't read the user's ~/.rhosts and ~/.shosts files
IgnoreRhosts yes
RhostsRSAAuthentication no
HostbasedAuthentication no
# Uncomment if you don't trust ~/.ssh/known_hosts for RhostsRSAAuthentication
#IgnoreUserKnownHosts yes

# Only allow specific groups or users
AllowGroups wheel admin ssh 
#AllowUsers kn@gentoo.org bs@gentoo.org

# Logging
SyslogFacility AUTH

## LogLevel VERBOSE logs user's key fingerprint on login. Needed to have a clear audit track of which key was using to log in.
LogLevel VERBOSE

## Log sftp level file access (read/write/etc.) that would not be easily logged otherwise. If it's enabled 
Subsystem sftp  /usr/lib/sftp-server -f AUTHPRIV -l INFO

# Make sure UsePAM is disabled
# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM no

# Ensure /bin/login is not used so that it cannot bypass PAM settings for sshd.
UseLogin no

#Privilege Separation is turned on for security
UsePrivilegeSeparation yes

# Disable X forwarding
X11Forwarding no
X11DisplayOffset 10

# Disable TCP Forwarding
AllowTcpForwarding no

# Miscelaneous
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
UseDNS no
MaxStartups 10:50:20

# Allow client to pass locale environment variables
AcceptEnv LANG LC_*

#------------------
# Cipher selection
#------------------

# Key Exchange
KexAlgorithms curve25519-sha256@libssh.org

# Server authentication
HostKey /etc/ssh/ssh_host_ed25519_key

# Symmetric cipher
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes256-ctr

# Message authentication codes
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-512

#------------------------
# Client authentication
#------------------------

AuthenticationMethods publickey
PubkeyAuthentication yes
#AuthorizedKeysFile     %h/.ssh/authorized_keys

# To enable empty passwords, change to yes (NOT RECOMMENDED)
PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no

# Change to no to disable tunnelled clear text passwords
PasswordAuthentication no

# Kerberos options
KerberosAuthentication no
#KerberosGetAFSToken no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes

# GSSAPI options
GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
  • Disable SSH1

This will also disable the horribly broken v1 protocol that you should not have enabled in the first place.

Protocol 2
  • Change port number

This is a common suggested measure, but it's security through obscurity, it's better to use fwknop as stated above.

Also if you run the service with a port above 1024 and by chance the server gets stopped, any regular user can run a sshd instance on that port therefore you open a window of vulnerability.

So my suggestion would be not to use this measure, and if you do so, to be a port under 1024.

Port 22
  • Listen to a specific interface

In case you have more than one interface and you want to listen just to one use

ListenAddress 192.168.0.10
#ListenAddress 127.0.0.1:22
  • Reduce the available login time
LoginGraceTime 30
  • Disable root login
PermitRootLogin no
  • Check the permissions of important files and diretories, they must be owned by root and group and world write permissions must be disabled.
StrictModes yes
  • Disable .rhost, hostbased and normal password authentication
IgnoreRhosts yes
RhostsRSAAuthentication no
HostbasedAuthentication no
  • Only allow specific groups or users to log in
AllowGroups wheel admin ssh
  • Logging
SyslogFacility AUTH
LogLevel VERBOSE
  • Log sftp level file access (read/write/etc.) that would not be easily logged otherwise. If it's enabled
Subsystem sftp  /usr/lib/sftp-server -f AUTHPRIV -l INFO
  • Make sure UsePAM is disabled

Also verify that you don't have UsePAM yes in your configuration file as it overrides the public key authentication mechanism, or you can disable either PasswordAuthentication or ChallengeResponseAuthentication. More information about these options can be found in the sshd_config manual page.

UsePAM no
  • Disable UseLogin

Ensure /bin/login is not used so that it cannot bypass PAM settings for sshd.

Also with UseLogin yes: * X forwarding is turned off, since sshd loses the chance to specifically handle ts xauth cookies for X authentication. * Privilege separation is turned off after user authentication in order to allow login to function correctly

UseLogin no
  • Use privilege separation

Specifies whether sshd(8) separates privileges by creating an unprivileged child process to deal with incoming network traffic. After successful authentication, another process will be created that has the privilege of the authenticated user. The goal of privilege separation is to prevent privilege escalation by containing any corruption within the unprivileged processes.

UsePrivilegeSeparation yes
  • Disable X forwarding

To avoid X bugs over ssh

X11Forwarding no
X11DisplayOffset 10
  • Disable TCP forwarding

When it's not needed disable tcp port forwarding

AllowTcpForwarding no
  • Show welcome message

It may be interesting to show messages to the user, both security oriented or whatever you like. For this we use the message of the day. It will cat the contents of /etc/motd

PrintMotd yes
  • Print last time the user logged in
PrintLastLog yes
  • TCPKeepAlive

The value yes ( the default) tells the server to set the TCP keepalive option on its connection to the client. This causes TCP to transmit and expect periodic keepalive messages. If it doesn't receive responses to these messages for a while, it returns an error to sshd, which then shuts down the connection. "Keepalive" is the wrong namefor it, it would be better named "detect dead"

TCPKeepAlive yes
  • Disable DNS lookup.

You might think security is increased by reverse DNS lookups, but in fact, DNS isn't secure enough to guarantee accurate lookups. And SSH connections can be tremendously slowed down or fail altogether if the client's DNS is hosed.

UseDNS no
  • Limit the number of connections

So as to prevent DoS attacks (although preventing the connection you are performing a DoS)

MaxStartups 10:50:20

This tells the server to refuse connections based on probabilities. If the number of connections is 10 or greater, sshd will beging rejecting connections. When tehre are 10 connections, the probability of rejction is 50%. When there are 20 connecitons, the probability of rejection is 100%. Between 10 and 20, the probability increases linearly from 50% to 100%.

  • Key exchange

There are basically two ways to do key exchange: Diffie-Hellman and Elliptic Curve Diffie-Hellman. Both provide forward secrecy which the NSA hates because they can’t use passive collection and key recovery later. The server and the client will end up with a shared secret number at the end without a passive eavesdropper learning anything about this number. After we have a shared secret we have to derive a cryptographic key from this using a key derivation function. In case of SSH, this is a hash function. Collision attacks on this hash function have been proven to allow downgrade attacks.

OpenSSH supports 8 key exchange protocols:

  1. curve25519-sha256: ECDH over Curve25519 with SHA2
  2. diffie-hellman-group1-sha1: 1024 bit DH with SHA1
  3. diffie-hellman-group14-sha1: 2048 bit DH with SHA1
  4. diffie-hellman-group-exchange-sha1: Custom DH with SHA1
  5. diffie-hellman-group-exchange-sha256: Custom DH with SHA2
  6. ecdh-sha2-nistp256: ECDH over NIST P-256 with SHA2
  7. ecdh-sha2-nistp384: ECDH over NIST P-384 with SHA2
  8. ecdh-sha2-nistp521: ECDH over NIST P-521 with SHA2

We have to look at 3 things here:

  • ECDH curve choice: This eliminates 6-8 because NIST curves suck. They leak secrets through timing side channels and off-curve inputs. Also, NIST is considered harmful and cannot be trusted.
  • Bit size of the DH modulus: This eliminates 2 because the NSA has supercomputers and possibly unknown attacks. 1024 bits simply don’t offer sufficient security margin.
  • Security of the hash function: This eliminates 2-4 because SHA1 is broken. We don’t have to wait for a second preimage attack that takes 10 minutes on a cellphone to disable it right now.

We are left with 1 and 5. 1 is better and it’s perfectly OK to only support that but for interoperability (with Eclipse, WinSCP), 5 can be included.

Recommended /etc/ssh/sshd_config snippet:

KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256

Recommended /etc/ssh/ssh_config snippet:

# Github needs diffie-hellman-group-exchange-sha1 some of the time but not always.
#Host github.com
#    KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1

Host *
    KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256

If you chose to enable 5, open /etc/ssh/moduli if exists, and delete lines where the 5th column is less than 2000.

awk '$5 > 2000' /etc/ssh/moduli > "${HOME}/moduli"
wc -l "${HOME}/moduli" # make sure there is something left
mv "${HOME}/moduli" /etc/ssh/moduli

If it does not exist, create it:

ssh-keygen -G /etc/ssh/moduli.all -b 4096
ssh-keygen -T /etc/ssh/moduli.safe -f /etc/ssh/moduli.all
mv /etc/ssh/moduli.safe /etc/ssh/moduli
rm /etc/ssh/moduli.all

This will take a while so continue while it’s running.

  • Server authentication

The key exchange ensures that the server and the client shares a secret no one else knows. We also have to make sure that they share this secret with each other and not an NSA analyst.

The server proves its identity to the client by signing the key resulting from the key exchange. There are 4 public key algorithms for authentication:

  1. DSA with SHA1
  2. ECDSA with SHA256, SHA384 or SHA512 depending on key size
  3. Ed25519 with SHA512
  4. RSA with SHA1

DSA keys must be exactly 1024 bits so let’s disable that. Number 2 here involves NIST suckage and should be disabled as well. Another important disadvantage of DSA and ECDSA is that it uses randomness for each signature. If the random numbers are not the best quality, then it is possible to recover the secret key. Fortunately, RSA using SHA1 is not a problem here because the value being signed is actually a SHA2 hash. The hash function SHA1(SHA2(x)) is just as secure as SHA2 (it has less bits of course but no better attacks).

HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key

The first time you connect to your server, you will be asked to accept the new fingerprint.

We should remove the unused keys and only generate a large RSA key and an Ed25519 key. Your init scripts may recreate the unused keys. If you don’t want that, remove any ssh-keygen commands from the init script.

cd /etc/ssh
rm ssh_host_*key*
ssh-keygen -t ed25519 -f ssh_host_ed25519_key
ssh-keygen -t rsa -b 4096 -f ssh_host_rsa_key
  • Client Authentication

The client must prove its identity to the server as well. There are various methods to do that.

Recommended /etc/ssh/ssh_config snippet:

Host *
    PubkeyAuthentication yes
    HostKeyAlgorithms ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ssh-rsa

Generate client keys using the following commands:

ssh-keygen -t ed25519 -o -a 100
ssh-keygen -t rsa -b 4096 -o -a 100

You can use ssh-keygen -o -a $number to slow down cracking attempts by iterating the hash function many times.

Unfortunately, you can’t encrypt your server key and it must be always available, or else sshd won’t start. The only thing protecting it is OS access controls.

  • Symmetric Ciphers

Symmetric ciphers are used to encrypt the data after the initial key exchange and authentication is complete.

Here we have quite a few algorithms:

  1. 3des-cbc
  2. aes128-cbc
  3. aes192-cbc
  4. aes256-cbc
  5. aes128-ctr
  6. aes192-ctr
  7. aes256-ctr
  8. aes128-gcm@openssh.com
  9. aes256-gcm@openssh.com
  10. arcfour
  11. arcfour128
  12. arcfour256
  13. blowfish-cbc
  14. cast128-cbc
  15. chacha20-poly1305@openssh.com

We have to consider the following:

  • Security of the cipher algorithm: This eliminates 1 and 10-12 - both DES and RC4 are broken. Again, no need to wait for them to become even weaker, disable them now.
  • Key size: At least 128 bits, the more the better.
  • Block size: Does not apply to stream ciphers. At least 128 bits. This eliminates 13 and 14 because those have a 64 bit block size.
  • Cipher mode: The recommended approach here is to prefer AE modes and optionally allow CTR for compatibility. CTR with Encrypt-then-MAC is probably secure.
  • Chacha20-poly1305 is preferred over AES-GCM because the SSH protocol does not encrypt message sizes when GCM (or EtM) is in use. This allows some traffic analysis even without decrypting the data. We will deal with that soon.

Recommended /etc/ssh/sshd_config snippet:

Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr

Recommended /etc/ssh/ssh_config snippet:

Host *
    Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
  • Message authentication codes

Encryption provides confidentiality, message authentication code provides integrity. We need both. If an AE cipher mode is selected, then extra MACs are not used, the integrity is already given. If CTR is selected, then we need a MAC to calculate and attach a tag to every message.

There are multiple ways to combine ciphers and MACs - not all of these are useful. The 3 most common:

  1. Encrypt-then-MAC: encrypt the message, then attach the MAC of the ciphertext.
  2. MAC-then-encrypt: attach the MAC of the plaintext, then encrypt everything.
  3. Encrypt-and-MAC: encrypt the message, then attach the MAC of the plaintext.

Only Encrypt-then-MAC should be used, period. Using MAC-then-encrypt have lead to many attacks on TLS while Encrypt-and-MAC have lead to not quite that many attacks on SSH. The reason for this is that the more you fiddle with an attacker provided message, the more chance the attacker has to gain information through side channels. In case of Encrypt-then-MAC, the MAC is verified and if incorrect, discarded. Boom, one step, no timing channels. In case of MAC-then-encrypt, first the attacker provided message has to be decrypted and only then can you verify it. Decryption failure (due to invalid CBC padding for example) may take less time than verification failure. Encrypt-and-MAC also has to be decrypted first, leading to the same kind of potential side channels. It’s even worse because no one said that a MAC’s output can’t leak what its input was. SSH by default, uses this method.

Here are the available MAC choices:

  1. hmac-md5
  2. hmac-md5-96
  3. hmac-ripemd160
  4. hmac-sha1
  5. hmac-sha1-96
  6. hmac-sha2-256
  7. hmac-sha2-512
  8. umac-64
  9. umac-128
  10. hmac-md5-etm@openssh.com
  11. hmac-md5-96-etm@openssh.com
  12. truncate hmac-ripemd160-etm@openssh.com;
  13. hmac-sha1-etm@openssh.com
  14. hmac-sha1-96-etm@openssh.com
  15. hmac-sha2-256-etm@openssh.com
  16. hmac-sha2-512-etm@openssh.com
  17. umac-64-etm@openssh.com
  18. umac-128-etm@openssh.com

The selection considerations:

  • Security of the hash algorithm: No MD5 and SHA1. Yes, I know that HMAC-SHA1 does not need collision resistance but why wait? Disable weak crypto today.
  • Encrypt-then-MAC: I am not aware of a security proof for CTR-and-HMAC but I also don’t think CTR decryption can fail. Since there are no downgrade attacks, you can add them to the end of the list. You can also do this on a host by host basis so you know which ones are less safe.
  • Tag size: At least 128 bits. This eliminates umac-64-etm.
  • Key size: At least 128 bits. This doesn’t eliminate anything at this point.

Recommended /etc/ssh/sshd_config snippet:

MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-ripemd160-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-ripemd160,umac-128@openssh.com

Recommended /etc/ssh/ssh_config snippet:

Host *
    MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-ripemd160-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-ripemd160,umac-128@openssh.com
  • Use Public key authentication

The simplest is password authentication. This should be disabled immediately after setting up a more secure method because it allows compromised servers to steal passwords. Password authentication is also more vulnerable to online bruteforce attacks.

Recommended /etc/ssh/sshd_config snippet:

AuthenticationMethods publickey
PasswordAuthentication no

The most common and secure method is public key authentication, basically the same process as the server authentication.

  • Don't permit empty passwords

By default disable them, if you need a passwordless access for a script use a public key with no encryption of the private key instead (explained below).

PermitEmptyPasswords no
  • Disable unused authentication methods
PermitEmptyPasswords no
ChallengeResponseAuthentication no
PasswordAuthentication no
KerberosAuthentication no
GSSAPIAuthentication no
  • Limit the number of times (six by default) that a user can attempt to authenticate in a single SSH session

It's better to use a fail2ban because this option allows to try again in a new connection

MaxAuthTries 6

Key generation

Now all that your users have to do is create a key (on the machine they want to login from) and type in a passphrase with the following command:

RSA keys are favored over ECDSA keys when backward compatibility ''is required'', thus, newly generated keys are always either ED25519 or RSA (NOT ECDSA or DSA).

ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa -C "Client key for xyz" -o -a 100
ssh-keygen -t rsa -b 4096 -f /etc/ssh/ssh_host_rsa_key -C "Server key for xyz" -o -a 100

ED25519 keys are favored over RSA keys when backward compatibility ''is not required''. This is only compatible with OpenSSH 6.5+ and fixed-size (256 bytes).

ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -C "Client key for xyz" -o -a 100
ssh-keygen -t ed25519 -f /etc/ssh/ssh_host_ed25519_key -C "Server key for xyz" -o -a 100

This will add two files in your ~/.ssh/ directory called id_ed25519 and id_ed25519.pub. The file called id_ed25519 is your private key and should be kept from other people than yourself. The other file id_ed25519.pub is to be distributed to every server that you have access to. Add the key to the users home directory in ~/.ssh/authorized_keys and the user should be able to login:

scp id_ed25519.pub other-host:/var/tmp/currenthostname.pub
ssh other-host
user $cat /var/tmp/currenthostname.pub >> ~/.ssh/authorized_keys

or use ssh-copy-id (see below).

It may be a good idea to set up different keys for different hosts

Add configuration to ~/.ssh/config

host *.host1.com
    IdentityFile ~/.ssh/id_...host1... # <= replace by your key's path

You may run this for any key file that you have.

ssh-keygen -lf id_rsa

Protect files and directories

Make sure the ssh directories and associated files have the proper permissions

  • The server host key should be readable only by root.
  • Don't save the ssh files in a NFS device.
  • Proctect the user files

On the O'Reilly reference they suggest 755 and 644 but I think it may give information to a non-root attacker that's inside the system, therefore I suggest:

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

Perform compile-time configuration

This are the advantages to compile yourself OpenSSH instead of using the binary from the repositories

  • Some configuration options can only be set at compile time.
  • Features that are disabled at compile time can’t be accidentally enabled by erroneous configuration files. Inflexibility can be an asset.
  • Removing code for unused features improves security, you can’t be burned by security holes in code that you don’t compile!
  • Similarly, code removal sometimes yields a performance advantage, since less memory and disk space is used.

I'm not going to dive deeper in this topic but you could :).

Of course if you use a Gentoo based distro you can turn off the desired USE flags, this are the available ones:

  • X
  • X509
  • bindist
  • debug
  • hpn
  • kerberos
  • ldap
  • ldns
  • libedit
  • libressl
  • livecd
  • pam
  • pie
  • sctp
  • selinux
  • skey
  • ssh1
  • ssl
  • static
  • test

Disable sftp

This section follows the principle of least binaries to exploit, because if you allow ssh anyone could perform the actions of scp or sftp.

But if this feature is not necessary for your server follow the next steps

groupadd sftpusers
usermod -a -G sftpusers <userthat_needs_ftp>
chgrp sftpusers /usr/lib/sftp-server /usr/lib/openssh/sftp-server
chmod 0750 /usr/lib/sftp-server /usr/lib/openssh/sftp-server

Restrict users to sftp and don't allow them to ssh

We'd like some user to be able to sftp but not execute any other command.

The version 4.8p1 of OpenSSH features a new configuration option : ChrootDirectory. This has been made possible by a new SFTP subsystem statically linked to sshd.

This makes it easy to replace a basic FTP service without the hassle of configuring encryption and/or bothering with FTP passive and active modes when operating through a NAT router. This is also simpler than packages such as rssh, scponly or other patches because it does not require setting up and maintaining (i.e. security updates) a chroot environment.

You need to configure OpenSSH to use its internal SFTP subsystem.

# File: /etc/ssh/sshd_config
Subsystem sftp  /usr/lib/sftp-server -f AUTHPRIV -l INFO

Then, I configured chrooting in a match rule.

Match group sftponly
    ChrootDirectory /home/%u
    X11Forwarding no
    AllowTcpForwarding no
    ForceCommand internal-sftp

If necessary create the group and add the users

groupadd sftponly
usermod -a -G sftponly <userthat_needs_ftp>

The directory in which to chroot must be owned by root. After the call to chroot, sshd changes directory to the chrooted directory.

chown root:root /home/$user
adduser $user sftponly

Also, set their shell to /usr/bin/false to prevent a normal ssh login:

usermod -s /bin/false username

Warning: Make sure that /bin/false exists in /etc/shells as well. Otherwise the login will fail with an invalid password error.

Only allow one command

As we all know, it is possible to use SSH not only for obtaining an interactive login session on a remote machine, but also for executing commands remotely. For instance, the following command will log on to myserver.example.com, execute “uname -a” and return to the local shell:

ssh myserver.example.com uname -a

(The local SSH client returns the exit code from the remote command, if you’re into this kind of detail.)

You might have some users (or scheduled automatisms) that you don’t want to be able to log on to that machine at all, but who should be permitted to execute only a given command. In order to achieve this, you can configure key-based authentication. Once this has been done, the key can be prefixed with a number of configuration options. Using one of these options, it is possible to enforce execution of a given command when this key is used for authentication.

Before we begin with in-depth examples of forced commands, let's discuss security. A forced command, carelessly used, may lull you into a sense of false security, believing that you have limited the client's capabilities when you haven't. This occurs if the forced command unintentionally permits a shell escape. Many UNIX programs have shell escapes, such as text editors (vi, Emacs), pagers (more,less), programs that invoke pagers (man), debuggers (gdb), non interactive commands (find, xargs, ...),etc.

When you define a forced command, you probably don't want its key used for arbitrary shell commands. Therefore, the author proposes the following safety rules for deciding whether a program is appropriate as a forced command:

  • Avoid programs that have shell escapes. Read their documentation carefully. If you still aren't sure, get help.
  • Avoid compilers, interpreters, or other programs that let the user generate and run arbitrary executable code.
  • Treat very carefully any program that creates or deletes files on disk in user specified locations. This includes not only applications (word processors, graphics programs, etc), but also command-line utilities that move or copy files (cp, mv, rm ,scp etc.).
  • Avoid programs with their setuid or setgid bits set, particularly setuid root.
  • If using a script as a forced command, follow traditional rules of safe script writing. Within the script, limit the search path to relevant directories (omitting "."), invoke al programs by absolute path, don't blindly execute user-supplied strings as commands, and don't make the script setuid anything. And again, don't invoke any program that has a shell escape.
  • Associate the forced command with a separate, dedicated SSH key, not the one used for your logins, so that you can conveniently disable the key without affecting your login capability.
  • Disable unnecessary SSH features using other options we cover later. Such as no-port-forwarding, no-x11-forwarding, no-agent-forwarding, no-pty

There are three ways to do this:

ForceCommand in sshd_config

We've seen in the previous section that it's possible to specify the only command to use with a match with something like:

Match user $user
    ForceCommand "/bin/ps"

But you can't specify arguments to that command, if you need to, write a script in the home directory and force it to execute that script

ForceCommand "./script"

Command in authorized_keys

But it can also be done at user level with the ssh keys and specify the exact command allowed in their ~/.ssh/authorized_keys file e.g.

For example if we just want to scp a directory to a server:

no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,command="/usr/bin/scp -v -r -d -t ~/content" ssh-rsa AAAAMYRSAKEY...

Create a link in ~ that links to the directory where the content should be accessible.

ln -s /path/to/directory/with/accessible/content ~/content

Now, from client side, the following command should work :

scp -r path/to/data user@server:~/content

If we want to extract some directory from the server:

no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,command="/usr/bin/scp -v -r -d -f ~/content" ssh-rsa AAAAMYRSAKEY...

It is the public key that contains the forced command. The public key is stored on the server where the command is going to be executed. So if someone has the ability to modify the authorized_keys file and change the command, then they already have shell access and access to that account, so breaking out of ‘jail’ isn’t really a concern.

Specify the shell as the wrapper

This solution is similar than the above, and if the developers of OpenSSH decided to use the forcecommand, I think it's better to use it. Nevertheless here it goes a third option.

If you don't want the user to be able to have shell access, then simply replace that user's shell with a script. If you look in /etc/passwd you will see that there is a field which assigns a shell command interpreter to each user. The script is used as the shell both for their interactive login ssh user@host as well as for commands ssh user@host command arg ....

Here is an example. The author of the original post created a user foo whose shell is a script. The script prints the message "my arguments are:" followed by its arguments (each on a separate line and in angle brackets) and terminates. In the log in case, there are no arguments. Here is what happens:

webserver:~# ssh foo@localhost
foo@localhost's password:
Linux webserver [ snip ]
[ snip ]
my arguments are:
Connection to localhost closed.

If the user tries to run a command, it looks like this:

webserver:~# ssh foo@localhost cat /etc/passwd
foo@localhost's password:
my arguments are:
<-c>
<cat /etc/passwd>

The "shell" receives a -c style invocation, with the entire command as one argument, just the same way that /bin/sh would receive it.

So as you can see, what we can do now is develop the script further so that it recognizes the case when it has been invoked with a -c argument, and then parses the string (say by pattern matching). Those strings which are allowed can be passed to the real shell by recursively invoking /bin/bash -c . The reject case can print an error message and terminate (including the case when -c is missing).

You have to be careful how you write this. I recommend writing only positive matches which allow only very specific things, and disallow everything else.

Note: if you are root, you can still log into this account by overriding the shell in the su command, like this su -s /bin/bash foo. (Substitute shell of choice.) Non-root cannot do this.

Here is an example script: restrict the user into only using ssh for git access to repositories under /git.

#!/bin/bash

if [ $# -ne 2 ] || [ "$1" != "-c" ] ; then
echo interactive login not permitted
  exit 1
fi

case "$2" in
  "git-upload-pack '/git/"* | "git-receive-pack '/git/"* )
     ;; # continue execution
  * )
     echo that command is not allowed
     exit 1
     ;;
esac

/bin/bash -c "$2"

Of course, we are trusting that these Git programs git-upload-pack and git-receive-pack don't have holes or escape hatches that will give users access to the system.

That is inherent in this kind of restriction scheme. The user is authenticated to execute code in a certain security domain, and we are kludging in a restriction to limit that domain to a subdomain. For instance if you allow a user to run the vim command on a specific file to edit it, the user can just get a shell with :!sh[Enter].

Allow more than one command

If we want the user to not only execute a single command, but a number of commands, such as:

  • Show process list (ps)
  • Show virtual memory statistics (vmstat)
  • Stop and start the print server (/etc/init.d/cupsys stop/start)

Following the second approach described above, this would give us four key pairs, four entries in ~/.ssh/authorized_keys, and four entirely different invocations of SSH on the client side, each of them using a dedicated private key. In other words: An administrative nightmare.

This is where the environment variable $SSH_ORIGINAL_COMMAND comes in.

Until now, all we know is that with a forced command in place, the SSH server ignores the command requested by the user. This is not entirely true, though. The SSH server does in fact remember the command that was requested, stores it in $SSH_ORIGINAL_COMMAND and thus makes it available within the environment of the forced command.

With this in mind, it is possible to allow more flexibility inside forced commands, without the need to go crazy with countless key pairs. Instead, it is possible to just create a wrapper script that is called as the forced command from within ~/.ssh/authorized_keys and decides what to do, based on the content of $SSH_ORIGINAL_COMMAND:

#!/bin/sh
# Script: /usr/local/bin/wrapper.sh 

case "$SSH_ORIGINAL_COMMAND" in
    "ps")
        /bin/ps -ef
        ;;
    "vmstat")
        /usr/bin/vmstat 1 100
        ;;
    *)
        echo "Sorry. Only these commands are available to you:"
        echo "ps, vmstat"
        exit 1
        ;;
esac

It is important to be aware of potential security issues here, such as the user escaping to a shell prompt from within one of the listed commands. Setting the “no-pty” option already makes this kind of attack somewhat difficult. In addition, some programs, such as “top”, for example, have special options to run them in a “secure” read-only mode. It is advisable to closely examine all programs that are called as SSH forced commands for well-meant “backdoors” and to find out about securing them.

If you encounter problems while debugging $SSH_ORIGINAL_COMMAND, please make absolutely sure that you are authenticating with the correct key. I found it helpful to unset SSH_AUTH_SOCK in the window where I do my testing, in order to prevent intervention from identies stored in the SSH agent.

SSH properly escapes $SSH_ORIGINAL_COMMAND, that is not to say you shouldn’t be diligent about what you do with user-supplied input.

(You should check some security checks for the user input, though)

Batch job authentication

To let a batch job be executed over ssh it need to get access to the private key, there are three ways of doing this.

Store the encrypted key and its passphrase in the filesystem

I don't recommend this method since you can store an unencrypted key in teh filesystem with the same level of security ( and considerably less complication). In either case you rely solely on the filesystem's protections to keep the key secure

Use a plaintext key

Plaintext keys are frightening. To steal the key, an attacker needs to override filesystem protections only one. If you need your batch jobs to continue working after an unattended system restart, plaintext keys are pretty much your best option.

Using an agent

ssh-agent provides another, somewhat less vulnerable method of key storage for batch jobs. A human invokes an agent and loads the needed keys from passphrase-protected key files, jsut once. Thereafter, unatttended jobs use this long-running agent for authentication.

In this case the keys are still in plaintext but within the memory space of the running agent rather than in a file on disk. AS a matter of practical cracking, it si more difficult to extract a data structure from the address space of a running process than to gain illicit access to a file. Also, this solution avoids the problem of an intruder walking off with a backup tape containing the plaintext key.

Security can still be compromised by other methods though. The agent provides access to its services via a Unix-domain socket, which appears as a node in the filesystem. Anyone who can read and write that socket might e able to instruct the agent o sign authentication requests and thus gain use of the keys. In any event, this compromise isn't quite so devastating since the attacker can't obtain the actual keys through the agent socket. She merely gains use of the keys for as long as the agent is running and as long as she can maintain her compromise of the host.

This method does have a down side: the system can't continue unattended after a reboot.

Another bit of complication with the agent method is that you must arrange for the batch jobs to find the agent. SSH clients locate an agent via an environmental variable pointing to the agent socket such as SSH_AUTH_SOCK for the Open SSH agent.

ssh-agent | head -2 > ~/agent-info
source ~/agent-info
rm ~/agent-info

And you need to ensure that he batch jobs (and nobody else!) can read and write the socket. If there's only one uid using the agent, the simplest thing to do is start the agent under that uid

Chrooting

The term chroot refers to a process of creating a virtualized environment in a Unix operating system, separating it from the main operating system and directory structure. This process essentially generates a confined space, with its own root directory, to run software programs. This virtual environment runs separately from the main operating system’s root directory. Any software program run in this environment can only access files within its own directory tree. It cannot access files outside of that directory tree.

The working of Chroot Jail is that it create a directory tree where we copy or link in all the system files needed for a process to run. Then use the chroot system call to change the root directory to be at the base of this new tree and start the process running in that chroot’d environment. Since it can’t actually reference paths outside the modified root, it can’t maliciously read or write to those locations.

In this section we will setup the chroot jail environment for SSH users to encounter situations where we need some specific user access to limited resources on the system like to a web server.

Creating Chroot Directory:

Create a new directory that will be used for chroot jail or you can also choose the existing directory to be used for chroot jail. We are going to two directories to increase the security level by using the following command.

mkdir -p /secure/jail

The Jail directories can only be owned by the root so that no user can break the jail and that’s only possible if we give it the ownership or permissions to do so. You change the ownership and permissions using below commands.

chown root:root /secure/jail
chmod 755 /secure/jail

Creating Dev Node Entries:

There are some necessary files that we called as device nodes for ssh users access, which are required for any user process to execute in the jailed area. Let’s run the following commands to assign them or create nodes pointing to that directory using ‘mknod’ commands shown below.

cd /secure/jail
mkdir dev/
mknod -m 666 dev/null c 1 3
mknod -m 666 dev/tty c 5 0
mknod -m 666 dev/zero c 1 5
mknod -m 666 dev/random c 1 8

Bash Installation:

We need to install bash in the jailed directory that we have created earlier by checking the dependency of shell using below ‘ldd’ command.

ldd /bin/bash

Now give the following command to create and install bash in the jailed directory.

mkdir bin
cp -v /bin/bash bin
mkdir -p lib64/
cp /usr/lib64/libtinfo.so.5 lib64/
cp /usr/lib64/ld-linux-x86-64.so.2 lib64/
cp /usr/lib64/libdl.so.2 lib64/
cp /usr/lib64/libc.so.6 lib64/

This is awful if you want to have several commands that depend on several libraries, I've slightly modified the script given by cyberciti which handles it automatically.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
#!/bin/bash
# Use this script to copy shared (libs) files to Apache/Lighttpd chrooted 
# jail server.
# ----------------------------------------------------------------------------
# Written by nixCraft <http://www.cyberciti.biz/tips/>
# (c) 2006 nixCraft under GNU GPL v2.0+
# + Added ld-linux support
# + Added error checking support
# ------------------------------------------------------------------------------
# See url for usage:
# http://www.cyberciti.biz/tips/howto-setup-lighttpd-php-mysql-chrooted-jail.html
# -------------------------------------------------------------------------------
# Set CHROOT directory name
BASE="/home/testscp/secure"

if [ $# -eq 0 ]; then
  echo "Syntax : $0 /path/to/executable"
  echo "Example: $0 /usr/bin/php5-cgi"
  exit 1
fi

[ ! -d $BASE ] && mkdir -p $BASE || : 

# iggy ld-linux* file as it is not shared one
FILES="$(ldd $1 | awk '{ print $3 }' |egrep -v ^'\(')"

echo "Copying shared files/libs to $BASE..."
for i in $FILES
do
  d="$(dirname $i)"
  [ ! -d $BASE$d ] && mkdir -p $BASE$d || :
  /bin/cp $i $BASE$d
done

# copy /lib/ld-linux* or /lib64/ld-linux* to $BASE/$sldlsubdir
# get ld-linux full file location 
sldl="$(ldd $1 | grep 'ld-linux' | awk '{ print $1}')"
# now get sub-dir
sldlsubdir="$(dirname $sldl)/"

if [ ! -f $BASE$sldl ];
then
  echo "Copying $sldl $BASE$sldlsubdir..."
  /bin/cp $sldl $BASE$sldlsubdir
else
  :
fi

SSH Users Setup to Chroot

Run the following command to simply move the users information of your already created users into the chroot directory (just if you need it!!) .

mkdir etc
cp -vf /etc/{passwd,group} etc/

Then open SSHD configuration file and restart its services after adding the following lines in it.

Match User username
    ChrootDirectory /secure/jail

Traffic analysis resistance

Set up Tor hidden services for your SSH servers. This has multiple advantages. It provides an additional layer of encryption and server authentication. People looking at your traffic will not know your IP, so they will be unable to scan and target other services running on the same server and client. Attackers can still attack these services but don’t know if it has anything to do with the observed traffic until they actually break in.

Now this is only true if you don’t disclose your SSH server’s fingerprint in any other way. You should only accept connections from the hidden service or from LAN, if required.

If you don’t need LAN access, you can add the following line to /etc/ssh/sshd_config:

ListenAddress 127.0.0.1:22

Add this to /etc/tor/torrc:

HiddenServiceDir /var/lib/tor/hidden_service/ssh
HiddenServicePort 22 127.0.0.1:22

Once you restart the tor service you will find the hostname you have to use in /var/lib/tor/hidden_service/ssh/hostname. You also have to configure the client to use Tor. For this, socat will be needed. Add the following line to /etc/ssh/ssh_config:

Host *.onion
    ProxyCommand nc -x localhost:9050 %h %p

In case you have the server protected under fwknop use this command to open the gates

fwknop -a "$(curl -s --proxy socks5h://127.0.0.1:9050 'https://check.torproject.org' | egrep -o "([[:digit:]]{1,4}\.?){4}" | tail -n1)" -n <server_stanza>

Given that tor is running on the default port

If you want to allow connections from LAN, don’t use the ListenAddress line, configure your firewall instead.

Keyserver key rotation

The original author of this section wanted the SSH protocol to provide a way to get users onto better host key algorithms for a while and finally got around to implementing it on OpenSSH 6.8, that shipped with a protocol extension that allows a server to inform a client of all of its host keys, and support in the client to update known_hosts when such a message is received. So, when an OpenSSH ≥6.8 client connects to a OpenSSH ≥6.8 server (or any other client/server that adopts the extension) where the user already trusts or explicitly accepts the host key, the user's known_hosts file will be updated with all the server's host keys, not just the one that authenticated the host during key exchange.

This fixes both the shortcomings I mentioned above: first, the client learns all the server's host key types, and can select the best possible host key algorithm (ed25519 is the current favourite) on subsequent connections. Secondly, it allows a server to gracefully rotate keys by publishing additional keys for a period to allow clients to learn them, before removing the deprecated key(s) and letting the new ones become the primary ones.

To practically rotate host keys, the operator of a sshd server should add additional HostKey statements to their sshd_config for the new keys, while keeping the existing keys in place. I.e.

# Old keys
HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_dsa_key
# New keys
HostKey /etc/ssh/ssh_host_ed25519_key_new

Once these lines are added, sshd can be restarted to begin offering the new keys to clients who connect. The old keys, appearing first in the configuration, will be the ones actually used to authenticate the host to the client. Once the operator is satisfied that enough users have had a chance to pick up the new keys, they can rename replace the original keys with the new keys and remove the extra HostKey lines from sshd_config.

This mechanism isn't perfect: first, it only works for users who actually connect to the server - if they don't happen to connect during the grace period when both old and new keys are offered then they will have to learn the new key manually afterwards. As such, it doesn't cope well with sudden key rotations (e.g. when the operator has reason to believe a key has been compromised).

Fortunately both these cases can be addressed with a bit of forethought: when setting up a server, generate some reserve keys. Keep their private halves offline (e.g in a safe), but list them in sshd_config as described above. OpenSSH ≥6.8 will notify clients of the reserve public keys on every connection, so if you ever need to rotate keys in a hurry then they are already ready to go. You could generate multiple sets of reserve keys if you like - you aren't limited to a single set.

ssh-copy-id

ssh-copy-id is a script that uses ssh(1) to log into a remote machine (pre‐ sumably using a login password, so password authentication should be enabled, unless you've done some clever use of multiple identities). It assembles a list of one or more fingerprints (as described below) and tries to log in with each key, to see if any of them are already installed (of course, if you are not using ssh-agent(1) this may result in you being repeatedly prompted for passphrases). It then assembles a list of those that failed to log in, and using ssh, enables logins with those keys on the remote server. By default it adds the keys by appending them to the remote user's ~/.ssh/authorized_keys (creating the file, and directory, if necessary). It is also capable of detecting if the remote system is a NetScreen, and using its ‘set ssh pka-dsa key ...’ command instead.

Multi-Factor authentication

I haven't found how to set up your proper TOTP multi-factor authentication server, and I wouldn't want to use third party servers, therefore I wont use this feature.

But for reference, it is also possible to use OTP authentication to reduce the consequences of lost passwords. Google Authenticator is one implementation of TOTP, or Timebased One Time Password. You can also use a printed list of one time passwords or any other PAM module, really, if you enable ChallengeResponseAuthentication.

Remove the r-Commands

If your distro has them installed remove them, in Debian 8 they are linked to their secure brothers.

Use subsystems

This is not a security measure although mixed with ForceCommand will do the trick, its basically a nice alias for sysadmins to execute remote commands. For example

Subsystem backups       /usr/local/sbin/tape-backups

Now we could execute the command with

ssh server.example.com -s backups

Hardware security

If having the unencrypted private key in memory is not secure enough for you. Instead you could use hardware security (smart cards) to avoid leaking keys even from memory dumps. It's not covered in this post, mainly because it requires a hardware device you need to buy and secondly because the limitations are device dependent.

You may want to store them on a pendrive and only plug it in when you want to use SSH. Are you more likely to lose your pendrive or have your system compromised? I don’t know.

Hide SSH version

Another measure of security through obscurity == no security at all. Anyway if that's not enough here are some other arguments to argue why it's not a good idea:

  • Nmap gets this information (I should think) from the way the software behaves, not it's version number. In much the same way that it differentiates between Linux 2.4 and 2.6 TCP/IP stacks... by passive analysis of how the software operates. That's not something that's going to be easy to change.
  • Crackers are not going to stop trying methods of breaking other versions of ssh just because you said it was other version -.-

Short answer from the OpenSSH FAQ:

This information is used by clients and servers to enable protocol compatibility tweaks to work around changed, buggy or missing features in the implementation they are talking to. This protocol feature checking is still required at present because versions with incompatibilities are still in wide use.

Flip Feng Shui

Although unlikely it's nice to know it's existence.

Flip Feng Shui (FFS) is a new exploitation vector that allows an attacker virtual machine (VM) to flip a bit in a memory page of a victim VM that runs on the same host as the attacker VM. FFS relies on a hardware vulnerability for flipping a bit and a physical memory massaging primitive to land a victim page on vulnerable physical memory location.

While the requirements for FFS may seem unrealistic, the study shows that it is possible to implement FFS reliably today in the cloud using Rowhammer, a wide-spread DRAM glitch, and memory deduplication, a popular memory management feature that reduces physical memory footprint of VMs by merging memory pages with the same content.

Their first attack flips a bit in the page cache of a victim VM storing the authorized_keys file of OpenSSH. authorized_keys files stores the (often) RSA public key. A user with the RSA private key associated with that public can then login to the SSH server.

The security of this scheme depends heavily on the fact that the private key cannot be easily derived from the (known) public key by factorization. They perform a FFS attack on the public key to flip one of its bits. A bit flipped public key becomes much easier to factorize. Once they have all the factors, they can generate a new private key corresponding to the bit flipped public key and SSH with an unmodified OpenSSH client.

For more information visit their project site

Client side

SSH client configuration

To configure the clients we have again three places to set the options. Configuration data is parsed as follows:

  1. command line options
  2. user-specific file
  3. system-wide file

We are going to explain the interesting options that can't be deduced from the server ones.

system-wide file

# File: /etc/ssh/ssh_config
# This is the ssh client system-wide configuration file.  See
# ssh_config(5) for more information.  This file provides defaults for
# users, and the values can be changed in per-user configuration files
# or on the command line.

# Configuration data is parsed as follows:
#  1. command line options
#  2. user-specific file
#  3. system-wide file
# Any configuration value is only changed the first time it is set.
# Thus, host-specific definitions should be at the beginning of the
# configuration file, and defaults at the end.

# Site-wide defaults for some commonly used options.  For a comprehensive
# list of available options, their meanings and defaults, please see the
# ssh_config(5) man page.

Host *
   Protocol 2
   Port 22
#   BindAddress 192.168.1.123
   ForwardAgent no
   ForwardX11 no
   ForwardX11Trusted yes
   RhostsRSAAuthentication no
   RSAAuthentication no
   PasswordAuthentication no
   ChallengeResponseAuthentication no
   PubkeyAuthentication yes
   HostKeyAlgorithms ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ssh-rsa
   Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes256-ctr
   MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
   KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
   HostbasedAuthentication no
   GSSAPIAuthentication no
   GSSAPIDelegateCredentials no
   GSSAPIKeyExchange no
   GSSAPITrustDNS no
   CheckHostIP yes
   AddressFamily inet
   StrictHostKeyChecking yes
   BatchMode no
   IdentityFile my-key
   IdentitiesOnly yes
#   IdentityFile ~/.ssh/identity
#   IdentityFile ~/.ssh/id_rsa
#   IdentityFile ~/.ssh/id_dsa
#   ConnectTimeout 0
#   EscapeChar ~
#   Tunnel no
#   TunnelDevice any:any
#   PermitLocalCommand no
#   VisualHostKey no
#   ProxyCommand ssh -q -W %h:%p gateway.example.com
#   RekeyLimit 1G 1h
    SendEnv LANG LC_*
    HashKnownHosts yes
    GSSAPIAuthentication no
    GSSAPIDelegateCredentials no

# Github needs diffie-hellman-group-exchange-sha1 some of the time but not always.
Host github.com
    KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1
  • Ensure the known_hosts file are unreadable if leaked - it is otherwise easier to know which hosts your keys have access to.
  • IdentitiesOnly, only check for the identities specified by IdentityFile, by default it will test all the identities available in the agent.
  • BindAddress: Force openssh to connet through the selected interface
  • AddressFamily: Force Openssh to use IPv4
  • BatchMode: If set to yes", passphrase/password querying will be disabled. Usefull for scripts and other batch jobs where no user is present to supply the password. This makes error reporting more straightforward, eliminating some confusing SSH messages about failing to access a tty.

User specific file

If we don't have access to the server configuration or we want to make an user based configuration we can edit ~/.ssh/config.

For example

Host *
    Protocol 2

Host hostname
    Hostname myserver.example.com
    User desired_username
    IdentityFile my-key
    IdentitiesOnly yes
    Port 22
    BindAddress 192.168.10.235
    AddressFamily inet

Host * 
    Compression yes

If a host matches more than one section of the config file it will take the value of the first, this can be used for avoiding errors, such is the case of the Protocol 2 in the first section of the previous config. Alternatively if you place Host * at the last section in the configuration file its settings are used only if no other section overrides it.

In the case of the IdentifyFile keyword, the values wont get overriden but accumulated.

Authorization file

Remember that these aren't strong restrictions, as long as the user has shell access she can change them.

If you have to set several flags separate them with ',' and with no whitespaces

Flags: no-port-forwarding no-x11-forwarding no-agent-forwarding idle-timeout=5m from=myhost command="/usr/bin/pine" environment="EDITOR=vim" no-pty * permitopen="server.example.com:12345" restrict port forwarding to local port 12345 connecting to remote host server.example.com

SSH Jumping

To connect to a remote hosts with jumps in between we could use ssh agent forwarding, but since OpenSSH >=7.3 we can use jumps which is a much more secure method.

It is well known that agent forwarding (forwarding of the ssh authentication agent connection) can be considered, under certain conditions, a security risk, because when you do that a socket is created on the remote host, and any user with access to the socket would be able to authenticate using the keys loaded in the agent.

The reality is that any user with enough privileges (root-level access, or access to the same user you are using to login) on the remote host will be able to use your forwarded agent through the local socket, and thus authenticate using the keys you loaded into the agent, potentially gaining access to other servers you manage, for as long as you keep your session open. It is worth noting that the access is limited to using the keys, but not fetching them, ie, the agent doesn't allow access to the private keys, but only answers authentication requests.

Because of this, it is not a good idea to forward the agent to a server you don't trust (either because you don't know and/or don't trust the other administrators, or because you can't tell for sure if the server is compromised). If in doubt, don't do it, if you really have to do it, limit the keys you add to the agent.

OpenSSH 7.3 onwards allow users to jump through several hosts in a rather automated fashion. It has full support for scp and sftp commands as well as regular ssh.

For example to reach a host behind a bastion/jumphost:

# Single jump
ssh -J ssh.jumpserver.com destination.server.com

# Jump through 2 hops
ssh -J ssh.jumpserver1.com,ssh.jumpserver2.com destination.server.com

# Copy data from a host
scp -oProxyJump=ssh.jumpserver.com destination.server.com:testfile .

You can also add these lines to your ~/.ssh/config

Host destination.server.com
    ProxyJump ssh.jumpserver.com

SSH-Agent

ssh-agent is a program to hold private keys used for public key authentication (RSA, DSA, ECDSA, ED25519). ssh-agent is usually started in the beginning of an X-session or a login session, and all other windows or programs are started as clients to the ssh-agent program. Through use of environment variables the agent can be located and automatically used for authentication when logging in to other machines using ssh(1).

This should not be used in a security minded environment because the private keys will live in the memory of the machine, something that it's not desired by default.

Nevertheless in case you want to add a key to the ssh-agent

Check the loaded keys

ssh-add -l

Add a key with a lifetime of 3600 seconds

ssh-add -t 3600 $key_file

Remove all the keys

ssh-add -D 

Use multiple keys

It's interesting to use multiple keys for the different hosts you access, so in case one get's cracked or compromised some way, it doesn't compromise the rest of your infrastructure.

This can be done through the command line

ssh -i key_file ...

Or via the ssh config file

Protection of the keys

Protection of user keys

  • Protected by strong passphrase.
  • Never copied to another system than your own workstation/personal physical disks/tokens.
  • Use SSH jumps, forwarding or tunneling if you need to jump between hosts. DO NOT maintain unnecessary agent forwarding when unused.

Protection of machine keys

When SSH keys are necessary for automation between systems, it is reasonable to use passphrase-less keys.

  • The recommended settings are identical to the user keys.
  • The keys must be accessible only by the admin user (root) and/or the system user requiring access.
  • Usage of machine keys should be registered in an inventory (a wiki page, ldap, an inventory database), to allow for rapid auditing of key usage across an infrastructure.
  • The machine keys should be unique per usage. Each new usage (different service, different script called, etc.) should use a new, different key.
  • Only used when strictly necessary.
  • Restrict privileges of the account (i.e. no root or "sudoer" machine account).
  • Using a ForceCommand returning only the needed results, or only allowing the machine to perform SSH-related tasks such as tunneling is prefered.

ssh-addhost

I've started a python automating script to manage multiple keys, generate new keys if necessary and be a front-end of ssh-copy, but I've seen that maybe I'm trying to rewrite Ansible :).

Follow ups

  • https://www.reddit.com/r/ssh/
  • https://www.openssh.com/list.html
  • https://www.ssh.com
  • http://blog.djm.net.au/
  • https://blog.g3rt.nl/

src

  • SSH: The Secure Shell, The definitive guide by Daniel J. Barret, Richard E Silverman and Robert G. Byrnes
  • man {sshd_config,ssh_config,ssh}
  • https://wiki.gentoo.org/wiki/Security_Handbook/Securing_services#ssh
  • https://wiki.mozilla.org/Security/Guidelines/OpenSSH
  • https://stribika.github.io/2015/01/04/secure-secure-shell.html
  • http://blog.djm.net.au/2015/02/key-rotation-in-openssh-68.html
  • https://debian-administration.org/article/590/OpenSSH_SFTP_chroot_with_ChrootDirectory
  • https://www.rackaid.com/blog/how-to-harden-or-secure-ssh-for-improved-security/
  • https://binblog.info/2008/10/20/openssh-going-flexible-with-forced-commands/
  • http://serverfault.com/questions/83856/allow-scp-but-not-actual-login-using-ssh
  • http://stackoverflow.com/questions/402615/how-to-restrict-ssh-users-to-a-predefined-set-of-commands-after-login
  • http://linuxpitstop.com/chroot-ssh-users-on-centos-7/
  • https://wiki.archlinux.org/index.php/SFTP_chroot
  • https://www.commandprompt.com/blog/security_considerations_while_using_ssh-agent/
  • https://blog.g3rt.nl/upgrade-your-ssh-keys.html
  • http://www.linuxquestions.org/questions/slackware-14/how-to-hide-openssh-version-331399/