Blog

Using SSH Through A Bastion Host Transparently

By Rob Giseburt

A Bastion host is a special purpose computer on a network specifically designed and configured to withstand attacks. The computer generally hosts a single application, for example a proxy server, and all other services are removed or limited to reduce the threat to the computer. It is hardened in this manner primarily due to its location and purpose, which is either on the outside of the firewall or in the DMZ and usually involves access from untrusted networks or computers.
Wikipedia

Having a bastion host is a good security practice commonly deployed to strengthen yet simplify security controls of an environment.  However, adding bastion hosts creates complexity in remote execution of scripts or deployment tasks.  Here we will be using a bastion host to serve as a SSH server that we can “hop” through into another machine (real or VM), allowing users to automate remote task execution over SSH. We will be demonstrating how to make that connection transparent and automatic, not only for manual SSH connections but also for programmatic SSH connections such as with GIT or Ansible.

The purpose of using a bastion host for access is clearly a matter of increased security. We won’t be going deeply into the security implications in this article. Instead, we will discuss

  1. The mechanics of using SSH to connect to the bastion host, and from there SSH to another machine without having to store authentication information on the bastion host.
  2. Making the connection through the bastion host to the destination machine in one step.
  3. Making the connection transparently using the bastion host based on destination.

In follow-up posts, I’ll cover these related topics:

  • Dynamically located SSH bastion hosts with AWS
  • SSH connection multiplexing, port forwarding, and use as a SOCKS proxy
  • Using SSH bastion hosts with AWS, and dynamically locating them with EC2 tags
  • Configuring Ansible to use an SSH bastion host

1. SSH to the bastion, and beyond

In order to efficiently and (arguably) securely connect to multiple machines from one machine, we’ll be using Public Key Authentication in SSH. There are many references on the internet about how to generate key pairs that can be used for ssh, such as for GitHub, Amazon AWS, Google Compute Engine, and there’s always the ssh-keygen(1) man page.

However you get there, we’ll need to end up with:

  • One or more key pairs in ~/.ssh/ that are secured by password(s).
    • From here on we’ll assume one key pair per machine you’re connection to, including one for the bastion host itself. You could of course use a single key pair for all machines, or one key pair for the bastion host and another for all the other machines, or any combination of the above.
  • That each machine (including the bastion host) already has the relevant public-key copied to ~/.ssh/authorized_keys, thereby granting access to the holder of the corresponding private key — hopefully, that’s just you! This should be tested and proven to allow login to that machine from the client machine without an account password. If it fails, check for incorrect permissions to the file and it’s containing directories, and verify that authorized-keys is not disabled in the sshd config of the destination machine.
  • ssh-agent(1) is running and all of the key pairs to be used have been ssh-add(1)ed, thus allowing each key to be used without (further) entering a password.
    • NOTE: In OS X, the ssh-agent is automatically started at login, and has been extended to grab an SSH key’s passphrase from the user’s keychain. When you generate a key, call ssh-add with the OS X-only -K option to add it’s passphrase to the user’s keychain. The automatically started ssh-agent will then be able to access those keys in the keychain as long as the keychain is unlocked. (The keychain is usually unlocked as long as the user is logged in and the screen isn’t locked.)
    • For linux/unix OSs, it’s recommended to add ssh-agent to the shell’s login, and then call ssh-add after login and before using ssh.

With that the basics of public-key authentication are handled: You can ssh into the bastion host without a password. If you have means to directly access to the various machines you intend to connect to through the bastion host (such as through a VPN tunnel or temporarily connecting to the same physical network) then you should be able to ssh into them without an account password as well.

However, you will likely not be able to ssh through the bastion host into the destination machine with simple ssh commands, since the bastion host doesn’t have the private keys to the destination machine. We don’t want to place our private keys onto the bastion host for many reasons. First, we don’t want to specifically give the keys to the inside machines right on the outwardly-accessible bastion host. Second, there may be many people accessing through that bastion host via that same account, and we don’t want the keys of those various users on the bastion.

In order to solve this problem without putting the private keys onto the bastion host, we’ll need to enable “SSH agent forwarding,” which allows the ssh agent running on your local machine to provide the keys to the connection from the bastion host to the destination machine. There’s a great article on setting up ssh agent forwarding on GitHub. In short, you can enable forwarding one of two ways:

  • Per-connection — add -A to the ssh line when connecting to the bastion host:
  • ssh -A user@bastion_host.com
  • Per-host via ~/.ssh/config — add a section for the bastion host with ForwardAgent yes:
    Host example.com
        ForwardAgent yes
    • Don’t use a wildcard host — you will be extending access to all of the unlocked keys in your ssh-agent to every machine you ssh to, including through other subsystems that may use ssh, such as Ansible and git. This is a bad idea.

We’ll go with the configuration-file method, since we’re going to build upon that shortly.

With ssh-agent running locally and usable, the keys ssh-add‘ed to it so that they are unlocked, each machine having the correct public key in it’s ~/.ssh/authorized_keys, and ssh agent forwarding enabled for the bastion host, you should be able to ssh through the bastion host and then ssh from there into the destination machines:

my_machine# ssh user@bastion.com
bastion.com# ssh user@destination.com
destination.com# echo 'Success!'
Success!

2. SSH through the bastion in one command

We can combine the two connection steps into one line, simplifying the process:

my_machine# ssh user@bastion.com ssh user@destination.com
destination.com# echo 'Success!'
Success!

This simply executes the second ssh call on the bastion host instead of opening a shell. This is almost as if we had done the connection in two steps, except that when we disconnect from the destination machine it’ll also close the connection to the bastion host.

3. SSH through the bastion, transparently

The next step is to eliminate the need to connect to the bastion host explicitly, but instead when we attempt to ssh to the destination machine, we will automatically and transparently connect to the bastion host and then hop to the destination machine in one call.

We’re going to utilize the ssh ProxyCommand setting applied to the specific host or hosts to tell ssh to setup our connection to the bastion host first, then use that connection to ssh to the final machine. On disconnect this will automatically tear down the intermediate connection as well.

Host destination.com
    ProxyCommand ssh -A user@bastion.com -W %h:%p

This tells ssh to run the command ssh -A user@bastion.com -W host:port and then talk to the standard in/out of that command as if it’s the remote connection.

That proxy ssh command has several things going on. First, it has the -A that we saw before to turn on ssh agent forwarding. The real magic is the -W host:port, which tells it to redirect the standard in/out to the specified host and port, which was designed specifically for using ssh as a ProxyCommand. And finally, we have the %h and %p tokens that will get replaced in the ProxyCommand call, which allows us to use wildcards and multiple host names in the Host ... line.

With this setup, connecting to destination.com through the bastion host bastion.com is automatic and transparent:

my_machine# ssh user@destination.com
destination.com# echo 'Success!'
Success!

 What’s next?

In the next post, we cover using bastion hosts in AWS and dynamically locating the SSH bastion hosts based on EC2 tags.

Categories: Blog, Uncategorized

Tags: , , , , , , ,

Rob Giseburt
14 Jan, 2015