Using Ansible through a Bastion Host

Blog Detail

This blog is going to go over the concepts of using Ansible to interact with a private network through a bastion host. The use of bastion hosts is not new to the industry—they have been used for a while by many companies that need to give users access to private networks. Bastion hosts are typically public facing, hardened systems that work as an entrypoint to systems that are behind a firewall or any other restricted locations.

Set up SSH public key authentication with bastion host

When interacting with a bastion host, the recommended first step is to set up SSH public keys and authenticate with the bastion host.

If an SSH key doe not already exist, create it by issuing the command ssh-keygen -t rsa

  root@eb54369adc49:/ntc# ssh-keygen -t rsa
  Generating public/private rsa key pair.
  Enter file in which to save the key (/root/.ssh/id_rsa):
  Created directory '/root/.ssh'.
  Enter passphrase (empty for no passphrase):
  Enter same passphrase again:
  Your identification has been saved in /root/.ssh/id_rsa.
  Your public key has been saved in /root/.ssh/id_rsa.pub.
  The key fingerprint is:
  SHA256:T5YDoF91QrJk2ZN1b7OYq9Pfn3OWVW518iCJHGIQKeQ root@eb54369adc49
  The key's randomart image is:
  +---[RSA 2048]----+
  |   .. +++++oo .  |
  |   ....+++=o . . |
  |    E. .+o + . .o|
  |     . . .o.o =.*|
  |      . S =  + *+|
  |         + .  . =|
  |          . .. .o|
  |           ... o=|
  |           .. .+*|
  +----[SHA256]-----+
  root@eb54369adc49:/ntc#
  root@eb54369adc49:/ntc#

Since this is just for demonstration, I’ve kept everything as default when creating the keys.

Note: Click here if you want to know more about how to set up an SSH public key with more details.

After creating the public key, the ssh-copy-id Linux command can be used to transfer the id_rsa.pub public key to the bastion host.

  root@eb54369adc49:/ntc#
  root@eb54369adc49:/ntc# ssh-copy-id ntc@bastion
  /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
  /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are   already installed
  /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to   install the new keys
  ntc@bastion password:
  
  Number of key(s) added: 1
  
  Now try logging into the machine, with:   "ssh 'ntc@bastion'"
  and check to make sure that only the key(s) you wanted were added.
  
  root@eb54369adc49:/ntc#

Finally, attempt to SSH into the bastion host and make sure it works as expected. The workstation should log in through SSH without having to provide a password.

  root@eb54369adc49:/ntc# ssh ntc@bastion
  Welcome to Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-59-generic x86_64)
  
   * Documentation:  https://help.ubuntu.com
   * Management:     https://landscape.canonical.com
   * Support:        https://ubuntu.com/advantage
  
  466 packages can be updated.
  321 updates are security updates.
  
  New release '18.04.3 LTS' available.
  Run 'do-release-upgrade' to upgrade to it.
  
  Last login: Mon Jan 13 13:44:58 2020 from 71.52.24.123
  ntc@bastion:~$
  ntc@bastion:~$ exit
  logout
  Connection to bastion closed.
  root@eb54369adc49:/ntc#

There are many different ways of creating a public key and transferring it to the remote host. This transfer can also be handled via Ansible, as the below playbook shows:


---
    - name: Generate Key
      hosts: localhost
      connection: local
      gather_facts: no

      tasks: 
          - name: Generate an OpenSSH keypair with the default values (4096 bits, rsa)
            openssh_keypair:
                path: /root/.ssh/id_rsa
                owner: root
                group: root

          - name: Deploy public key to remote host
            shell: ssh-copy-id ntc@bastion

Setting up Ansible to Interact with a Cisco IOS Device through a Bastion Host

Now that it is possible to authenticate into the bastion host only using SSH keys and not require user/passwords, it’s time to build the Ansible inventory file with the devices that are going to be targeted by Ansible. Before creating the inventory file, notice how the only way to interact with the Cisco csr1 device is through the bastion host.

 # Topology
 ┏━━━━━━━━━━━━━━┓          ┏━━━━━━━━━━━━━━┓          ┏━━━━━━━━━━━━┓         
 ┃  Container   ┃──────────┃ Bastion host ┃──────────┃    csr1    ┃
 ┗━━━━━━━━━━━━━━┛          ┗━━━━━━━━━━━━━━┛          ┗━━━━━━━━━━━━┛

This first test is using the ping command to ping the csr1 device from the local container. The results should come back with 100% packet loss.

  root@eb54369adc49:/ntc#
  root@eb54369adc49:/ntc#
  root@eb54369adc49:/ntc# ping csr1 -c 5
  PING csr1 (54.84.81.49) 56(84) bytes of data.
  
  --- csr1 ping statistics ---
  5 packets transmitted, 0 received, 100% packet loss, time 4176ms

After using the SSH command to access the bastion host, the ping command is used again to show that there is IP connectivity between the bastion host and the networking device.

  root@eb54369adc49:/ntc#
  root@eb54369adc49:/ntc# ssh ntc@bastion
  Welcome to Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-59-generic x86_64)
  
   * Documentation:  https://help.ubuntu.com
   * Management:     https://landscape.canonical.com
   * Support:        https://ubuntu.com/advantage
  
  466 packages can be updated.
  321 updates are security updates.
  
  New release '18.04.3 LTS' available.
  Run 'do-release-upgrade' to upgrade to it.
  
  Last login: Mon Jan 13 14:38:26 2020 from 71.52.24.123
  ntc@bastion:~$
  ntc@bastion:~$ ping csr1 -c 5
  PING csr1 (10.0.0.51) 56(84) bytes of data.
  64 bytes from csr1 (10.0.0.51): icmp_seq=2 ttl=255 time=2.34 ms
  64 bytes from csr1 (10.0.0.51): icmp_seq=3 ttl=255 time=2.26 ms
  64 bytes from csr1 (10.0.0.51): icmp_seq=4 ttl=255 time=1.99 ms
  64 bytes from csr1 (10.0.0.51): icmp_seq=5 ttl=255 time=2.23 ms
  
  --- csr1 ping statistics ---
  5 packets transmitted, 4 received, 20% packet loss, time 4013ms
  rtt min/avg/max/mdev = 1.994/2.209/2.342/0.138 ms
  ntc@bastion:~$

Now that you can verify direct connectivity from the container to the csr1 device is only possible through the bastion host, the ProxyCommand can be used to gain access to the Cisco device from the container.

Note: ProxyCommand is an SSH builtin command feature to provide support for proxy use cases.

  root@eb54369adc49:/ntc#
  root@eb54369adc49:/ntc# ssh -o ProxyCommand="ssh -W %h:%p ntc@bastion" ntc@csr1
  The authenticity of host 'csr1 (<no hostip for proxy command>)' can't be established.
  RSA key fingerprint is SHA256:BaRERnJrQ4vzALKQELc6lIs7Kujbe3UCPffPcAhcRKg.
  Are you sure you want to continue connecting (yes/no)? yes
  Warning: Permanently added 'csr1' (RSA) to the list of known hosts.
  Password:
  
  
  
  csr1#
  csr1#

Playbook

For this test the ios_command and ios_config will be used. The backup parameter has been enabled to show that the backup files are stored in the local machine and not on the bastion host.


---

    - name: Cisco IOS Playbook
      hosts: ios
      connection: network_cli
      gather_facts: no

      tasks:

        - name: SHOW VERSION
          ios_command:
            commands: show version

        - name: CONFIGURE SNMP COMMUNITY
          ios_config:
            commands: snmp-server community ntc-blog RW
            backup: true

Inventory

For now, only the basic credentials and device type are required to target the device.

  [all:vars]
  ansible_user=ntc
  ansible_ssh_password=ntc123
  
  [ios]
  csr1 
  
  [ios:vars]
  ansible_network_os=ios

Note: There is a dedicated group vars for IOS only for now, since a Juniper device will be tested later in this blog too.

First Execution of Playbook

Similar to the previous tests, the local machine should not be able to communicate with the Cisco device, since it does not have direct access to the private network.

  root@eb54369adc49:ntc$ ansible-playbook -i inventory bastion_playbook.yml
  
  PLAY [Cisco IOS Playbook] ******************************************************************
  
  TASK [SHOW VERSION] ************************************************************************
  fatal: [csr1]: FAILED! => {"changed": false, "msg": "timed out"}
  
  PLAY RECAP *********************************************************************************
  csr1 : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
  
  root@eb54369adc49:ntc$

Executing a Playbook against a Cisco IOS Device through a Bastion Host

To be able to communicate with the Cisco device the ProxyCommand needs to be executed when trying to SSH into the device. To start using the ProxyCommand with Ansible the ansible_ssh_common_args, a variable needs to be added to the inventory file.

  [all:vars]
  ansible_user=ntc
  ansible_ssh_password=ntc123
  
  [ios]
  csr1 
  
  [ios:vars]
  ansible_network_os=ios
  ansible_ssh_common_args=ProxyCommand="ssh -W %h:%p ntc@bastion"

More details can be found in the Ansible documentation about how to use the ansible_ssh_common_args and ProxyCommand. For now, I’ve added a device to the inventory file with the needed credentials, and the ProxyCommand used earlier and tested through the Linux terminal.

The Ansible playbook can now be executed normally using the standard ansible-playbook command. The only modification needed to interact with the networking device was enabling SSH keys and using the ansible_ssh_common_args in the inventory file.

  root@eb54369adc49:/ntc# ansible-playbook -i inventory bastion_playbook.yml -v
  Using /ntc/ansible.cfg as config file
  
  PLAY [Cisco IOS Playbook] ***********************************************************
  
  TASK [SHOW VERSION] ***************************************************************
  
  ok: [csr1] => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}
  "changed": false, "stdout": ["Cisco IOS XE Software, 
  Version 16.06.02\nCisco IOS Software [Everest], Virtual XE Software
  (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.6.2, RELEASE
  SOFTWARE (fc2)\nTechnical Support: http://www.cisco.com
  techsupport\nCopyright (c) 1986-2017 by Cisco Systems, Inc
  \nCompiled Wed 01-Nov-17 07:24 by mcpre\n\n\nCisco IOS-X
  software, Copyright (c) 2005-2017 by cisco Systems, Inc
  \nAll rights reserved.  Certain components of Cisco IOS-XE
  software are\nlicensed under the GNU General Public License" ...omitted]]}
  
  TASK [CONFIGURE SNMP COMMUNITY] ***********************************************************
  changed: [csr1] => {"backup_path": "/ntc/backup/csr1_config.2020-01-13@22:02:16", "banners": 
  {}, "changed": true, "commands": ["snmp-server community ntc-blog RW"],
  "date": "2020-01-13", "filename": "csr1_config.2020-01-13@22:02:16", "shortname": "/ntc
  backup/csr1_config", "time": "22:02:16", "updates": ["snmp-server community ntc-blog RW"]}
  
  PLAY RECAP *********************************************************************************
  csr1    : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Setting up Ansible to Interact with a Juniper vMX Device using NETCONF through a Bastion Host

To interact with a Juniper device via a bastion host that uses NETCONF, a different variable is required and a configuration file needs to be built. This link gives some details on different ways to enable the feature, but for this test I’ll be adding the ansible_netconf_ssh_config=./ntc/netconf_proxy_config.cfg variable with the path to my netconf_proxy_config.cfg file.

Note: Even thought the Ansible documentation is not entirely clear why ansible_ssh_common_args does not work with NETCONF devices, maybe this link can provide insight to that question.

  [all:vars]
  ansible_user=ntc
  ansible_ssh_password=ntc123
  
  [ios]
  csr1 
  
  [ios:vars]
  ansible_network_os=ios
  ansible_ssh_common_args=ProxyCommand="ssh -W %h:%p ntc@bastion"
  
  [vmx]
  vmx1
  
  [vmx:vars]
  ansible_network_os=junos
  ansible_netconf_ssh_config=./ntc/netconf_proxy_config.cfg

Setting up the netconf_proxy_config.cfg and ansible.cfg file

The Ansible documentation shows how to build a proper netconf_proxy_config.cfg file. However, since I’m only testing with a single device, the only requirement is the ProxyCommand that will establish the SSH connection with the bastion host and the Juniper device.

ProxyCommand ssh -W %h:%p ntc@bastion

The next step will be to add a second play using the correct modules to interact with a Juniper device.

Executing a Playbook against a Juniper vMX Device using NETCONF through a Bastion Host

The example playbook has been extended to include not only the original Cisco IOS tasks, but an additional play to run the equivalent on Juniper devices. The goal is to run a show version and make a configuration change to add an SNMP community to the device configuration.


---

    - name: Cisco IOS Playbook
      hosts: ios
      connection: network_cli
      gather_facts: no
      tags: cisco

      tasks:

        - name: SHOW VERSION
          ios_command:
            commands: show version

        - name: CONFIGURE SNMP COMMUNITY
          ios_config:
            commands: snmp-server community ntc-blog RW
            backup: true

    - name: Juniper vMX Playbook
      hosts: vmx
      connection: netconf
      gather_facts: no
      tags: juniper
      
      tasks:

        - name: SHOW VERSION
          junos_command:
            commands: show version

        - name: CONFIGURE SNMP COMMUNITY
          junos_config:
            commands: set snmp community ntc-blog authorization read-write

After building the playbook, it’s time to execute it and see the results. In this case I ran the playbook and added a tag so only the Juniper play gets executed.

  root@eb54369adc49:/ntc# ansible-playbook -i inventory bastion_playbook.yml --tags juniper -v
  Using /ntc/ansible.cfg as config file
  
  PLAY [Cisco IOS Playbook] *******************************************************************
  
  PLAY [Juniper IOS Playbook] *****************************************************************
  
  TASK [SHOW VERSION] *************************************************************************
  ok: [vmx1] => {"changed": false, "stdout": ["Hostname: vmx1\nModel: vmx\nJunos: 15.1F4.15
  \nJUNOS Base OS boot [15.1F4.15]\nJUNOS Base OS Software Suite [15.1F4.15]\nJUNOS Crypto 
  Software Suite [15.1F4.15]\nJUNOS OnlineDocumentation [15.1F4.15]\nJUNOS 64-bit Kernel Software 
  Suite [15.1F4.15]\nJUNOS Routing Software Suite [15.1F4.15]\nJUNOS Runtime Software Suite [15.1F4.  15]
  \nJUNOS 64-bit Runtime Software Suite [15.1F4.15] \nJUNOS Services AACL PIC package [15.1F4.15]  \nJUNOS 
  Services Application Level Gateway (xlp64) [15.1F4.15]\nJUNOS Services ..omitted"]]}
  
  TASK [CONFIGURE SNMP COMMUNITY] *************************************************************
  changed: [vmx1] => {"changed": true}
  
  PLAY RECAP **********************************************************************************
  vmx1  : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Conclusion

In order to use Ansible through a bastion host with core modules, SSH keys are recommended. When interacting with devices that support NETCONF, the ansible_netconf_ssh_config variable is required to be enabled and for traditional SSH devices the ansible_ssh_common_args is required.

Other third party modules like ntc_show_command or NAPALM can also be used but will not be able to support ansible_ssh_common_args or ansible_netconf_ssh_config. In this case, the delegate_to argument will be needed to use a third party module with a jump-host/bastion-host.

-Hector



ntc img
ntc img

Contact Us to Learn More

Share details about yourself & someone from our team will reach out to you ASAP!

How to Create an SSH Tunnel via Command line

Blog Detail

Do you have a network device or server that can only be reached behind a jumphost? This is not an uncommon scenario, as security best practice often requires such. This can cause some challanges. However, as long as you have access to the jumphost, you may be able to use an ssh tunnel to mimic being directly connected to a network with access to the otherwise inaccessible hosts.

What is an SSH Tunnel?

An ssh tunnel aka ssh port forwarding, allows an encrypted tunnel to be established over an untrusted network between an SSH Client and SSH server. You specify a local port for SSH to listen on, such as 4001, and all connections destined for port 4001 will be tunneled via SSH to a specified remote port, such as 22.

Access a network device/server that is only accessible via a jumphost

ssh username@172.18.50.100 -Nf -p 20622 -L 127.0.0.1:4001:192.168.20.10:22

Let’s break this down:

  • ssh is the command we are using for our ssh tunnel.
  • username is the username to log into the jump host.
  • 172.18.50.100 is the SSH server that we will be connecting to. This is the jump server
  • -p 20622 is optional. It tells ssh to establish the tunnel on the remote port (destination port) 20622. Typically, this will be 22, however there could be some security controls in place that do not allow SSH on the common port. This option is not always necessary.
  • -L 127.0.0.1:4001:192.168.20.10:22 tells ssh to listen locally on port 4001. Any connections to 127.0.0.1 on port 4001 will be forwarded to 192.168.20.10 on port 22 through the SSH tunnel. You could also use localhost in place of 127.0.0.1, assuming you haven’t modified that entry in your /etc/hosts file.
  • -N is optional, tells ssh to not execute a remote command. This is useful for just forwarding ports.
  • -f is optional, requests ssh to go to background just before command execution. This is useful if ssh is going to ask for passwords or passphrases, but the user wants it in the background.

Observe the connections

If you are curious about what your system is doing from a network perspective, open a separate terminal and run the following command before you create the SSH tunnel:

watch 'netstat -abn | grep 4001'

This will run the command every 2 seconds and print the output to the screen. Observe the output:

Every 2.0s: netstat -abn | grep 4001              My-Cool-Macbook..local: Mon Sep 16 16:18:48 2019

tcp4       0      0  127.0.0.1.4001         *.*                    LISTEN               0          0

Connect to host using local port

Now, connect to 192.168.20.10 using the following command:

username@127.0.0.1 -p 4001

Notice the netstat output:

Every 2.0s: netstat -abn | grep 4001                     My-Cool-Macbook.local: Mon Sep 16 16:08:03 2019

tcp4       0      0  127.0.0.1.4001         127.0.0.1.64356        ESTABLISHED       3877          0
tcp4       0      0  127.0.0.1.64356        127.0.0.1.4001         ESTABLISHED       3683          0
tcp4       0      0  127.0.0.1.4001         *.*                    LISTEN               0          0

The SSH client is forarding traffic over local port 4001 to randomly selected open local port 64356 which is then sent over the SSH tunnel, which eventually lands at 192.168.20.10 port 22. This allows for multiple connections to be forwarded through port 4001. So if you were to make another connection, you will see an additional netstat entry with another local port generated for the second connection.

Terminal the SSH Tunnel

To terminate the ssh tunnel, run ps aux | grep ssh, search for the correct tunnel and PID, and then run kill 12345 replacing 12345 with the PID on your machine.

SOCKS Proxy

Another cool feature enabled on many systems is SOCKS. This allows you to proxy application traffic and send it to a jump host. This is useful for browsing to a website that is normally not directly accessible. You can use this technique to access internal websites that remain only accessible behind a jump host. Multiple SOCKS proxies can be created, meaning multiple endpoints can be configured to proxy your local machine’s traffic.

Let’s say we wanted to use the same jump host in the example above and send our web traffic to the jumphost to access a website hosted behind it on an internal network. In this example, we will set up a local SOCKS proxy and SSH tunnel. The SOCKS proxy will send traffic via the SSH tunnel to the jump host. We will be using Firefox, however many other browsers such as Google Chrome support SOCKS5.

At the command-line, run the following: ssh -D 4000 -C -N -q -f username@192.168.20.10 -p 64356

Let’s break this down:

  • -D 4000 is used for dynamic application-level port forwarding. This will open a SOCKS proxy on port 4000. Ensure that this port is not already being used on your local machine.
  • C optional, is used to compress data in the tunnel to conserve bandwidth.
  • q optional, this is for quiet mode.
  • N optional, directs ssh to not execute remote commands.
  • username is the username for the jump host.
  • 192.168.20.10 is the jump host IP address.
  • -p 20622 is optional, it tells ssh to establish the tunnel on the remote port 20622. Typically, this will be 22, however there could be some security controls in place that do not allow SSH on the common port. This option is not always necessary.
  • f is optional, requests ssh to go to background just before command execution. This is useful if ssh is going to ask for passwords or passphrases, but the user wants it in the background.

Configure your browser to use SOCKS

Now head over to your browser:

  1. Click the hamburger menu button in the top right-hand corner
  2. Click Preferences.
  3. Click on the General tab if not already there and scroll all the way down to Network Settings > Settings...
  4. Select the Manual Proxy configuration radio button and enter 127.0.0.1 with port 4000, and ensure SOCKS v5 is selected.

Firefox SOCKS Proxy Configuration ScreenshotFirefox SOCKS Proxy Configuration Screenshot

Once you click OK you should now be proxying your web connections and should be able to access internal websites. To kill the SSH tunnel, thus the SOCKS proxy connection, search for the PID and kill the process.

This is not an exaustive list of cool things you can do with SSH tunnels, but are some of the most common methods. We hope this article was informative to you, Happy coding!

-Kenny B



ntc img
ntc img

Contact Us to Learn More

Share details about yourself & someone from our team will reach out to you ASAP!