connect to ssh tunnel on Mac host from inside docker container

In some very edge use cases you may want to temporarily connect to a remote service on a different network from within a docker container. In this example I needed to provide a solution to connect from a docker container on a Mac laptop to a database hosted in a subnet far far away. Here’s a proof of concept.

setup the ssh tunnel from the Mac host:

ssh -L 56789: fordodone@

This starts an ssh tunnel with the TCP port 56789 open on localhost, forwarding through the ssh tunnel to port 3306 on a host with IP address on the remote network

start a container:

docker run --rm -it alpine:latest sh

This runs an interactive shell in a temporary docker container from the alpine:latest image

inside the container add mysql client for testing:

/ # apk add --update --no-cache mysql-client

This fetches a temporary package list and installs mysql-client

create a database connection:

/ # mysql -h docker.for.mac.localhost --port 56789 -udbuser -p

Enter password: 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 675073323
Server version: 5.6.27-log MySQL Community Server (GPL)

Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [(none)]> 
MySQL [(none)]> show databases;
| Database           |
| information_schema |
| awesome_app        |
| innodb             |
| mysql              |
| performance_schema |
5 rows in set (0.08 sec)

MySQL [(none)]> 

The magic here comes from the special docker.for.mac.localhost hostname. The internal Docker for Mac DNS resolver uses this special entry to return the internal IP address used by the Mac host. Tell the mysql client to use the port-forwarded TCP port --port 56789 and the mysql client inside the container connects through the ssh tunnel to the remote database.

use netstat to monitor receive queue Recv-Q

# i=0; while true; do i=$(($i+1)); echo $i ==============================; netstat -natlp | grep ^tcp | sort -nk1 | awk '{ if($2 != 0) {print}}' ; sleep 1;  done;
1 ==============================
2 ==============================
3 ==============================
4 ==============================
5 ==============================
tcp      100      0        ESTABLISHED 99304/sshd: fordodone
6 ==============================
7 ==============================
8 ==============================
9 ==============================
tcp    43520      0          ESTABLISHED 119789/mysqldump
10 ==============================
11 ==============================
12 ==============================
13 ==============================
14 ==============================
15 ==============================
16 ==============================
tcp6       1      0        CLOSE_WAIT  3880/apache2    
17 ==============================
18 ==============================

nagios aggressive cli smtp monitoring

# while true; do date | tr '\n' ' ' && /usr/local/nagios/libexec/check_smtp -H somemailhost -p 25 ; sleep 5s; done;
Fri Aug  7 11:15:35 MST 2015 SMTP OK - 0.108 sec. response time|time=0.108085s;;;0.000000
Fri Aug  7 11:15:41 MST 2015 SMTP OK - 0.111 sec. response time|time=0.111096s;;;0.000000
Fri Aug  7 11:15:46 MST 2015 SMTP OK - 0.110 sec. response time|time=0.110013s;;;0.000000

Vyatta monitor and log NAT translation

Logging to record NAT translations. This might be helpful for finding users using bit torrent (along with tshark), or for watching what IPs are connecting to what external services, and when.

while true; do d=`date +%Y%m%d%H%M`; show nat translations detail | grep -v Pre-NAT | paste - - | sort -nk1 >nats.$d.log; sleep 5m; done;
# tail -1 nats.201408261250.log
a.a.a.a:21845    z.z.z.z:443     b.b.b.b:21845       z.z.z.z:443         tcp: snat: a.a.a.a ==> b.b.b.b  timeout: 42 use: 1 

ftp to dev null to test bandwidth

When testing bandwidth, and troubleshooting bottlenecks, I prefer to use iperf. If you insist on testing bandwidth with FTP, it’s important NOT to use regular files. If you transfer actual files, the transfer could be limited by disk i/o, due to reads and writes. To eliminate this you can FTP from /dev/zero to /dev/null. It sounds super easy, but you have to use FTP in a special way to get it to read and write to special devices.

Here’s a little script. Be sure to replace the destination IP address, username and password with actual values:

# cat
/usr/bin/ftp -n <IP address of machine> <<END
verbose on
user <usernanme> <password>
put "|dd if=/dev/zero bs=32k" /dev/null
Verbose mode on.
331 Password required for fordodone
230 User fordodone logged in
Remote system type is UNIX.
Using binary mode to transfer files.
200 Type set to I
local: |dd if=/dev/zero bs=32k remote: /dev/null
200 PORT command successful
150 Opening BINARY mode data connection for /dev/null
send aborted
waiting for remote to finish abort
129188+0 records in
129187+0 records out
4233199616 bytes (4.2 GB) copied, 145.851 s, 29.0 MB/s
226 Transfer complete
4233142272 bytes sent in 145.82 secs (28350.0 kB/s)
221 Goodbye.

In this case I was getting around 230Mbits per second (over an IPSec tunnel) between my client and the FTP server. Not too bad.

F5 BigIP ssh monitor

I created a pool of load balanced ssh servers. In order to monitor them for availability, I needed to create a custom monitor. This is a very old F5 load balancer:

# uname -r
BIG-IP 4.5.14

It seems like the easiest way to monitor ssh servers would be with ssh. After tinkering with it, I didn’t like the idea. I didn’t like the interactive quality of ssh, and didn’t want to make a custom user just for health checks. I also, didn’t want to put shared keys on the load balancers themselves. These are very custom ssh servers, and trigger filesystem mounting, and all sorts of other auth methods. I don’t actually want to ssh to them, I just want to see if ssh port is open. expect or nc or nmap were not available. I hit tab a few times and viewed the 500 or so commands available. I saw curl and gave that a try.

For our purposes, all we care about is that the port is open and we get a response to a request on that port:

node_ip=`echo $1 | sed 's/::ffff://'`

pidfile="/var/run/`basename $0`.$node_ip..$"
if [ -f $pidfile ]
   kill -9 `cat $pidfile` > /dev/null 2>&1
echo "$$" > $pidfile

curl http://${node_ip}:22 --connect-timeout 5 > /dev/null 2>&1

if [ $status -eq 0 ]
    echo "UP"

rm -f $pidfile

If the server is down completely, curl returns 7. If sshd has crashed the port is closed and curl again returns 7. If the port is open, curl exits 0.

scan range of IP addresses

nmap is an amazing utility. With all of it’s flags and options it really gives you the power to know what is out in a network.

I used this to do a simple ping scan of my home network:

# nmap -nsP
Starting Nmap 5.00 ( ) at 2013-08-22 09:06 MST
Host is up (0.0051s latency).
MAC Address: 00:18:39:4E:82:60 (Cisco-Linksys)
Host is up.
Nmap done: 256 IP addresses (2 hosts up) scanned in 5.51 seconds

Hosts that allow icmp echo requests will show up. I can see my router and my workstation, but no other hosts (no other pingable hosts at least). I’m looking for a RPi that I recently plugged into the network. It doesn’t look like it’s up. Time to drag out the spare monitor and keyboard…