use netstat to monitor receive queue Recv-Q

# i=0; while true; do i=$(($i+1)); echo $i ==============================; netstat -natlp | grep ^tcp | sort -nk1 | awk '{ if($2 != 0) {print}}' ; sleep 1;  done;
1 ==============================
2 ==============================
3 ==============================
4 ==============================
5 ==============================
tcp      100      0        ESTABLISHED 99304/sshd: fordodone
6 ==============================
7 ==============================
8 ==============================
9 ==============================
tcp    43520      0          ESTABLISHED 119789/mysqldump
10 ==============================
11 ==============================
12 ==============================
13 ==============================
14 ==============================
15 ==============================
16 ==============================
tcp6       1      0        CLOSE_WAIT  3880/apache2    
17 ==============================
18 ==============================

nagios aggressive cli smtp monitoring

# while true; do date | tr '\n' ' ' && /usr/local/nagios/libexec/check_smtp -H somemailhost -p 25 ; sleep 5s; done;
Fri Aug  7 11:15:35 MST 2015 SMTP OK - 0.108 sec. response time|time=0.108085s;;;0.000000
Fri Aug  7 11:15:41 MST 2015 SMTP OK - 0.111 sec. response time|time=0.111096s;;;0.000000
Fri Aug  7 11:15:46 MST 2015 SMTP OK - 0.110 sec. response time|time=0.110013s;;;0.000000

Vyatta monitor and log NAT translation

Logging to record NAT translations. This might be helpful for finding users using bit torrent (along with tshark), or for watching what IPs are connecting to what external services, and when.

while true; do d=`date +%Y%m%d%H%M`; show nat translations detail | grep -v Pre-NAT | paste - - | sort -nk1 >nats.$d.log; sleep 5m; done;
# tail -1 nats.201408261250.log
a.a.a.a:21845    z.z.z.z:443     b.b.b.b:21845       z.z.z.z:443         tcp: snat: a.a.a.a ==> b.b.b.b  timeout: 42 use: 1 

ftp to dev null to test bandwidth

When testing bandwidth, and troubleshooting bottlenecks, I prefer to use iperf. If you insist on testing bandwidth with FTP, it’s important NOT to use regular files. If you transfer actual files, the transfer could be limited by disk i/o, due to reads and writes. To eliminate this you can FTP from /dev/zero to /dev/null. It sounds super easy, but you have to use FTP in a special way to get it to read and write to special devices.

Here’s a little script. Be sure to replace the destination IP address, username and password with actual values:

# cat
/usr/bin/ftp -n <IP address of machine> <<END
verbose on
user <usernanme> <password>
put "|dd if=/dev/zero bs=32k" /dev/null
Verbose mode on.
331 Password required for fordodone
230 User fordodone logged in
Remote system type is UNIX.
Using binary mode to transfer files.
200 Type set to I
local: |dd if=/dev/zero bs=32k remote: /dev/null
200 PORT command successful
150 Opening BINARY mode data connection for /dev/null
send aborted
waiting for remote to finish abort
129188+0 records in
129187+0 records out
4233199616 bytes (4.2 GB) copied, 145.851 s, 29.0 MB/s
226 Transfer complete
4233142272 bytes sent in 145.82 secs (28350.0 kB/s)
221 Goodbye.

In this case I was getting around 230Mbits per second (over an IPSec tunnel) between my client and the FTP server. Not too bad.


F5 BigIP ssh monitor

I created a pool of load balanced ssh servers. In order to monitor them for availability, I needed to create a custom monitor. This is a very old F5 load balancer:

# uname -r
BIG-IP 4.5.14

It seems like the easiest way to monitor ssh servers would be with ssh. After tinkering with it, I didn’t like the idea. I didn’t like the interactive quality of ssh, and didn’t want to make a custom user just for health checks. I also, didn’t want to put shared keys on the load balancers themselves. These are very custom ssh servers, and trigger filesystem mounting, and all sorts of other auth methods. I don’t actually want to ssh to them, I just want to see if ssh port is open. expect or nc or nmap were not available. I hit tab a few times and viewed the 500 or so commands available. I saw curl and gave that a try.

For our purposes, all we care about is that the port is open and we get a response to a request on that port:

node_ip=`echo $1 | sed 's/::ffff://'`

pidfile="/var/run/`basename $0`.$node_ip..$"
if [ -f $pidfile ]
   kill -9 `cat $pidfile` > /dev/null 2>&1
echo "$$" > $pidfile

curl http://${node_ip}:22 --connect-timeout 5 > /dev/null 2>&1

if [ $status -eq 0 ]
    echo "UP"

rm -f $pidfile

If the server is down completely, curl returns 7. If sshd has crashed the port is closed and curl again returns 7. If the port is open, curl exits 0.


scan range of IP addresses

nmap is an amazing utility. With all of it’s flags and options it really gives you the power to know what is out in a network.

I used this to do a simple ping scan of my home network:

# nmap -nsP
Starting Nmap 5.00 ( ) at 2013-08-22 09:06 MST
Host is up (0.0051s latency).
MAC Address: 00:18:39:4E:82:60 (Cisco-Linksys)
Host is up.
Nmap done: 256 IP addresses (2 hosts up) scanned in 5.51 seconds

Hosts that allow icmp echo requests will show up. I can see my router and my workstation, but no other hosts (no other pingable hosts at least). I’m looking for a RPi that I recently plugged into the network. It doesn’t look like it’s up. Time to drag out the spare monitor and keyboard…


monitor host for slow ping times

When there is intermittent network latency to a host, it’s important to monitor a it for a pattern. Using ping can help narrow down what is causing the latency. VMWare load, bandwidth limitations, employee work patterns, backups, and many other sources could be the cause of the latency.

while true; do j=`ping <slowhost> -i1 -c1 2>&1 | grep icmp_req | awk '{print $7}' | cut -d = -f2 | cut -d . -f1`; if [ $j -gt 30 ]; then date | tr '\n' ' ';  echo $j; fi; sleep 1s; done;

This does a ping every second, and if it’s over a threshold (30ms in this case) it is considered unacceptable and logged with date.