ftp to dev null to test bandwidth

When testing bandwidth, and troubleshooting bottlenecks, I prefer to use iperf. If you insist on testing bandwidth with FTP, it’s important NOT to use regular files. If you transfer actual files, the transfer could be limited by disk i/o, due to reads and writes. To eliminate this you can FTP from /dev/zero to /dev/null. It sounds super easy, but you have to use FTP in a special way to get it to read and write to special devices.

Here’s a little script. Be sure to replace the destination IP address, username and password with actual values:

# cat ftp_dev_null.sh
#!/bin/bash
/usr/bin/ftp -n <IP address of machine> <<END
verbose on
user <usernanme> <password>
bin
put "|dd if=/dev/zero bs=32k" /dev/null
bye
END
# ftp_dev_null.sh
Verbose mode on.
331 Password required for fordodone
230 User fordodone logged in
Remote system type is UNIX.
Using binary mode to transfer files.
200 Type set to I
local: |dd if=/dev/zero bs=32k remote: /dev/null
200 PORT command successful
150 Opening BINARY mode data connection for /dev/null
^C
send aborted
waiting for remote to finish abort
129188+0 records in
129187+0 records out
4233199616 bytes (4.2 GB) copied, 145.851 s, 29.0 MB/s
226 Transfer complete
4233142272 bytes sent in 145.82 secs (28350.0 kB/s)
221 Goodbye.
#

In this case I was getting around 230Mbits per second (over an IPSec tunnel) between my client and the FTP server. Not too bad.

Vyatta create and update IP based ban lists from Spamhaus

You can use Spamhaus, or a number of other lists in a Vyatta firewall configuration. In this case we create a network group called ‘blocked’ from the Spamhaus blacklists. Then this network group can be used in firewalls to drop traffic. Use cron to update the list every day, or once a week.

#!/bin/bash
# FILE: /usr/local/sbin/updateBanList.sh
# AUTHOR: ForDoDone fordodone@fordodone.com
# DATE: 2013-10-01
# NOTES: Script to update IP ban list.  Run from cron, and integrate into firewall
# 

# variables
VERBOSE=0
DROPURL='http://www.spamhaus.org/drop/drop.txt'
EDROPURL='http://www.spamhaus.org/drop/edrop.txt'

# simple logger function
logger(){
  if [ "$VERBOSE" == "1" ]
  then
    echo "$@"
  fi
}

# set verbose flag if given
if [ "$1" == "-v" ]
then
VERBOSE=1;
fi

# create or truncate tmp file
>/tmp/block

# get drop file
wget -q $DROPURL -O - | grep ^[0-9] | sed -e 's/;.*//' >> /tmp/block
if [ $? -ne 0 ]
then
  logger "error getting drop file"
  logger "exiting..."
exit
fi

# get edrop file
wget -q "$EDROPURL" -O - | grep ^[0-9] | sed -e 's/;.*//' >> /tmp/block
if [ $? -ne 0 ]
then
  logger "error getting edrop file"
  logger "exiting..."
exit
fi
logger "received `wc -l /tmp/block | awk '{print $1}'` networks to block..."

logger "starting vyatta cmd wrapper"
/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper begin

# remove existing list, in case a network has been removed"
logger "deleting existing blocked network group"
/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper delete firewall group network-group blocked

# add each network to the block list
logger "building new blocked network group"
logger "this might take a while..."
for i in `cat /tmp/block`;
do
  /opt/vyatta/sbin/vyatta-cfg-cmd-wrapper set firewall group network-group blocked network $i
done;

# now commit the changes
logger "committing changes"
/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper commit

logger "ending vyatta cmd wrapper"
/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper end

# clean up
rm -rf /tmp/block >/dev/null 2>&1

get list of set environmental variables

Use printenv

# printenv
TERM=screen
SHELL=/bin/bash
SSH_CLIENT=10.171.0.141 60941 22
SSH_TTY=/dev/pts/0
USER=root
MAIL=/var/mail/root
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/etc/pam.d
LANG=en_US.UTF-8
SHLVL=1
HOME=/root
LOGNAME=root
SSH_CONNECTION=10.171.0.141 60941 10.122.0.33 22
_=/usr/bin/printenv
OLDPWD=/etc
#

read configuration files without comments and spaces

Long configuration files have a helpful/annoying amount of comments and explanations about directives. Put this function into your .bashrc to read config files without any comments or white space:

rconf(){ cat "$@" | egrep -v '(#|^\s#*$|^$)' ; }

Then re-load .bashrc file and next time you need to read a config file just:

# rconf <config file>

get last occurrence of string in file

Here’s just a few ways to skin this cat:

# tac /etc/fstab | grep -m 1 fs144
10.239.11.144:/vol/vol22         /mnt/fs144/vol22        nfs     auto,rw,soft,mountvers=3 0 0

# grep fs144 /etc/fstab | tail -1
10.239.11.144:/vol/vol22         /mnt/fs144/vol22        nfs     auto,rw,soft,mountvers=3 0 0

# awk '{ if ( /fs144/ ) j=$0;} END {print j}' /etc/fstab
10.239.11.144:/vol/vol22         /mnt/fs144/vol22        nfs     auto,rw,soft,mountvers=3 0 0

get absolute path of running script

Using $0 or basename in a script works well if you are calling it using it’s absolute path or if it’s in $PATH. The test script shows how basename and $0 are displayed. Calling the test script from within /usr/local/sbin, does not give any absolute path.

# pwd
/usr/local/sbin
#
# cat test.sh
#!/bin/bash
echo $0
echo `basename $0`
#
#
#
# test.sh 
/usr/local/sbin/test.sh
test.sh
#
# ./test.sh
./test.sh
test.sh
#
#echo 'echo $(readlink -f $0)' >> test.sh
#
# ./test.sh 
./test.sh
test.sh
/usr/local/sbin/test.sh

readlink gives us the full path to the script, no matter how it’s called.

F5 BigIP ssh monitor

I created a pool of load balanced ssh servers. In order to monitor them for availability, I needed to create a custom monitor. This is a very old F5 load balancer:

# uname -r
BIG-IP 4.5.14

It seems like the easiest way to monitor ssh servers would be with ssh. After tinkering with it, I didn’t like the idea. I didn’t like the interactive quality of ssh, and didn’t want to make a custom user just for health checks. I also, didn’t want to put shared keys on the load balancers themselves. These are very custom ssh servers, and trigger filesystem mounting, and all sorts of other auth methods. I don’t actually want to ssh to them, I just want to see if ssh port is open. expect or nc or nmap were not available. I hit tab a few times and viewed the 500 or so commands available. I saw curl and gave that a try.

For our purposes, all we care about is that the port is open and we get a response to a request on that port:

#!/bin/sh
node_ip=`echo $1 | sed 's/::ffff://'`

pidfile="/var/run/`basename $0`.$node_ip..$2.pid"
if [ -f $pidfile ]
then
   kill -9 `cat $pidfile` > /dev/null 2>&1
fi
echo "$$" > $pidfile

curl http://${node_ip}:22 --connect-timeout 5 > /dev/null 2>&1

status=$?
if [ $status -eq 0 ]
then
    echo "UP"
fi

rm -f $pidfile

If the server is down completely, curl returns 7. If sshd has crashed the port is closed and curl again returns 7. If the port is open, curl exits 0.