Vyatta create and update IP based ban lists from Spamhaus

You can use Spamhaus, or a number of other lists in a Vyatta firewall configuration. In this case we create a network group called ‘blocked’ from the Spamhaus blacklists. Then this network group can be used in firewalls to drop traffic. Use cron to update the list every day, or once a week.

#!/bin/bash
# FILE: /usr/local/sbin/updateBanList.sh
# AUTHOR: ForDoDone fordodone@fordodone.com
# DATE: 2013-10-01
# NOTES: Script to update IP ban list.  Run from cron, and integrate into firewall
# 

# variables
VERBOSE=0
DROPURL='http://www.spamhaus.org/drop/drop.txt'
EDROPURL='http://www.spamhaus.org/drop/edrop.txt'

# simple logger function
logger(){
  if [ "$VERBOSE" == "1" ]
  then
    echo "$@"
  fi
}

# set verbose flag if given
if [ "$1" == "-v" ]
then
VERBOSE=1;
fi

# create or truncate tmp file
>/tmp/block

# get drop file
wget -q $DROPURL -O - | grep ^[0-9] | sed -e 's/;.*//' >> /tmp/block
if [ $? -ne 0 ]
then
  logger "error getting drop file"
  logger "exiting..."
exit
fi

# get edrop file
wget -q "$EDROPURL" -O - | grep ^[0-9] | sed -e 's/;.*//' >> /tmp/block
if [ $? -ne 0 ]
then
  logger "error getting edrop file"
  logger "exiting..."
exit
fi
logger "received `wc -l /tmp/block | awk '{print $1}'` networks to block..."

logger "starting vyatta cmd wrapper"
/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper begin

# remove existing list, in case a network has been removed"
logger "deleting existing blocked network group"
/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper delete firewall group network-group blocked

# add each network to the block list
logger "building new blocked network group"
logger "this might take a while..."
for i in `cat /tmp/block`;
do
  /opt/vyatta/sbin/vyatta-cfg-cmd-wrapper set firewall group network-group blocked network $i
done;

# now commit the changes
logger "committing changes"
/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper commit

logger "ending vyatta cmd wrapper"
/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper end

# clean up
rm -rf /tmp/block >/dev/null 2>&1

awk print range of lines

print lines 31 through 34 inclusive:

# awk 'NR==31,NR==34' share/vyatta-cfg/templates/policy/prefix-list/node.tag/rule/node.def
        if [ $VAR(./le/@) -ne 32 ] && [ -n "$VAR(./ge/@)" ] && [ $VAR(./le/@) -le $VAR(./ge/@) ]; then 
          echo "le must be greater than or equal to ge"; 
          exit 1 ; 
        fi ; 
#

sort nested directories by last modified using find

Using ls -lt to sort a file listing by last modified time is simple and easy. If you have a large directory tree with tens of thousands of directories, using find with some massaging might be the way to go. In this example there is a directory with many directories in a tree like this:

./1
./1/1
./1/1/1
./1/1/2
./1/2
./1/2/3
./2
./2/3
./2/3/4
./2/3/5
./2/3/7
./2/3/8

we are interested in the 3rd level directory and getting a list of which ones were most recently modified

# find . -mindepth 3 -maxdepth 3 -ls | awk '$10 !~ /^20[01]/' | sed -e 's/:/ /' | sort -k8,8M -nk9,9n -nk10 -nk11 | awk '{print $12" "$8" "$9" "$10":"$11}'| column -t | tail -10

We start by finding only 3rd level directories with extended listings (there are no files at this level, so -type d is unnecessary). Then use awk to only print directories that have been modified this year (i.e. anything with a year like 200* or 201* instead of a hour:minute in column 10). Replace the time colon HH:MM so that we can sort by minute after we sort by hour. Then rearrange the columns, add back the hour:minute colon, run it through column to get nice columns, then get the last 10 results.

./586/1586/1311586  Sep  16  16:11
./980/6980/2326980  Sep  16  16:18
./616/3616/513616   Sep  16  16:20
./133/9133/2119133  Sep  16  16:21
./422/6422/2106422  Sep  16  16:24
./566/6566/2326566  Sep  16  16:46
./672/672/2310672   Sep  16  16:51
./680/680/2290680   Sep  16  17:42
./573/5573/2325573  Sep  16  17:47
./106/1106/2321106  Sep  16  17:49

connect to git host via ssh on non-standard port

Sometimes people run sshd on a non-standard port. It takes time to scan an IP block, and scanning each host for all 65,535 ports makes it take even longer. Most scanning scripts and utilities target common known open ports, like telnet, smb, and ssh. For this reason someone might opt to run sshd on a port other than 22. This is a problem if you are using git over ssh to connect to one of these repositories.

Add the follwing to your ssh config:

cat >>/home/<yourusername>/.ssh/config << EOF
Host <git server IP address>
  Port <obscure sshd port number>
  IdentityFile /home/<yourusername>/.ssh/id_git
EOF

get list of set environmental variables

Use printenv

# printenv
TERM=screen
SHELL=/bin/bash
SSH_CLIENT=10.171.0.141 60941 22
SSH_TTY=/dev/pts/0
USER=root
MAIL=/var/mail/root
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/etc/pam.d
LANG=en_US.UTF-8
SHLVL=1
HOME=/root
LOGNAME=root
SSH_CONNECTION=10.171.0.141 60941 10.122.0.33 22
_=/usr/bin/printenv
OLDPWD=/etc
#

quick memcache testing and troubleshooting

Here’s a quick way to test your new memcached setup. At first it’s a bit confusing to understand that memcache has no knowledge of what keys it stores, just whether or not it has a particular key. Unlike relational databases, you can’t query memcache and get all of the values it has. You have to know the key that you want to fetch before you ask memcache about it. Let’s ask it the value for the key foo:

telnet memcacheserver 11211
Trying 10.131.215.181...
Connected to 10.131.215.181
Escape character is '^]'.
get foo
END

It returns nothing, so it doesn’t have any value for that key. The key foo is unset. Let’s set it:

set foo 0 0 3
bar
STORED

When you set a key like this, follow the syntax “set <keyname> <flag> <ttl> <storebytes>”. In this case our key is foo, we have no important flags (0), there is no expiration of the key value pair (0), and the data we are about to store is 3 bytes (aka 3 characters). Let’s fetch it now:

get foo
VALUE foo 0 3
bar
END

It returns the value of key foo as bar. Now delete it:

delete foo
DELETED

Now another, this time the key is “foobar”, and the data is a 12 byte string “barbarbarbar”:

set foobar 0 0 12
barbarbarbar
STORED
get foobar
VALUE foobar 0 12
barbarbarbar
END
delete foobar
DELETED
^]
telnet> close
Connection closed.