AWS find unused instance reservations

AWS Trusted Advisor has some good metrics on Cost Optimization when it comes to looking at your infrastructure portfolio and potential savings, however, it doesn’t offer a good way to see underutilized reservations. By using awscli you can check your reservations against what is actually running in order to find under provisioned reservations:

#!/bin/bash
# super quick hack to see difference between reserved and actual usage for RI

IFS="
"

first="true"
for az in us-east-1b us-east-1d
do
  if [ "$first" == "true" ]
  then
    echo "AvailabilityZone InstanceType Reserved Actual Underprovisioned"
    first="false"
  fi
  for itype in `aws ec2 describe-reserved-instances --filters Name=state,Values=active Name=availability-zone,Values=$az | grep InstanceType | sed -e 's/ //g' | cut -d'"' -f 4 | sort -u`
  do
    rcount=$(aws ec2 describe-reserved-instances --filters Name=state,Values=active Name=availability-zone,Values=$az Name=instance-type,Values=$itype | egrep 'InstanceCount' | awk '{print $NF}' | awk 'BEGIN{i=0}{i=i+$1}END{print i}')
    icount=$(aws ec2 describe-instances --filter Name=availability-zone,Values=$az Name=instance-type,Values=$itype | grep InstanceId | wc -l)

    echo "$az $itype $rcount $icount" | awk '{ if($4<$3){alert="YES"} print $1" "$2" "$3" "$4" "alert}'

  done;

done | column -t

Output:

$ compare_reserved_instances_vs_actual.sh
AvailabilityZone  InstanceType  Reserved  Actual  Underprovisioned
us-east-1b        m3.2xlarge    2         0       YES
us-east-1b        m3.medium     2         8
us-east-1b        m3.xlarge     11        4       YES
us-east-1b        m4.xlarge     2         46
us-east-1d        m3.medium     2         8
us-east-1d        m3.xlarge     6         0       YES
us-east-1d        m4.xlarge     1         17

perl validate email address

There’s a great perl module Email::Valid to check if a string is a well-formed email address. It doesn’t actually check to see if the email exists in the destination domain, but in it’s simplest form we can just make sure the string follows the proper email address format specification.

Install the module:

# apt-get install -y libemail-valid-perl

quick and dirty:

#!/usr/bin/perl
use strict;
use Email::Valid;

# do it once with an email address as an argument
#if ( ! Email::Valid->address("$ARGV[0]")){print "$ARGV[0]\n"};

my $file="to_be_tested2_emails.txt";

# do it with a list of emails
  open(FH,"$file") or die("Can\'t read $file.\n");
  foreach my $line (<FH>) {
    chomp $line;
    if ( ! Email::Valid->address("$line")){print "$line\n"};

  };
  close(FH);

TODO: think of loading into array; removing malformed addresses; having option to run once vs. run list; test the rfc822 validation

vyatta create firewall network group from geo ip table

The global Geo IP table is huge, and contains ranges of IP addresses that don’t necessarily fit into CIDR networks. Converting the IP ranges in the table into networks results in around 129,000 separate networks. Trying to define every network scope in the global table for inclusion in a firewall is well outside the realm of possibility. In the case of a few small countries, their ranges do actually fit into network groups. This is just a fun little exercise not to be used in production of course.

#!/usr/bin/perl
# FILE: netGroupCountry.pl
# AUTHOR: ForDoDone
# DATE: 2013-12-02
# NOTES: create a firewall network group for a specific country using a geoip table

use strict;
use Net::Netmask;

# download geoip table
# TODO: wget geoip table if doesn't exist, or if 1st wed of month

# set country here
# TODO: read from ARGV
# use Estonia for example it only has 236 IP ranges in table
my $country = 'EE';

# set geoip database CSV
my $geoIPDB='/var/lib/geoip/GeoIPCountryWhois.csv';

# open geoip table
open FILE, "$geoIPDB" or die $!;

# start the vyatta config wrapper
system("/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper begin");

# clear the old network group
system("/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper delete firewall group network-group $country");

# commit here optional
#system("/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper commit");

# setup commit counter for chunked commits
#my $i=0;
#my $chunk=100;

# read each line of the geoip table (~82k lines)
while(<FILE>){
  chomp;

  my @line = split(/"/,$_);

  # if country matches the one we want
  # convert the ip range to CIDR(s)
  # add the network to the network group
  if ("$line[9]" eq "$country" ){

    my @blocks = range2cidrlist("$line[1]","$line[3]");

        foreach(@blocks){
          my $setcmd = "set firewall group network-group $country network $_";
          system("/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper $setcmd");

          #print "set firewall group network-group $country network $_\n";

          # chunked commit
          #if ( $i % $chunk ){
          #  # do nothing
          #} else {
          #  print "committing after 100 entries... total: $i\n";
          #  system("/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper commit");
          #}
          #$i++;

        }

  }

}

# close geoip table
close FILE;

# run vyatta config wrapper to commit and end
system("/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper commit");
system("/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper end");
vyatta@vyatta:/usr/local/sbin$ time ./netGroupCountry.pl
Nothing to delete (the specified node does not exist)

real	1m3.515s
user	0m22.660s
sys	0m27.650s
vyatta@vyatta:/usr/local/sbin$

This definitely won’t work for larger networks, and who knows how it actually affects performance, I’ve never actually tried using the network group in the firewall.

Vyatta create and update IP based ban lists from Spamhaus

You can use Spamhaus, or a number of other lists in a Vyatta firewall configuration. In this case we create a network group called ‘blocked’ from the Spamhaus blacklists. Then this network group can be used in firewalls to drop traffic. Use cron to update the list every day, or once a week.

#!/bin/bash
# FILE: /usr/local/sbin/updateBanList.sh
# AUTHOR: ForDoDone fordodone@fordodone.com
# DATE: 2013-10-01
# NOTES: Script to update IP ban list.  Run from cron, and integrate into firewall
# 

# variables
VERBOSE=0
DROPURL='http://www.spamhaus.org/drop/drop.txt'
EDROPURL='http://www.spamhaus.org/drop/edrop.txt'

# simple logger function
logger(){
  if [ "$VERBOSE" == "1" ]
  then
    echo "$@"
  fi
}

# set verbose flag if given
if [ "$1" == "-v" ]
then
VERBOSE=1;
fi

# create or truncate tmp file
>/tmp/block

# get drop file
wget -q $DROPURL -O - | grep ^[0-9] | sed -e 's/;.*//' >> /tmp/block
if [ $? -ne 0 ]
then
  logger "error getting drop file"
  logger "exiting..."
exit
fi

# get edrop file
wget -q "$EDROPURL" -O - | grep ^[0-9] | sed -e 's/;.*//' >> /tmp/block
if [ $? -ne 0 ]
then
  logger "error getting edrop file"
  logger "exiting..."
exit
fi
logger "received `wc -l /tmp/block | awk '{print $1}'` networks to block..."

logger "starting vyatta cmd wrapper"
/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper begin

# remove existing list, in case a network has been removed"
logger "deleting existing blocked network group"
/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper delete firewall group network-group blocked

# add each network to the block list
logger "building new blocked network group"
logger "this might take a while..."
for i in `cat /tmp/block`;
do
  /opt/vyatta/sbin/vyatta-cfg-cmd-wrapper set firewall group network-group blocked network $i
done;

# now commit the changes
logger "committing changes"
/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper commit

logger "ending vyatta cmd wrapper"
/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper end

# clean up
rm -rf /tmp/block >/dev/null 2>&1

F5 BigIP ssh monitor

I created a pool of load balanced ssh servers. In order to monitor them for availability, I needed to create a custom monitor. This is a very old F5 load balancer:

# uname -r
BIG-IP 4.5.14

It seems like the easiest way to monitor ssh servers would be with ssh. After tinkering with it, I didn’t like the idea. I didn’t like the interactive quality of ssh, and didn’t want to make a custom user just for health checks. I also, didn’t want to put shared keys on the load balancers themselves. These are very custom ssh servers, and trigger filesystem mounting, and all sorts of other auth methods. I don’t actually want to ssh to them, I just want to see if ssh port is open. expect or nc or nmap were not available. I hit tab a few times and viewed the 500 or so commands available. I saw curl and gave that a try.

For our purposes, all we care about is that the port is open and we get a response to a request on that port:

#!/bin/sh
node_ip=`echo $1 | sed 's/::ffff://'`

pidfile="/var/run/`basename $0`.$node_ip..$2.pid"
if [ -f $pidfile ]
then
   kill -9 `cat $pidfile` > /dev/null 2>&1
fi
echo "$$" > $pidfile

curl http://${node_ip}:22 --connect-timeout 5 > /dev/null 2>&1

status=$?
if [ $status -eq 0 ]
then
    echo "UP"
fi

rm -f $pidfile

If the server is down completely, curl returns 7. If sshd has crashed the port is closed and curl again returns 7. If the port is open, curl exits 0.

get directory mtime in unix time

In scripts when you need to compare last modification date of directories, you can get the date using stat in a unix timestamp or seconds from the Epoch:

# stat -c '%Z' /usr/local/sbin
1373673278

Using date you can get the same format like this:

# date +%s
1373673486

You could use this in a script to do something if a directory is older or newer than some amount of time:

#!/bin/bash
# FILE: sync_usr_local_sbin.sh
# AUTHOR: ForDoDone <fordodone at email.com>
# DATE: 2013-07-12
# NOTES: syncs /usr/local/sbin to hostxyz if it's been modified in the last 5 minutes
#

now=`date +%s`

uls_lastmtime=`stat -c '%Z' /usr/local/sbin`

uls_diff=$(echo $now - $uls_lastmtime |bc)

if [ $uls_diff -lt 300 ]
then
  rsync -a /usr/local/sbin/ hostxyz:/usr/local/sbin
fi

Of course rsync has a bunch of options to check whether it needs to do an update of files, this is just an example.

SCP files from one host to another

Everyone knows how to copy files around using SCP, but it can be a pain if you have to enter passwords for every copy. If you have an administration host with shared ssh keys to every other host, you can just use a quick little one liner to drag files from hostA, through the admin box, over to hostB:

adminbox # ssh hostA "tar cf - /usr/local/sbin/myscript.sh 2>/dev/null" | ssh hostB "cd / && tar xvf - 1>/dev/null"

Using tar, the file is output to STDOUT and piped over ssh, then read from STDIN. It copies /usr/local/sbin/myscript.sh from hostA to hostB. Because the admin box has ssh keys to both hostA and hostB, the process is automatic and does not require password authentication. This means you can use this method in scripts for batch copies, etc. Also, you won’t have to create a temporary copy on the admin host.

Drop it into a simple shell script and it will be even easier:

#!/bin/bash
# FILE: file_dragger.sh
# AUTHOR: fordodone <fordodone at email.com>
# DATE: 2013/07/11
# NOTES: drags a file from one host to another
#

if [ $# -ne 3 ]
then
echo ""
echo "usage: </full/path/to/file> <src> <dst>"
echo ""
exit
fi

ssh $2 "tar cf - $1 2>/dev/null" | ssh $3 "cd / && tar xvf - 1>/dev/null"

To use it for the original copy example do this:

# file_dragger.sh /usr/local/sbin/myscript.sh hostA hostB
#