make Makefile target for help or usage options

Using make and Makefiles with a docker based application development strategy are a great way to track shortcuts and allow team members to easily run common docker or application tasks without having to remember the syntax specifics. Without a “default” target make will attempt to run the first target (the default goal). This may be desirable in some cases, but I find it useful to have make just print out a usage, and require the operator to specify the exact target they need.


#Makefile 
DC=docker-compose
DE=docker-compose exec app

.PHONY: help
help: 
  @sh -c "echo ; echo 'usage: make <target> ' ; cat Makefile | grep ^[a-z] | sed -e 's/^/            /' -e 's/://' -e 's/help/help (this message)/'; echo"

docker-up:
  $(DC) up -d

docker-down:
  $(DC) stop

docker-rm:
  $(DC) rm -v

docker-ps:
  $(DC) ps

docker-logs:
  $(DC) logs

test:
  $(DE) sh -c "vendor/bin/phpunit"

Now without any arguments make outputs a nice little usage message:


$ make 

usage: make <target> 
            help (this message) 
            docker-up
            docker-down
            docker-rm
            docker-ps
            docker-logs
            test
$

This assumes a bunch of things like you must be calling make from the correct directory, but is a good working proof of concept.

wget use gzip header to received compressed output

This test endpoint returns Content-Type: application/json

Without gzip enabled header:

$ wget -qO test https://testendpoint
$ file test
test: ASCII text, with very long lines, with no line terminators
$ du -b test
7307    test

Setting the gzip enabled header:

$ wget --header="accept-encoding: gzip" -qO test.gz https://testendpoint
$ file test.gz
test.gz: gzip compressed data, from Unix
$ du -b test.gz
1694    test.gz

Telling the server that wget can accept gzip compressed content results in 77% reduction in bytes transferred.

use iozone to benchmark NFS

iozone -t 32 -s1g -r16k -i 0 -i 1

32 threads operating on a 1G file in 16k chunks doing test 0 (write/re-write) and test 1 (read/re-read)

NFSv3 mounted AWS c4.xlarge 500GB gp2 EBS vol formatted EXT4

        Iozone: Performance Test of File I/O
                Version $Revision: 3.397 $
                Compiled for 64 bit mode.
                Build: linux-AMD64 

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
                     Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
                     Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer.
                     Ben England.  

        Run began: Fri May 13 15:46:09 2016

        File size set to 1048576 KB
        Record Size 16 KB
        Command line used: iozone -t 32 -s1g -r16k -i 0 -i 1
        Output is in Kbytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
        Throughput test with 32 processes
        Each process writes a 1048576 Kbyte file in 16 Kbyte records



        Children see throughput for 32 initial writers  =   82456.18 KB/sec
        Parent sees throughput for 32 initial writers   =   73910.24 KB/sec
        Min throughput per process                      =    2501.01 KB/sec
        Max throughput per process                      =    2746.40 KB/sec
        Avg throughput per process                      =    2576.76 KB/sec
        Min xfer                                        =  954736.00 KB

        Children see throughput for 32 rewriters        =   81779.61 KB/sec
        Parent sees throughput for 32 rewriters         =   78548.79 KB/sec
        Min throughput per process                      =    2509.20 KB/sec
        Max throughput per process                      =    2674.01 KB/sec
        Avg throughput per process                      =    2555.61 KB/sec
        Min xfer                                        =  983856.00 KB

        Children see throughput for 32 readers          =   91412.23 KB/sec
        Parent sees throughput for 32 readers           =   90791.54 KB/sec
        Min throughput per process                      =    2761.06 KB/sec
        Max throughput per process                      =    2910.95 KB/sec
        Avg throughput per process                      =    2856.63 KB/sec
        Min xfer                                        =  998720.00 KB

        Children see throughput for 32 re-readers       =   91781.74 KB/sec
        Parent sees throughput for 32 re-readers        =   91620.13 KB/sec
        Min throughput per process                      =    2832.08 KB/sec
        Max throughput per process                      =    2885.37 KB/sec
        Avg throughput per process                      =    2868.18 KB/sec
        Min xfer                                        = 1029440.00 KB



iozone test complete.

start screen session with 4-way split screen

There are several terminals that allow splitting the screen to accommodate multiple regions in a single window. When logged into a non-desktop/server environment Linux screen is great for this. It has support for splitting the screen vertically and/or horizontally. You can use ctrl + a + S or ctrl + a + | to split regions horizontally or vertically. Here’s an excerpt from a .screenrc file to split the screen into 4 regions, and start ssh sessions to four separate servers in each of those regions.


split
split -v
focus down
split -v

screen -t bash /bin/bash
screen -t deploy1 /usr/bin/ssh deploy1
screen -t deploy2 /usr/bin/ssh deploy2
screen -t deploy3 /usr/bin/ssh deploy3
screen -t deploy4 /usr/bin/ssh deploy4

focus up
focus left
select 1
focus right
select 2
focus left
focus down
select 3
focus right
select 4

Now I can start the screen session…

# screen -c .screenrc-multiwindow

and automatically get this:

4-way-screen-split

use wget to recursively download files via FTP

A command line ftp client is good for many things. You can turn off prompting, and use mget with wildcard to get many files. The problem is that mget doesn’t create directories locally, so when it tries to recurse into destination directories in order to place incoming files into them, it fails. We can use wget instead to traverse the directory structure, create folders, and download

# wget -r 'ftp://username:password@ftp.example.com'

Note: rsync would be ideal for this, but there are some cases where the source only offers ftp as a connection protocol.

perl validate email address

There’s a great perl module Email::Valid to check if a string is a well-formed email address. It doesn’t actually check to see if the email exists in the destination domain, but in it’s simplest form we can just make sure the string follows the proper email address format specification.

Install the module:

# apt-get install -y libemail-valid-perl

quick and dirty:

#!/usr/bin/perl
use strict;
use Email::Valid;

# do it once with an email address as an argument
#if ( ! Email::Valid->address("$ARGV[0]")){print "$ARGV[0]\n"};

my $file="to_be_tested2_emails.txt";

# do it with a list of emails
  open(FH,"$file") or die("Can\'t read $file.\n");
  foreach my $line (<FH>) {
    chomp $line;
    if ( ! Email::Valid->address("$line")){print "$line\n"};

  };
  close(FH);

TODO: think of loading into array; removing malformed addresses; having option to run once vs. run list; test the rfc822 validation

add auto statements for interface aliases to /etc/network/interfaces

someone added a bunch of iface statements for configuration but forgot the auto part…

# sed -i 's/iface eth0:\([0-9]\{3\}\)/auto eth0:\1\niface eth0:\1/' /etc/network/interfaces
<snip>
auto eth0:196
iface eth0:196 inet static
  address 1.2.3.4
  netmask 255.255.255.0
auto eth0:197
iface eth0:197 inet static
  address 1.2.3.5
  netmask 255.255.255.0
auto eth0:198
iface eth0:198 inet static
  address 1.2.3.6
  netmask 255.255.255.0
</snip>