Category Archives: Bash

percentile apache server request response times

Posted on by
2

I needed a hack to quickly find the 95th percentile of apache request response times. For example I needed to be able to say that “95% of our apache requests are served in X milliseconds or less.” In the apache2 config the LogFormat directive had %D (the time taken to serve the request, in microseconds) as the last field. Meaning the last field of each log line would be the time it took to serve the request. This would make it easy to pull out with $NF in awk

# PCT=.95; NR=`cat access.log | wc -l `; cat /var/log/apache2/access.log | awk '{print $NF}' | sort -rn | tail -n+$(echo "$NR-($NR*$PCT)" |bc | cut -d. -f1) |head -1
938247

In this case 95% of the apache requests were served in 938 milliseconds or less (WTF?!). Then run on an aggregated group of logs, or change the date/time range to just run for logs on a particular day, or for multiple time periods.

Note: I couldn’t get scale to work here in bc for some reason.

wget use gzip header to received compressed output

Posted on by
Reply

This test endpoint returns Content-Type: application/json

Without gzip enabled header:

$ wget -qO test https://testendpoint
$ file test
test: ASCII text, with very long lines, with no line terminators
$ du -b test
7307    test

Setting the gzip enabled header:

$ wget --header="accept-encoding: gzip" -qO test.gz https://testendpoint
$ file test.gz
test.gz: gzip compressed data, from Unix
$ du -b test.gz
1694    test.gz

Telling the server that wget can accept gzip compressed content results in 77% reduction in bytes transferred.

list tables in mysql dump

Posted on by
Reply 

$ zgrep -o '^CREATE.*' database_backup.sql.gz

CREATE TABLE `aa_migrations` (
CREATE TABLE `abcdefg_bar` (
CREATE TABLE `abcdefg_foo` (
CREATE TABLE `abcdefg_images` (
CREATE TABLE `abcdefg_table12` (
CREATE TABLE `abcdefg_table13` (
CREATE TABLE `abcdefg_table14` (
CREATE TABLE `abcdefg_table15` (
CREATE TABLE `abcdefg_users` (

You could also just count them:

$ zgrep -o '^CREATE.*' database_backup.sql.gz | wc -l
9

AWS Lambda function to call bash

Posted on by
Reply

Use this node.js snippet to get Lambda execution of a bash command. In this case bash is just doing a curl to a URL. You could also write an entire bash script and call it.

var child_process = require('child_process');

exports.handler = function(event, context) {
  var child = child_process.spawn('/bin/bash', [ '-c', 'curl --silent --output - --max-time 1 https://fordodone.com/jobtrigger || true' ], { stdio: 'inherit' });

  child.on('close', function(code) {
    if(code !== 0) {
      return context.done(new Error("non zero process exit code"));
    }

    context.done(null);
  });
}

use tee to send output to STDOUT and a pipe

Posted on by
Reply

Generally, tee is used to send output from a command to both STDOUT and to a file. You can also use tee to send output to STDOUT and pass the output on to another command like you would normally using a pipe. This uses a process substitution syntax.

~$ echo "some error message" | tee >(logger -t test_error_msg)
some error message
~$ grep test_error_msg /var/log/syslog
Feb 24 11:37:54 foohost test_error_msg: some error message
~$

The error message is output to STDOUT and also sent to the logger command (which writes it to syslog). This can be usefull when you have silent cron jobs (>/dev/null 2>&1) that log errors to syslog, but at the same time if you run the shell script by hand you can see the error output.

convert dmesg timestamp to date time

Posted on by
3

If you have messages in dmesg that log the event time as “seconds since last boot”, it can be difficult to tell when they happened. Here’s an example of one of these messages:

# tail -1 /var/log/dmesg
[8969653.483175] poorcoding.php[14798]: segfault at 7f2efca36ed0 ip 00007f2efca36ed0 sp 00007f2efaf0be98 error 14
#

You could use something like this to parse out the timestamp and convert it to a date:

# date --date=@$((`date +%s --date="\`who -b | awk '{print $3" "$4}'\`"` + `dmesg | tail -1 | awk '{print $1}' | sed -e 's/\[//g' -e 's/\..*//g'`))
Thu Nov  5 01:21:53 PDT 2015
#

If your application logs to kern.log, it uses timestamps as well as seconds since uptime, so the error will already have a timestamp on it

Aug 14 09:57:59 myhostname kernel: [8969653.483177] poorcoding.php[14800]: segfault at 7f2efca36ec8 ip 00007f2efca36ec8 sp 00007f2ef10c4e98 error 14

TODO: make a quick function; convert all the messages not just a tailed or greped one;

bash random number generator using seq and sort

Posted on by
1

Create the sequence between 0 and 9 and then do a random sort and get the first one.

# seq 0 9 | sort -R | head -1

You can count up the instances of each one and see the distribution looks normal to a human.

# for i in `seq 1 100000`; do seq 0 9 | sort -R | head -1 >> /tmp/rando; done;

# cat /tmp/rando | sort -n | uniq -c | sort -nk2
   9896 0
  10140 1
   9928 2
   9975 3
   9929 4
  10129 5
   9951 6
  10007 7
   9882 8
  10163 9

TODO: test with chi-square? the -R flag can’t be truly random?

count character occurrence rates in filenames

Posted on by
Reply

find all the files in a directory. Take out the first dot . put in by find. Remove slashes (can’t be a character in a filename). Use fold -w 1 (–width) the width option limits column output to 1 character, which puts each character on it’s own line. Don’t count spaces (we don’t care about them). Sort the output, count how many occurrences of each character happened. Sort output by least to most occurrences of characters.

find . -type f | sed -e 's/\.//' -e 's/\// /g' | fold -w 1 | grep -v '^ $' | sort | uniq -c | sort -nk1
      1 '
      7 ^
     22 ,
     29 (
     29 )
     40 #
     51 =
     72 ~
    214 @
    312 :
    672 Y
   1141 +
   1217 J
   1497 Z
   2813 G
   3696 U
   3727 H
   5168 O
   5654 N
   5700 X
   5721 K
  10185 R
  10590 W
  11414 F
  12412 A
  13114 E
  13424 C
  13904 z
  15369 Q
  15698 j
  18746 I
  20582 S
  30232 M
  39547 q
  44301 B
  44946 P
  54675 7
  74749 9
  74777 L
  78077 T
  83720 8
  86739 D
  87151 4
  92824 k
  93168 y
  94261 5
  96495 w
 105734 V
 135527 6
 193306 f
 215943 0
 239003 g
 274810 3
 284082 v
 291777 1
 305769 h
 329499 _
 353852 2
 397075 b
 493086 m
 513388 p
 523439 d
 539160 x
 654812 -
 697485 l
 717868 a
 728134 n
 843460 t
 862742 u
 883640 .
1059771 i
1060749 c
1109991 o
1227620 r
1326244 s
1440326 e

make large directory tree

Posted on by
Reply

I needed to create a large directory structure for some testing. I hacked together this quick script that makes a small or large directory tree easily. You define how many branch points there are and how many branches each branch has all the way from the trunk to the leaves.

#!/bin/bash

i=0;
l=0;
levels=2
dirsperlevel=3

rm -rf tree
mkdir tree && cd tree

while [ $i -lt $levels ];
do
  for j in `find . -mindepth $i -maxdepth $i -type d`
  do
    pushd $j > /dev/null 2>&1
    for k in `seq 1 $dirsperlevel`;
    do
      mkdir $k;
    done;
    popd > /dev/null 2>&1
  done;
  i=$(($i+1))
  l=`echo "($dirsperlevel^$i)+$l"|bc`
done;

echo "$l dirs created in ./tree"

using 2 levels, and 3 directories per level we get 12 total directories created like so:

# mktree.sh 
12 dirs created in ./tree
# find tree    
tree
tree/2
tree/2/2
tree/2/3
tree/2/1
tree/3
tree/3/2
tree/3/3
tree/3/1
tree/1
tree/1/2
tree/1/3
tree/1/1

using something like 6 levels and 6 directories per level would give us 55,986 total directories.