percentile apache server request response times

I needed a hack to quickly find the 95th percentile of apache request response times. For example I needed to be able to say that “95% of our apache requests are served in X milliseconds or less.” In the apache2 config the LogFormat directive had %D (the time taken to serve the request, in microseconds) as the last field. Meaning the last field of each log line would be the time it took to serve the request. This would make it easy to pull out with $NF in awk

# PCT=.95; NR=`cat access.log | wc -l `; cat /var/log/apache2/access.log | awk '{print $NF}' | sort -rn | tail -n+$(echo "$NR-($NR*$PCT)" |bc | cut -d. -f1) |head -1
938247

In this case 95% of the apache requests were served in 938 milliseconds or less (WTF?!). Then run on an aggregated group of logs, or change the date/time range to just run for logs on a particular day, or for multiple time periods.

Note: I couldn’t get scale to work here in bc for some reason.

share:

mercurial hg clone turn off host key checking for bitbucket.org

If you clone a repository during an automated code deploy (for example in AWS CodeDeploy or Atlassian Bamboo) then you probably need to turn off host key checking for the clone of your repository. This prevents hg (or git) from raising a user prompt about the authenticity of the host key.

$ echo -e "Host bitbucket.org\nStrictHostKeyChecking no\n" >> ~/.ssh/config
share:

count new connections per minute to a tcp port

I was running a custom FTP service out of inetd, when it intermittently stopped responding to requests (Connection refused.) In the logs inetd was logging:

Mar 23 06:54:36 fordodone inetd[1510]: ftp/tcp server failing (looping), service terminated for 10 min

After some searching I discovered this error happens when there are too many connections to an inetd service per minute. How many is too many? From the man page for inetd.conf we can see that the default is 256. So the aggregate number of opening connections was over 256 per minute and inetd stops responding for 10 minutes to protect itself and the system from running out of resources. I increased the default to 512 (debian system) and restarted inetd for now.

# echo 'OPTIONS="-R"' >> /etc/default/openbsd-inetd  && service openbsd-inetd restart

How close am I to the 256 default? How often would it happen? Is there a pattern? Could this be legit traffic or a DoS attack? I wrote this one liner to see new or opening connections to the ftp control port per minute. You could change it a little for other services.

# tcpdump -lni eth1 "tcp[13] & 2 != 0" and dst port 21 2>/dev/null | while read i ; do j=`echo $i | cut -d : -f -2`; if [ "$k" == "$j" ]; then l=$(($l+1)); else echo "$k -- $l"; k=$j; l=1; fi; done;

Start with tcpdump on the interface you want to listen(-i eth1), no need to resolve hostnames(-n), or buffer output(-l), and look at the TCP flags byte (tcp[13]) (13th byte) for the SYN bit (2) to see if it is set, and only if the destination port is 21. Pipe it to a while loop and read in the lines as they come. Note the hour:minute, and count packets for that minute. If the minute changes, output the last minutes count, and reset the counter.

You have to ignore the first 2 lines. The first one means nothing, and the second one is missing the portion of the minute that was before you started it. The real results start to roll in on iteration 3.

 --
17:26 -- 6
17:27 -- 21
17:28 -- 20
17:29 -- 34
17:30 -- 38
17:31 -- 27
17:32 -- 37
17:33 -- 22
17:34 -- 23
17:35 -- 33
17:36 -- 29
17:37 -- 23
17:38 -- 28
17:39 -- 26
17:40 -- 73
17:41 -- 99
17:42 -- 132
17:43 -- 110
17:44 -- 130
17:45 -- 112
17:46 -- 109
17:47 -- 104
17:48 -- 182
17:49 -- 155
17:50 -- 145
17:51 -- 110
17:52 -- 154
17:53 -- 147
17:54 -- 86
17:55 -- 39
17:56 -- 39
17:57 -- 30
17:58 -- 30
17:59 -- 38
share:

make many directories with Bash sequences

Sometimes it’s necessary to do things with sequences in Bash. If you want to create a bunch of directories that will be mount points for NFS servers you could do this:


# mkdir /mnt/fs42
# mkdir /mnt/fs42/vol0
# mkdir /mnt/fs42/vol1
# mkdir /mnt/fs42/vol2
# mkdir /mnt/fs42/vol3

But, even the best experts at the “up arrow key” wouldn’t want to do this for 50 file servers. A nested for loop would work, but it’s not necessarily the easiest way to go. The command seq makes a sequence of integers bounded by the numbers specified.

ok:

for i in `seq 1 50`; do for j in `seq 0 3`; do mkdir -p /mnt/fs$i/vol$j; done; done;

Using Bash sequences, you can tell the shell to interpret this {1..50} as a list of integers between 1 and 50. This also works with letters like {a..t}.

# echo {a..t}
a b c d e f g h i j k l m n o p q r s t
# echo {0..50}
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50

way better:


# mkdir -p /mnt/fs{1..50}/vol{0..3}

of course if you have non-sequential lists, you may have to specify each element like this:

{1,3,4,5,6,9}

share: