Category Archives: One Liners

insert characters into string with sed

Posted on by
Reply 

# echo 20150429 | sed -e 's/\(.\{4\}\)\(.\{2\}\)/\1\/\2\//'
2015/04/29

Start with a single flat directory with thousands of log files…

# ls | head -5
db301.20140216.log.gz
db301.20140217.log.gz
db301.20140218.log.gz
db301.20140219.log.gz
db301.20140220.log.gz

Now move timestamped files into sorted directory by day

# for i in `ls`; do j=$(echo $i| cut -d . -f 2  | sed -e 's/\(.\{4\}\)\(.\{2\}\)/\1\/\2\//'); mkdir -p $j && mv $i $j; done;


check your work

# find . -type f | head -5
./2014/02/16/db301.20140216.log.gz
./2014/02/17/db301.20140217.log.gz
./2014/02/18/db301.20140218.log.gz
./2014/02/19/db301.20140219.log.gz
./2014/02/20/db301.20140220.log.gz

forgot to rename files

# for i in `find . -mindepth 3 -type d `; do pushd $i; for j in `ls`; do k=$(echo $j | sed -e 's/\(\.[0-9]\{8\}\)//' ); mv $j $k;done; popd; done;

check your work

# find . -type f | head -5
./2014/02/16/db301.log.gz
./2014/02/17/db301.log.gz
./2014/02/18/db301.log.gz
./2014/02/19/db301.log.gz
./2014/02/20/db301.log.gz

git show commits between tags

Posted on by
Reply

Instead of making our developers use annotated tags, I just use the git log as a reference. This shows all the commits (minus the trivial ones) between a set number of tags back.

# tagsback=2; tagdiff=$(git tag | tail -$(($tagsback+1)) |tr '\n' ' '| awk '{print $1"..."$NF}'); echo -e "COMMITS BETWEEN $tagdiff\n"; git log --pretty=oneline $tagdiff | cut -d " " -f 2- | grep -v ^Merge
COMMITS BETWEEN 2014092401...2014102101

commit message 1
commit message 2
fixed some bug 
Refs #404885
some other message
#

Vyatta monitor and log NAT translation

Posted on by
Reply

Logging to record NAT translations. This might be helpful for finding users using bit torrent (along with tshark), or for watching what IPs are connecting to what external services, and when.

while true; do d=`date +%Y%m%d%H%M`; show nat translations detail | grep -v Pre-NAT | paste - - | sort -nk1 >nats.$d.log; sleep 5m; done;
# tail -1 nats.201408261250.log
a.a.a.a:21845    z.z.z.z:443     b.b.b.b:21845       z.z.z.z:443         tcp: snat: a.a.a.a ==> b.b.b.b  timeout: 42 use: 1

Posted on by
1

Using find to act on files is very useful, but if the files that are found need different actions based on their filetype, it gets a bit trickier. For example there are some log files foo.log but after 10 days they get compressed to foo.log.gz. So you are finding regular text files, as well as gzipped text files. Extend your find with an -exec and a bash shell to determine what file extension it is, and to run the appropriate grep or zgrep based on that. Then run it through awk or whatever else to parse out what you need.

# find . -type f -name 'foo.log*' -exec bash -c 'if [[ $0 =~ .log$ ]]; then grep foobar $0; elif [[ $0 =~ .log.gz$ ]]; then zgrep foobar $0; fi' {} \; | awk '{if(/typea/)a++; if(/typeb/)b++; tot++} END {print "typea: "a" - "a*100/tot"%"; print "typeb: "b" - "b*100/tot"%"; print "typec: "tot-(a+b)" - "(tot-(a+b))*100/tot"%"; print "total: "tot;}'
typea: 5301 - 67.4771%
typeb: 2539 - 32.3192%
typec: 16 - 0.203666%
total: 7856

monitor NetApp SnapMirror transfer speed

Posted on by
Reply

You may want to monitor the speed of a current snapmirror to get an idea of how fast the transfer is going. The speed might change throughout the day due to load, or disk bottleneck, etc. I started with this one-liner:

i=0;j=0; while true; do j=$i; i=`ssh toaster01 "snapmirror status -l volname" | grep Progress | awk '{print $2}'| cut -d \( -f2`; if [ $j -eq 0 ]; then sleep 1m; continue; fi; echo -n "$i-$j = ";echo "scale=2;($i-$j)/(1024*1024)" | bc | tr '\n' ' '; echo "GB/min"; sleep 1m; done;

Which lead to this short script:

#!/bin/bash
# FILE: netapp_sm_monitor.sh
# AUTHOR: For Do Done <fordodone@fordodone.com>
# DATE: 2014/03/26
# NOTES: 
# 

if [ $# -lt 2 ]
then
  echo ""
  echo "usage: netapp_sm_monitor.sh <filer> <srcvol> [-v]"
  echo ""
  exit
fi

i=0;
j=0; 

while true; 
do 
  j=$i; 
  i=`ssh $1 "snapmirror status -l $2" | grep Progress | awk '{print $2}'| cut -d \( -f2`; 
  if [ $j -eq 0 ]; 
    then 
    sleep 1m; 
    continue; 
  fi; 
  if [ "$3" == "-v" ]
  then
    echo -n "$i-$j = ";
  fi
  echo "scale=2;($i-$j)/(1024*1024)" | bc | tr '\n' ' '; echo "GB/min"; 
  sleep 1m; 
done;

use eval to run commands generated by awk

Posted on by
Reply

Here’s one way to generate a set of commands with awk, and then run them in a loop with eval. 

# cat snippet
field1 /mnt/somedir/785/8785/948785 41 /mnt/somedir2/785/8785/948785 1 2
field1 /mnt/somedir/791/8791/948791 2 /mnt/somedir2/791/8791/948791 6 2
field1 /mnt/somedir/924/8924/948924 2 /mnt/somedir2/924/8924/948924 23 2
field1 /mnt/somedir/993/8993/948993 2 /mnt/somedir2/993/8993/948993 19876 2
field1 /mnt/somedir/3/9003/949003 8 /mnt/somedir2/3/9003/949003 273 2
field1 /mnt/somedir/70/9070/949070 341 /mnt/somedir2/70/9070/949070 6 2
field1 /mnt/somedir/517/4517/954517 2 /mnt/somedir2/517/4517/954517 14 2
field1 /mnt/somedir/699/4699/954699 210 /mnt/somedir2/699/4699/954699 1 2
field1 /mnt/somedir/726/4726/954726 1 /mnt/somedir2/726/4726/954726 6 2

Now use awk to get the output you want and generate commands. Use a forloop and eval to run them.

# for i in `awk '{if($3>$5) print "rsync -a --ignore-existing "$2"/ "$4}' left.compare.sorted  `; do echo $i; eval $i; done;
rsync -a --ignore-existing /mnt/somedir/70/9070/949070/ /mnt/somedir2/70/9070/949070
rsync -a --ignore-existing /mnt/somedir/699/4699/954699/ /mnt/somedir2/699/4699/954699
#

find directories owned by root

Posted on by
1

Find the directories owned by root in a certain part of the tree:

# find . -depth -mindepth 1 -maxdepth 3 -type d -ls | awk '$5 ~ /root/ {print}'
  7930    0 drwxr-xr-x  12 root root      115 Oct 11 16:44 ./562
3805069    0 drwxr-xr-x   3 root root       20 Oct 11 16:44 ./562/8562
  7946    0 drwxr-xr-x   5 root root       46 Dec  8 23:52 ./563/6563
  7947    0 drwxr-xr-x   3 root root      21 Oct 21  2008 ./563/6563/456563
3464735    0 drwxr-xr-x   2 root root        6 Sep 26 17:29 ./563/6563/436563
4075144    4 drwxr-xr-x   2 root root     4096 Dec  9 00:39 ./563/6563/2366563

Change all the ownership to www-data:

# find . -depth -mindepth 1 -maxdepth 3 -type d -exec chown www-data: {} \;

You could do this:

# cd .. && chown -R www-data: dirname

But we only suspect the problem at a certain level in the tree, and it would be way slow to recursively chown hundreds of millions of files.