Tag Archives: grep

Posted on by
1

Using find to act on files is very useful, but if the files that are found need different actions based on their filetype, it gets a bit trickier. For example there are some log files foo.log but after 10 days they get compressed to foo.log.gz. So you are finding regular text files, as well as gzipped text files. Extend your find with an -exec and a bash shell to determine what file extension it is, and to run the appropriate grep or zgrep based on that. Then run it through awk or whatever else to parse out what you need.

# find . -type f -name 'foo.log*' -exec bash -c 'if [[ $0 =~ .log$ ]]; then grep foobar $0; elif [[ $0 =~ .log.gz$ ]]; then zgrep foobar $0; fi' {} \; | awk '{if(/typea/)a++; if(/typeb/)b++; tot++} END {print "typea: "a" - "a*100/tot"%"; print "typeb: "b" - "b*100/tot"%"; print "typec: "tot-(a+b)" - "(tot-(a+b))*100/tot"%"; print "total: "tot;}'
typea: 5301 - 67.4771%
typeb: 2539 - 32.3192%
typec: 16 - 0.203666%
total: 7856

get last occurrence of string in file

Posted on by
Reply

Here’s just a few ways to skin this cat:

# tac /etc/fstab | grep -m 1 fs144
10.239.11.144:/vol/vol22         /mnt/fs144/vol22        nfs     auto,rw,soft,mountvers=3 0 0

# grep fs144 /etc/fstab | tail -1
10.239.11.144:/vol/vol22         /mnt/fs144/vol22        nfs     auto,rw,soft,mountvers=3 0 0

# awk '{ if ( /fs144/ ) j=$0;} END {print j}' /etc/fstab
10.239.11.144:/vol/vol22         /mnt/fs144/vol22        nfs     auto,rw,soft,mountvers=3 0 0

show progress for dd

Posted on by
Reply

By default dd is silent. It just copies whatever blocks you want from in to out. In order to see progress, send it a USR1 signal using kill.

Start a useless dd:

# dd if=/dev/zero of=/dev/null

In another terminal find the pid:

# ps aux | grep dd | grep -v grep
root      7784 90.5  0.0   2884   560 pts/9    R+   10:01   0:06 dd if /dev/zero of /dev/null
#
# kill -USR1 7784

The original window will now show this:

# dd if=/dev/zero of=/dev/null
14501614+0 records in
14501614+0 records out
7424826368 bytes (7.4 GB) copied, 16.2149 seconds, 458 MB/s

Then you can ctrl+c it to get the final output:

# dd if=/dev/zero of=/dev/null
14501614+0 records in
14501614+0 records out
7424826368 bytes (7.4 GB) copied, 16.2149 seconds, 458 MB/s
16888077+0 records in
16888076+0 records out
8646694912 bytes (8.6 GB) copied, 19.3507 seconds, 447 MB/s

This one liner will start your dd, then monitor it and output progress every 20 seconds. Once the dd is finished it will stop and give your shell back.

dd if=/dev/zero of=/dev/null & pid=$! && sleep 20s && while true; do i=`ps aux | awk '{print $2}' | grep ^$pid$`; if [ "${i:-a}" !=  "$pid" ]; then break; fi; kill -USR1 $pid; sleep 20s; done;