Category Archives: One Liners

dhcpd lease information sorted by date and time

Posted on by
Reply

When looking on a pxe boot install server, you can see what the newest clients were to boot. If you don’t have KVM access on new servers to be installed, just look at the newest lease info, and make an educated guess about which new one to login to the auto-installer environment (preseed) via ssh.

Here’s a snippet from the leases file:

lease 10.101.40.85 {
  starts 3 2013/05/15 19:54:36;
  ends 3 2013/05/15 20:54:36;
  cltt 3 2013/05/15 19:54:36;
  binding state active;
  next binding state free;
  hardware ethernet 00:30:48:5c:cf:34;
  uid "\001\0000H\\\3174";
}

And after some parsing:

# cat /var/lib/dhcp/dhcpd.leases | grep -e lease -e hardware -e start -e end | grep -v format | grep -v written | sed -e '/start/s/\// /g' -e 's/;//g' -e '/starts/s/:/ /g' | paste - - - - | awk '{print $2" "$18" "$6" "$7" "$8" "$9" "$10" "$11" "$14" "$15}' | sort -k 3,3n -k 4,4n -k 5,5n -k 6,6n -k 7,7n -k 8,8n | awk '{print $1" "$2" "$3"/"$4"/"$5" "$6":"$7":"$8" "$9" "$10}' | column -t


10.101.40.127  00:1e:68:9a:e5:ac  2013/04/26  22:02:58  2013/04/26  23:02:58
10.101.40.129  00:1e:68:9a:e5:ac  2013/04/26  23:10:01  2013/04/27  00:10:01
10.101.40.122  00:1e:68:9a:e5:ac  2013/04/26  23:27:57  2013/04/26  23:30:42
10.101.40.118  00:1e:68:9a:ee:69  2013/05/14  16:21:28  2013/05/14  17:21:28
10.101.40.85   00:30:48:5c:cf:34  2013/05/14  16:54:43  2013/05/14  17:54:43
10.101.40.118  00:1e:68:9a:ee:69  2013/05/14  17:14:04  2013/05/14  18:14:04
10.101.40.85   00:30:48:5c:cf:34  2013/05/14  17:24:43  2013/05/14  18:24:43
10.101.40.85   00:30:48:5c:cf:34  2013/05/14  17:54:42  2013/05/14  18:54:42
#

rsync migration with manifest of transfer

Posted on by
Reply

I once had a migration project to move 40TB of data that needed to be moved from source to destination NFS volumes. Naturally, I went with rsync. This was basically the command for the initial transfers:

rsync -a --out-format="transfer:%t,%b,%f" --itemize-changes /mnt/srcvol /mnt/destvol >> /log/file.log

Pretty simple right? The logs are a simple csv file that looked like this:

transfer:2013/05/02 10:16:13,35291,mnt/srcvol/archive/foo/bar/barfoo/IMAGES/1256562131100.jpg

The customer asked for daily updates on progress. I said no problem, and this one liner takes care of it:

# grep transfer /log/file.log | awk -F "," '{if ($2!=0) i=i+$2; x++} END {print "total Gbytes: "i/1073741824"\ntotal files: "x}'
total Gbytes: 1153.29
total files: 123686

From the rsync command above, the %t means timestamp (2013/05/02 10:16:13), the %b means bytes transferred (35291), and the %f means the whole file path. By adding up the %b column of output and counting how many times you added it, you get both the total bytes transferred and the total number of files transferred. Directories show up as 0 byte transfers so in awk we don’t count them. Also, I threw in the divide by 1073741824 (1024*1024*1024), which converts bytes to Gebibytes.

I ended up putting it in a shell script and adding options such as, just find transfers for a particular day/hour, better handling for the Gbytes number, rate calculation, and the ability to add logs from multiple data moving servers.

inspect .zip contents from command line

Posted on by
Reply

If you are browsing around in a terminal and want to see the contents of a .zip file, unzip is able to just list the contents but not actually extract them:

unzip -l myarchive.zip

Alternatively, if you want to open a .zip file in a GUI window to browse it’s contents do this:

file-roller myarchive.zip

It will open up Archive Manager to look at the contents.

rsync on different ssh port

Posted on by
1

By default ssh runs on TCP port 22. If you have a ssh configured to listen on a non standard port, you may need a special option to make rsync connect to that port. I was writing a quick backup script for this worpress site, and ran into this issue. In my case I was trying to rsync to a remote server with ssh listening on 4590. You have to give rsync a special ssh option:

# rsync -a --rsh='ssh -p 4590' /srv/www/wp-uploads/ backupsite.com:/backups/wp/wp-uplodas

find recently modified files

Posted on by
Reply

To find the 10 newest items in your home directory you can just use ls.

# cd 
# ls -lt | head
total 1197116
-rw-r--r--  1 fordodone     fordodone        5353 2013-04-23 10:42 file1
-rw-r--r--  1 fordodone     fordodone        2945 2013-04-23 10:21 file2
drwxr-xr-x  2 fordodone     fordodone       12288 2013-04-12 08:53 bin
-rw-r--r--  1 fordodone fordodone         0 2013-03-27 08:45 file3
-rw-------  1 fordodone fordodone     90420 2013-03-23 09:03 file4
-rw-r--r--  1 fordodone     fordodone          83 2013-03-19 10:35 file5
-rw-r--r--  1 fordodone     fordodone        8683 2013-03-15 10:26 file6
-rw-r--r--  1 fordodone     fordodone       28628 2013-03-15 09:15 file7
-rw-r--r--  1 fordodone     fordodone       81303 2013-03-15 09:15 file8

You could even get more aggressive by throwing the recursive flag in. Simple, right? But what if you need to recurse a large number of files and directories on a storage system, say 123,000 directories and 37 million files. I think find might be the way to go.

This will find files modified in the last 24 hours: 

# find . -type f -mtime -1 -ls

find doesn’t really provide granular control of searching for files with a certain modified time. If you just want to find files that have been modified today (i.e. since 12am) we can use the -newer flag. First touch a temporary file with a timestamp to compare to files you want to find. In this case we make a date string of 04240000, or today at 00:00, and touch a file with that timestamp. Then use find to find files that are newer than the timestamp of the file you just touched.


# touch -t `date +%m%d0000` /tmp/compare
# find . -type f -newer /tmp/compare
(long output)
# rm /tmp/compare

print last 3 characters of a string

Posted on by
Reply

If you need to get the last few characters of a string, you can just use awk.  This works great because you don’t have to know how long or short the string is to begin with. It first gets the length of the string. So in this example the length is 9 characters, and the ‘1’ would be in position 9 of the string. Then it subtracts 2 from position 9, to get position 7 (which is the ‘3’), then it gets 3 characters starting from position 7: ‘371’

# echo server371 | awk '{print substr($0,length($0)-2,3)}'
371

# echo anotherserver892 | awk '{print substr($0,length($0)-2,3)}'
892