Author Archives: ForDoDone

drop messages from mail queue

Posted on by
Reply

During massive outages (which thankfully happen rarely), I like to keep my Nagios monitoring machines online and working. This is because I like to have a view of the servers with remaining problems, or processes that didn’t come back online correctly. However, I stop our MTA (postfix) on those servers, because I don’t want to receive texts and emails complaining about all the servers that are still down. Once the problem is resumed, I could just startup postfix, but lets take a look at the mailqueue: 

# mailq 2>&1 | tail -1 | cut -d " " -f 5-
428 Requests.

Hmm… seems a bit high. If we start postfix again, guess how many text messages are going to wind up on my phone? Let’s drop all of the messages in the queue:

# postsuper -d ALL
postsuper: Deleted: 428 messages

Now we can start postfix without excessive messages being sent.

Alternatively, if the main MX relays go down for a period of time, you will see the mailqueue fill up with undelivered mail. After you bring the MXs back online, the mail may be sent to them immediately. Your MTA probably has an increasing retry interval, which could lead to a one hour delay or longer. Do this to attempt to relay all the mail in the queue:

# postqueue -f

It will try to reconnect immediately to the MX relay, and deliver all mail if it can.

get total host memory on VMWare ESXi

Posted on by
Reply

It is my opinion that Graphical User Interfaces (GUIs) can make administration both easier, and more difficult. I highly prefer commandline, and many appliances and non desktop systems have a plethora of advanced options when configuring from the command line. I always void the warranty and get under the hood. All of my ESXi servers have SSH and Console enabled. If you have someone in the datacenter replacing suspect memory on an ESXi system, you don’t want to have them re-rack it and cable it up, just so you can log in through vSphere, just to see if the host sees all the memory. Just do it from commandline. Oh yeah, and free is not a recognized command on the ESXi shell.

# free
-ash: free: not found
#
# vim-cmd hostsvc/hosthardware | grep memorySize | sed -e 's/,//' -e 's/^ *//' 
memorySize = 34182787072
#

This was done on ESXi 4.1.0 Build 345043

rsync migration with manifest of transfer

Posted on by
Reply

I once had a migration project to move 40TB of data that needed to be moved from source to destination NFS volumes. Naturally, I went with rsync. This was basically the command for the initial transfers:

rsync -a --out-format="transfer:%t,%b,%f" --itemize-changes /mnt/srcvol /mnt/destvol >> /log/file.log

Pretty simple right? The logs are a simple csv file that looked like this:

transfer:2013/05/02 10:16:13,35291,mnt/srcvol/archive/foo/bar/barfoo/IMAGES/1256562131100.jpg

The customer asked for daily updates on progress. I said no problem, and this one liner takes care of it:

# grep transfer /log/file.log | awk -F "," '{if ($2!=0) i=i+$2; x++} END {print "total Gbytes: "i/1073741824"\ntotal files: "x}'
total Gbytes: 1153.29
total files: 123686

From the rsync command above, the %t means timestamp (2013/05/02 10:16:13), the %b means bytes transferred (35291), and the %f means the whole file path. By adding up the %b column of output and counting how many times you added it, you get both the total bytes transferred and the total number of files transferred. Directories show up as 0 byte transfers so in awk we don’t count them. Also, I threw in the divide by 1073741824 (1024*1024*1024), which converts bytes to Gebibytes.

I ended up putting it in a shell script and adding options such as, just find transfers for a particular day/hour, better handling for the Gbytes number, rate calculation, and the ability to add logs from multiple data moving servers.

inspect .zip contents from command line

Posted on by
Reply

If you are browsing around in a terminal and want to see the contents of a .zip file, unzip is able to just list the contents but not actually extract them:

unzip -l myarchive.zip

Alternatively, if you want to open a .zip file in a GUI window to browse it’s contents do this:

file-roller myarchive.zip

It will open up Archive Manager to look at the contents.

mount iso file in linux

Posted on by
Reply

I had to copy the contents of a .iso file. It’s easy to mount it and see what’s in the iso.


# mkdir /mnt/iso
# mount -o loop VMware-VMvisor-Installer-5.1.0-799733.x86_64.iso /mnt/iso
# cd /mnt/disk
# ls
a.b00         ata_pata.v05  boot.cfg      ima_qla4.v00  isolinux.bin  misc_dri.v00  net_e100.v01  net_r816.v00  ohci_usb.v00  sata_sat.v02  scsi_bnx.v00  scsi_meg.v01  scsi_qla.v01  upgrade                  xlibs.v00
ata_pata.v00  ata_pata.v06  chardevs.b00  imgdb.tgz     isolinux.cfg  net_be2n.v00  net_enic.v00  net_r816.v01  safeboot.c32  sata_sat.v03  scsi_fni.v00  scsi_meg.v02  scsi_rst.v00  user.b00                 xorg.v00
ata_pata.v01  ata_pata.v07  efi           imgpayld.tgz  k.b00         net_bnx2.v00  net_forc.v00  net_s2io.v00  sata_ahc.v00  sata_sat.v04  scsi_hps.v00  scsi_mpt.v00  s.v00         useropts.gz
ata_pata.v02  b.b00         efiboot.img   ipmi_ipm.v00  mboot.c32     net_bnx2.v01  net_igb.v00   net_sky2.v00  sata_ata.v00  scsi_aac.v00  scsi_ips.v00  scsi_mpt.v01  tboot.b00     vmware-esx-base-osl.txt
ata_pata.v03  block_cc.v00  ehci_ehc.v00  ipmi_ipm.v01  menu.c32      net_cnic.v00  net_ixgb.v00  net_tg3.v00   sata_sat.v00  scsi_adp.v00  scsi_lpf.v00  scsi_mpt.v02  tools.t00     vmware-esx-base-readme
ata_pata.v04  boot.cat      esx_dvfi.v00  ipmi_ipm.v02  misc_cni.v00  net_e100.v00  net_nx_n.v00  net_vmxn.v00  sata_sat.v01  scsi_aic.v00  scsi_meg.v00  scsi_qla.v00  uhci_usb.v00  weaselin.t00
#

Now the contents of /mnt/disk appear as if you had burned the .iso and put it in the CD drive.

make many directories with Bash sequences

Posted on by
Reply

Sometimes it’s necessary to do things with sequences in Bash. If you want to create a bunch of directories that will be mount points for NFS servers you could do this:


# mkdir /mnt/fs42
# mkdir /mnt/fs42/vol0
# mkdir /mnt/fs42/vol1
# mkdir /mnt/fs42/vol2
# mkdir /mnt/fs42/vol3

But, even the best experts at the “up arrow key” wouldn’t want to do this for 50 file servers. A nested for loop would work, but it’s not necessarily the easiest way to go. The command seq makes a sequence of integers bounded by the numbers specified.

ok:

for i in `seq 1 50`; do for j in `seq 0 3`; do mkdir -p /mnt/fs$i/vol$j; done; done;

Using Bash sequences, you can tell the shell to interpret this {1..50} as a list of integers between 1 and 50. This also works with letters like {a..t}.

# echo {a..t}
a b c d e f g h i j k l m n o p q r s t
# echo {0..50}
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50

way better:


# mkdir -p /mnt/fs{1..50}/vol{0..3}

of course if you have non-sequential lists, you may have to specify each element like this:

{1,3,4,5,6,9}

rsync on different ssh port

Posted on by
1

By default ssh runs on TCP port 22. If you have a ssh configured to listen on a non standard port, you may need a special option to make rsync connect to that port. I was writing a quick backup script for this worpress site, and ran into this issue. In my case I was trying to rsync to a remote server with ssh listening on 4590. You have to give rsync a special ssh option:

# rsync -a --rsh='ssh -p 4590' /srv/www/wp-uploads/ backupsite.com:/backups/wp/wp-uplodas