Monthly Archives: June 2013

find fastest Ubuntu mirror with netselect

Posted on by
Reply

Using some internal metrics, netselect can tell you what mirror is “best” for you to use for downloading packages, or setting up your own mirror. netselect uses icmp to determine latency, and hop count between you and a mirror. It can take a list of many mirrors, tests them, and reports which one has the best (lowest) metric. If you want rsync or ftp as a preferred transport, you could change it to only look for one of those. Throw in a few verbose flags to get more output.

# netselect -s 20 `wget https://launchpad.net/ubuntu/+archivemirrors -q -O - | grep '>http' |cut -d / -f 3 | tr '\n' ' '`
    3 mirror.tcpdiag.net
   14 149.20.4.71
   17 nz.archive.ubuntu.com
   17 ftp.citylink.co.nz
   17 mirrors.easynews.com
   18 mirrors.nl.eu.kernel.org
   18 ubuntu.securedservers.com
   45 mirrors.cat.pdx.edu
   58 mirror.peer1.net
   67 mirror.pnl.gov
   77 76.73.4.58
   90 ubuntu.mirrors.tds.net
   95 mirror.steadfast.net
  100 ubuntu-archives.mirror.nexicom.net
  102 mirrors.gigenet.com
  105 mirrors.xmission.com
  109 ubuntu.mirror.constant.com
  115 mirror.cs.umn.edu
  117 ubuntu.bhs.mirrors.ovh.net
  120 mirrors.rit.edu

In this case it looks like mirror.tcpdiag.net is the best choice. 

# ping -c 3 mirror.tcpdiag.net
PING mirror.tcpdiag.net (69.160.243.150) 56(84) bytes of data.
64 bytes from ip-69-160-243-150.static.atlanticmetro.net (69.160.243.150): icmp_req=1 ttl=59 time=3.11 ms
64 bytes from ip-69-160-243-150.static.atlanticmetro.net (69.160.243.150): icmp_req=2 ttl=59 time=2.85 ms
64 bytes from ip-69-160-243-150.static.atlanticmetro.net (69.160.243.150): icmp_req=3 ttl=59 time=3.27 ms

--- mirror.tcpdiag.net ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 2.852/3.081/3.275/0.185 ms

3ms is pretty close.

extract Debian package

Posted on by
Reply

To extract a Debian package and view it’s contents first make a temporary directory to work in, move the .deb file to that location, and extract it:

# mkdir /tmp/rssh
# mv rssh_2.3.2-13squeeze3_i386.deb /tmp/rssh
# cd /tmp/rssh
# ar vx rssh_2.3.2-13squeeze3_i386.deb 
# ls -l
total 128
-rw-r--r-- 1 root root  5094 Jun  5 09:13 control.tar.gz
-rw-r--r-- 1 root root 54142 Jun  5 09:13 data.tar.gz
-rw-r--r-- 1 root root     4 Jun  5 09:13 debian-binary
-rw-r--r-- 1 root root 59428 Nov 27  2012 rssh_2.3.2-13squeeze3_i386.deb

As you can see, the contents have been extracted, and we see the typical 3 files, that are needed for a proper Debian package. The debian-binary file contains information about the version of the package. The control.tar.gz file contains configuration scripts, and tells dpkg about how to install and configure the package. data.tar.gz is the one we are interested in. It contains the actual files that will be set in place at time of installation. The contents of data.tar.gz mirror the typical Linux filesystem tree. Make a directory for the contents of data.tar.gz and extract the tarball into it.


# mkdir debroot
# tar -C debroot -xzf data.tar.gz 
# find debroot/ -ls
   950    4 drwxr-xr-x   4 root     root         4096 Nov 22  2012 debroot/
   956    4 drwxr-xr-x   3 root     root         4096 Nov 22  2012 debroot/etc
   960    4 -rw-r--r--   1 root     root         1867 Nov 22  2012 debroot/etc/rssh.conf
   957    4 drwxr-xr-x   3 root     root         4096 Nov 22  2012 debroot/etc/logcheck
   958    4 drwxr-xr-x   2 root     root         4096 Nov 22  2012 debroot/etc/logcheck/ignore.d.server
   959    4 -rw-r--r--   1 root     root          455 Nov 22  2012 debroot/etc/logcheck/ignore.d.server/rssh
   961    4 drwxr-xr-x   5 root     root         4096 Nov 22  2012 debroot/usr
   965    4 drwxr-xr-x   5 root     root         4096 Nov 22  2012 debroot/usr/share
   971    4 drwxr-xr-x   3 root     root         4096 Nov 22  2012 debroot/usr/share/doc
   972    4 drwxr-xr-x   3 root     root         4096 Nov 22  2012 debroot/usr/share/doc/rssh
   977    8 -rw-r--r--   1 root     root         4162 Nov 27  2005 debroot/usr/share/doc/rssh/SECURITY.gz
   975    4 -rw-r--r--   1 root     root         2207 Nov 22  2012 debroot/usr/share/doc/rssh/README.Debian
   981    4 -rw-r--r--   1 root     root           46 Jul 21  2003 debroot/usr/share/doc/rssh/TODO
   976    8 -rw-r--r--   1 root     root         6297 Nov 22  2012 debroot/usr/share/doc/rssh/changelog.Debian.gz
   983    4 -rw-r--r--   1 root     root          680 Nov 22  2012 debroot/usr/share/doc/rssh/NEWS.Debian.gz
   973    4 -rw-r--r--   1 root     root         2575 Nov 22  2012 debroot/usr/share/doc/rssh/copyright
   979    4 drwxr-xr-x   2 root     root         4096 Nov 22  2012 debroot/usr/share/doc/rssh/examples
   980    8 -rwxr-xr-x   1 root     root         6479 Nov 22  2012 debroot/usr/share/doc/rssh/examples/mkchroot.sh
   974    8 -rw-r--r--   1 root     root         4409 Nov 27  2005 debroot/usr/share/doc/rssh/CHROOT.gz
   982    4 -rw-r--r--   1 root     root         1488 Jan  3  2006 debroot/usr/share/doc/rssh/changelog.gz
   978    4 -rw-r--r--   1 root     root         1636 Dec 29  2004 debroot/usr/share/doc/rssh/README
   984    4 drwxr-xr-x   2 root     root         4096 Nov 22  2012 debroot/usr/share/rssh
   985    4 -rwxr-xr-x   1 root     root         1321 Nov 22  2012 debroot/usr/share/rssh/conf_convert
   966    4 drwxr-xr-x   4 root     root         4096 Nov 22  2012 debroot/usr/share/man
   969    4 drwxr-xr-x   2 root     root         4096 Nov 22  2012 debroot/usr/share/man/man1
   970    8 -rw-r--r--   1 root     root         5771 Nov 22  2012 debroot/usr/share/man/man1/rssh.1.gz
   967    4 drwxr-xr-x   2 root     root         4096 Nov 22  2012 debroot/usr/share/man/man5
   968    4 -rw-r--r--   1 root     root         2277 Nov 22  2012 debroot/usr/share/man/man5/rssh.conf.5.gz
   986    4 drwxr-xr-x   2 root     root         4096 Nov 22  2012 debroot/usr/bin
   987   24 -rwxr-xr-x   1 root     root        22288 Nov 22  2012 debroot/usr/bin/rssh
   962    4 drwxr-xr-x   3 root     root         4096 Nov 22  2012 debroot/usr/lib
   963    4 drwxr-xr-x   2 root     root         4096 Nov 22  2012 debroot/usr/lib/rssh
   964   24 -rwxr-xr-x   1 root     root        21184 Nov 22  2012 debroot/usr/lib/rssh/rssh_chroot_helper

Now we can browse, and take a look at the contents of the files.

diff command outputs, not files

Posted on by
Reply

You can easily diff the output of commands instead of files. In this case hexdump prints thousands of lines, but I’m only interested in the difference:

# diff <(hexdump file1.bin) <(hexdump file2.bin)
1,2c1,2
< 0000000 6a49 b610 0000 0000 5733 7261 4465 4243
< 0000010 0000 0000 0001 0000 9006 4e0b 0b28 000f
---
> 0000000 6a49 b616 0000 0000 5733 7261 4465 4243
> 0000010 0000 0000 0001 0000 9006 4e11 0b28 000f

Run the hexdump in subshell using parenthesis, then redirect the output back to diff. I’m only interested in the 2 pieces that are different for each binary file:

# for i in `ls *.bin | sort -nk1.7`; do echo -n "$i: "; hexdump -C $i | grep '33 57 61 72 65 44\|4e 28 0b 0f 00' | awk '{if(NR==1) print $4;if(NR==2) print $12}' | paste - -; done | column -t 2>/dev/null
file0.bin:   1a  15
file1.bin:   19  14
file2.bin:   18  13
file3.bin:   17  12
file4.bin:   16  11
file5.bin:   15  10
file6.bin:   14  0f
file8.bin:   12  0d
file9.bin:   11  0c
file10.bin:  10  0b
file12.bin:  0e  09
file13.bin:  0d  08
file14.bin:  0f  0a
file15.bin:  0b  06
file16.bin:  0a  05
file17.bin:  09  04
file18.bin:  08  03
file19.bin:  07  02
file20.bin:  06  01
file21.bin:  05  00
file22.bin:  0c  07

generate SSL Certificate Signing Request

Posted on by
Reply

First generate a new 2048 bit key:

# openssl genrsa -out fordodone.com.key 2048

I choose not to encrypt the key, because when reloading 240 apache servers, I don’t want to have to enter the passphrase each time. Now we use our key to generate a Certificat Signing Request.

# openssl req -new -key fordodone.com.key -out fordodone.com.csr

To inspect the CSR:

openssl req -in fordodone.com.csr -noout -text

Now you can sign your own cert, or upload/paste the CSR to a 3rd party provider to issue an official SSL Certificate.

power on/off VM guests in ESXi

Posted on by
Reply

This is helpful for running scripts, etc. Get a list of VMs so you can find the Vmid.

# vim-cmd vmsvc/getallvms
Vmid     Name                     File                      Guest OS       Version   Annotation
16     guest1      [guest1] guest1/guest1.vmx         debian6_64Guest   vmx-09              
32     guest2      [guest2] guest2/guest2.vmx         debian6_64Guest   vmx-09              
48     guest3      [guest3] guest3/guest3.vmx         debian6_64Guest   vmx-09              
64     guest4      [guest4] guest4/guest4.vmx         debian6_64Guest   vmx-09              
80     guest5      [guest5] guest5/guest5.vmx         debian6_64Guest   vmx-09

See if the machine is on or off:

# vim-cmd vmsvc/power.getstate 16
Retrieved runtime info
Powered on

Shut it down:

# vim-cmd vmsvc/power.shutdown 16

Then you can turn it back on:

# vim-cmd vmsvc/power.on 16

download currently installed Debian package without reinstalling

Posted on by
Reply

Sometimes it’s necessary to download a Debian package from a repository without actually intsalling or reinstalling it. 

# apt-get -d --reinstall -o=dir::cache=/tmp install rssh
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Suggested packages:
  cvs rdist subversion makejail
The following NEW packages will be installed:
  rssh
0 upgraded, 1 newly installed, 0 to remove and 102 not upgraded.
Need to get 59.4 kB of archives.
After this operation, 225 kB of additional disk space will be used.
Get:1 http://ftp.us.debian.org/debian/ squeeze/main rssh i386 2.3.2-13squeeze3 [59.4 kB]
Fetched 59.4 kB in 0s (178 kB/s)
Download complete and in download only mode
#

This downloaded the rssh .deb package to /tmp/archives without reinstalling over the existing version.

# ls -l /tmp/archives
total 64
-rw-r----- 1 root root     0 Jun  5 09:05 lock
drwxr-xr-x 2 root root  4096 Jun  5 09:05 partial
-rw-r--r-- 1 root root 59428 Nov 27  2012 rssh_2.3.2-13squeeze3_i386.deb

MySQL group by modulus

Posted on by
Reply

I have a series of scripts that acts on a subset of entries in a large (2M rows) MySQL table. There are 10 instances in the application, and each instance works on id‘s ending in 0-9 from a single table. The ending digit could also be considered the modulus. Instance 0 works on id‘s ending in 0, instance 1 works on id‘s ending in 1, etc. 

mysql> select id from table limit 10;
+---------+
| id      |
+---------+
| 1099138 | 
| 1811048 | 
|  794297 | 
|  985812 | 
|  424120 | 
|  353809 | 
|  478882 | 
|  506101 | 
| 2184449 | 
| 1189511 | 
+---------+

To see what progress the application is making or if it is stuck at I use a query grouping the id‘s remaining to be worked on by modulus:

mysql> select substr(id, -1) as modulus,count(*) from table where field='criteria' group by modulus with rollup;
+---------+----------+
| modulus | count(*) |
+---------+----------+
| 0       |        9 | 
| 1       |       14 | 
| 2       |       13 | 
| 3       |     1985 | 
| 4       |       10 | 
| 5       |       16 | 
| 6       |       16 | 
| 7       |       11 | 
| 8       |       10 | 
| 9       |       20 | 
| NULL    |     2104 | 
+---------+----------+

The substr(id,-1) takes the id and returns the last digit (-1) as modulus. Then it’s grouped by modulus, and using rollup is summed at the end. As you can see the application still has 2104 id‘s to work on, and something is different about instance 3. Either there is an uneven distribution of id‘s ending in 3 (probably not the case, as id is primary key auto increment), or instance 3 in the application is borked.