Tag Archives: wget

wget use gzip header to received compressed output

Posted on by
Reply

This test endpoint returns Content-Type: application/json

Without gzip enabled header:

$ wget -qO test https://testendpoint
$ file test
test: ASCII text, with very long lines, with no line terminators
$ du -b test
7307    test

Setting the gzip enabled header:

$ wget --header="accept-encoding: gzip" -qO test.gz https://testendpoint
$ file test.gz
test.gz: gzip compressed data, from Unix
$ du -b test.gz
1694    test.gz

Telling the server that wget can accept gzip compressed content results in 77% reduction in bytes transferred.

use wget to recursively download files via FTP

Posted on by
Reply

A command line ftp client is good for many things. You can turn off prompting, and use mget with wildcard to get many files. The problem is that mget doesn’t create directories locally, so when it tries to recurse into destination directories in order to place incoming files into them, it fails. We can use wget instead to traverse the directory structure, create folders, and download 

# wget -r 'ftp://username:password@ftp.example.com'

Note: rsync would be ideal for this, but there are some cases where the source only offers ftp as a connection protocol.

docker get list of tags in repository

Posted on by
1

The native docker command has an excellent way to search the docker hub repository for an image. Just use docker search <search string> to look in their registry.

# docker search debian
NAME                          DESCRIPTION                                     STARS     OFFICIAL   AUTOMATED
ubuntu                        Ubuntu is a Debian-based Linux operating s...   2338      [OK]       
debian                        Debian is a Linux distribution that's comp...   763       [OK]       
google/debian                                                                 47                   [OK]
neurodebian                   NeuroDebian provides neuroscience research...   12        [OK]       
jesselang/debian-vagrant      Stock Debian Images made Vagrant-friendly ...   4                    [OK]
eboraas/debian                Debian base images, for all currently-avai...   3                    [OK]
armbuild/debian               ARMHF port of debian                            3                    [OK]
mschuerig/debian-subsonic     Subsonic 5.1 on Debian/wheezy.                  3                    [OK]
fike/debian-postgresql        PostgreSQL 9.4 until 9.0 version running D...   2                    [OK]
maxexcloo/debian              Docker base image built on Debian with Sup...   1                    [OK]
kalabox/debian                                                                1                    [OK]
takeshi81/debian-wheezy-php   Debian wheezy based PHP repo.                   1                    [OK]
webhippie/debian              Docker images for debian                        1                    [OK]
eeacms/debian                 Docker image for Debian to be used with EE...   1                    [OK]
reinblau/debian               Debian with usefully default packages for ...   1                    [OK]
mariorez/debian               Debian Containers for PHP Projects              0                    [OK]
opennsm/debian                Lightly modified Debian images for OpenNSM      0                    [OK]
konstruktoid/debian           Debian base image                               0                    [OK]
visono/debian                 Docker base image of debian 7 with tools i...   0                    [OK]
nimmis/debian                 This is different version of Debian with a...   0                    [OK]
pl31/debian                   Basic debian image                              0                    [OK]
idcu/debian                   mini debian os                                  0                    [OK]
sassmann/debian-chromium      Chromium browser based on debian                0                    [OK]
sassmann/debian-firefox       Firefox browser based on debian                 0                    [OK]
cloudrunnerio/debian                                                          0                    [OK]

We can see the official debian repository right at the top. Unfortunately there’s no way to see what tags and images are available for us to pull down and deploy. However, there is a way to query the registry for all the tags in a repository, returned in JSON format. You can use a higher level programming language to get the list and parse the JSON for you. Or you can just use a simple one-liner:

# wget -q https://registry.hub.docker.com/v1/repositories/debian/tags -O -  | sed -e 's/[][]//g' -e 's/"//g' -e 's/ //g' | tr '}' '\n'  | awk -F: '{print $3}'
latest
6
6.0
6.0.10
6.0.8
6.0.9
7
7.3
7.4
7.5
7.6
7.7
7.8
7.9
8
8.0
8.1
8.2
experimental
jessie
jessie-backports
oldstable
oldstable-backports
rc-buggy
sid
squeeze
stable
stable-backports
stretch
testing
unstable
wheezy
wheezy-backports

Wrap that in a little bash script and you have an easy way to list the tags of a repository. Since a tag is just a pointer to a image commit multiple tags can point to the same image. Get fancy:

# wget -q https://registry.hub.docker.com/v1/repositories/debian/tags -O -  | sed -e 's/[][]//g' -e 's/"//g' -e 's/ //g' | tr '}' '\n' | sed -e 's/^,//' | sort -t: -k2 | awk -F[:,] 'BEGIN {i="image";j="tags"}{if(i!=$2){print i" : "j; i=$2;j=$4}else{j=$4" | "j} }END{print i" : "j}'
image : tags
06af7ad6 : 7.5
19de96c1 : wheezy | 7.9 | 7
1aa59f81 : experimental
20096d5a : rc-buggy
315baabd : stable
37cbf6c3 : testing
47921512 : 7.7
4a5e6db8 : 8.1
4fbc238a : oldstable-backports
52cb7765 : wheezy-backports
84bd6e50 : unstable
88dc7f13 : jessie-backports
8c00acfb : latest | jessie | 8.2 | 8
91238ddc : stretch
b2477d24 : stable-backports
b5fe16f2 : 7.3
bbe78c1a : 7.8
bd4b66c4 : oldstable
c952ddeb : squeeze | 6.0.10 | 6.0 | 6
d56191e1 : 6.0.8
df2a0347 : 8.0
e565fbbc : 7.4
e7d52d7d : sid
feb75584 : 7.6
fee2ea4e : 6.0.9

wget monitor website download speed

Posted on by
Reply 
# while true; do date | tr '\n' '-' | sed -e 's/-/ --- /'; wget http://testsite.com/fancy.pdf -O /dev/null 2>&1 | grep saved | awk -F"[()]" '{print $2}'; sleep 1s; done;
Thu Oct 30 15:18:26 PDT 2014 --- 1.25 MB/s
Thu Oct 30 15:18:28 PDT 2014 --- 1.20 MB/s
Thu Oct 30 15:18:29 PDT 2014 --- 958.95 KB/s
Thu Oct 30 15:18:31 PDT 2014 --- 1.36 MB/s
Thu Oct 30 15:18:32 PDT 2014 --- 873.98 KB/s
Thu Oct 30 15:18:33 PDT 2014 --- 1.38 MB/s
Thu Oct 30 15:18:35 PDT 2014 --- 261.90 KB/s
Thu Oct 30 15:18:37 PDT 2014 --- 1.38 MB/s
Thu Oct 30 15:18:38 PDT 2014 --- 360.14 KB/s
Thu Oct 30 15:18:40 PDT 2014 --- 1.37 MB/s
Thu Oct 30 15:18:42 PDT 2014 --- 427.06 KB/s
Thu Oct 30 15:18:44 PDT 2014 --- 1.37 MB/s
Thu Oct 30 15:18:45 PDT 2014 --- 397.54 KB/s

Vyatta create and update IP based ban lists from Spamhaus

Posted on by
2

You can use Spamhaus, or a number of other lists in a Vyatta firewall configuration. In this case we create a network group called ‘blocked’ from the Spamhaus blacklists. Then this network group can be used in firewalls to drop traffic. Use cron to update the list every day, or once a week.

#!/bin/bash
# FILE: /usr/local/sbin/updateBanList.sh
# AUTHOR: ForDoDone fordodone@fordodone.com
# DATE: 2013-10-01
# NOTES: Script to update IP ban list.  Run from cron, and integrate into firewall
# 

# variables
VERBOSE=0
DROPURL='http://www.spamhaus.org/drop/drop.txt'
EDROPURL='http://www.spamhaus.org/drop/edrop.txt'

# simple logger function
logger(){
  if [ "$VERBOSE" == "1" ]
  then
    echo "$@"
  fi
}

# set verbose flag if given
if [ "$1" == "-v" ]
then
VERBOSE=1;
fi

# create or truncate tmp file
>/tmp/block

# get drop file
wget -q $DROPURL -O - | grep ^[0-9] | sed -e 's/;.*//' >> /tmp/block
if [ $? -ne 0 ]
then
  logger "error getting drop file"
  logger "exiting..."
exit
fi

# get edrop file
wget -q "$EDROPURL" -O - | grep ^[0-9] | sed -e 's/;.*//' >> /tmp/block
if [ $? -ne 0 ]
then
  logger "error getting edrop file"
  logger "exiting..."
exit
fi
logger "received `wc -l /tmp/block | awk '{print $1}'` networks to block..."

logger "starting vyatta cmd wrapper"
/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper begin

# remove existing list, in case a network has been removed"
logger "deleting existing blocked network group"
/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper delete firewall group network-group blocked

# add each network to the block list
logger "building new blocked network group"
logger "this might take a while..."
for i in `cat /tmp/block`;
do
  /opt/vyatta/sbin/vyatta-cfg-cmd-wrapper set firewall group network-group blocked network $i
done;

# now commit the changes
logger "committing changes"
/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper commit

logger "ending vyatta cmd wrapper"
/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper end

# clean up
rm -rf /tmp/block >/dev/null 2>&1

authorized ssh keys for ESXi 5.1

Posted on by
Reply

They go here:

# ls -l /etc/ssh/keys-root/authorized_keys
-rw------T    1 root     root          1572 Jul  9 22:24 /etc/ssh/keys-root/authorized_keys

You can put them on the kickstart server and get them during install using this line in the ks.cfg file:

# install ssh keys
wget http://10.101.40.41/preseed/ESXi/5.1/authorized_keys -O /etc/ssh/keys-root/authorized_keys

find fastest Ubuntu mirror with netselect

Posted on by
Reply

Using some internal metrics, netselect can tell you what mirror is “best” for you to use for downloading packages, or setting up your own mirror. netselect uses icmp to determine latency, and hop count between you and a mirror. It can take a list of many mirrors, tests them, and reports which one has the best (lowest) metric. If you want rsync or ftp as a preferred transport, you could change it to only look for one of those. Throw in a few verbose flags to get more output.

# netselect -s 20 `wget https://launchpad.net/ubuntu/+archivemirrors -q -O - | grep '>http' |cut -d / -f 3 | tr '\n' ' '`
    3 mirror.tcpdiag.net
   14 149.20.4.71
   17 nz.archive.ubuntu.com
   17 ftp.citylink.co.nz
   17 mirrors.easynews.com
   18 mirrors.nl.eu.kernel.org
   18 ubuntu.securedservers.com
   45 mirrors.cat.pdx.edu
   58 mirror.peer1.net
   67 mirror.pnl.gov
   77 76.73.4.58
   90 ubuntu.mirrors.tds.net
   95 mirror.steadfast.net
  100 ubuntu-archives.mirror.nexicom.net
  102 mirrors.gigenet.com
  105 mirrors.xmission.com
  109 ubuntu.mirror.constant.com
  115 mirror.cs.umn.edu
  117 ubuntu.bhs.mirrors.ovh.net
  120 mirrors.rit.edu

In this case it looks like mirror.tcpdiag.net is the best choice. 

# ping -c 3 mirror.tcpdiag.net
PING mirror.tcpdiag.net (69.160.243.150) 56(84) bytes of data.
64 bytes from ip-69-160-243-150.static.atlanticmetro.net (69.160.243.150): icmp_req=1 ttl=59 time=3.11 ms
64 bytes from ip-69-160-243-150.static.atlanticmetro.net (69.160.243.150): icmp_req=2 ttl=59 time=2.85 ms
64 bytes from ip-69-160-243-150.static.atlanticmetro.net (69.160.243.150): icmp_req=3 ttl=59 time=3.27 ms

--- mirror.tcpdiag.net ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 2.852/3.081/3.275/0.185 ms

3ms is pretty close.

append file with wget

Posted on by
Reply

To append a file with one fetched from a URL, use wget and output to STDOUT then redirect and append where needed.

# wget -q 'http://10.11.178.141/all_pub_keys' -O - >>/root/.ssh/authorized_keys

In this case I wanted to add some public keys to an existing authorized_keys file.