Tag Archives: forloop

list contents of all docker volumes

Posted on by
Reply

To list the contents of a docker named volume, run a temporary container and mount the volume into the container, then do a directory listing. Loop over all the volumes to see what each one holds.

~$ for i in `docker volume ls -q`; do echo "volume: ${i}"; docker run --rm -it -v ${i}:/vol alpine:latest ls /vol; echo; done;
volume: 140f898b1c69b85585942aa7f25cf03eba6ac66125d4a122e2fe99455c4a1a3f

volume: 1fa7b49173076a3a1fdb07ea7ce65d7187ff80e8b1a56e2fa667ebbbc0543f3a
dump.rdb

volume: 5564a11a1945567ffcc231145c01c806afe13a02b3e0a548f1504a1cd36c9374
dump.rdb

volume: 6d0b313416430d2abc0c872b98fd4180bbda4d14560c0a5d98f534f33b792164
app                 ib_buffer_pool      private_key.pem
auto.cnf            ib_logfile0         public_key.pem
ca-key.pem          ib_logfile1         server-cert.pem
ca.pem              ibdata1             server-key.pem
client-cert.pem     mysql               sys
client-key.pem      performance_schema

volume: 8b08da4a38ca8f5924b90220db8c84384d90fe331a953a5aaa1a1944d826fc68

volume: 976239a3528b8b0b074b6b7438552e1d22c4f069cf20582d2250fcc1c068dc4f
dump.rdb

volume: aac262a93155286f4c551271d8d2a70f81ed1ca4cd56925e94006248e458895e
dump.rdb

volume: b385ebee063b72350fbc1158788cbb43d7da9b37ec95196b74caa6b22b1c115b
dump.rdb

volume: c73f2001f829e5574bd4246b2ab7a261a3f4d9a7ef89997765d7bf43883e5c24
dump.rdb

volume: ce73f9f85c475b1fd9cf4fede20fd04250ee7702e83db67c29c7118055275c28
dump.rdb

volume: foovolume1

volume: efc8a8855ac2c13d83c23573aebfd53e15072ec68d23e2793262f662ea0ae308

volume: foovolume2
auto.cnf                     ibdata1
ca-key.pem                   mysql
ca.pem                       performance_schema
client-cert.pem              private_key.pem
client-key.pem               public_key.pem
ib_buffer_pool               server-key.pem
ib_logfile0                  sys
ib_logfile1

volume: foovolume3

volume: phpsockettest
php-fpm.sock

volume: foovolume4
auto.cnf            ib_logfile0         public_key.pem
ca-key.pem          ib_logfile1         server-cert.pem
ca.pem              ibdata1             server-key.pem
client-cert.pem     mysql               sys
client-key.pem      performance_schema  
ib_buffer_pool      private_key.pem

testvol
asdf   asdf1  asdf2

This installation has some test files, backing files from a few different mysql databases, a unix socket, redis files, and empty volumes.

make large directory tree

Posted on by
Reply

I needed to create a large directory structure for some testing. I hacked together this quick script that makes a small or large directory tree easily. You define how many branch points there are and how many branches each branch has all the way from the trunk to the leaves.

#!/bin/bash

i=0;
l=0;
levels=2
dirsperlevel=3

rm -rf tree
mkdir tree && cd tree

while [ $i -lt $levels ];
do
  for j in `find . -mindepth $i -maxdepth $i -type d`
  do
    pushd $j > /dev/null 2>&1
    for k in `seq 1 $dirsperlevel`;
    do
      mkdir $k;
    done;
    popd > /dev/null 2>&1
  done;
  i=$(($i+1))
  l=`echo "($dirsperlevel^$i)+$l"|bc`
done;

echo "$l dirs created in ./tree"

using 2 levels, and 3 directories per level we get 12 total directories created like so:

# mktree.sh 
12 dirs created in ./tree
# find tree    
tree
tree/2
tree/2/2
tree/2/3
tree/2/1
tree/3
tree/3/2
tree/3/3
tree/3/1
tree/1
tree/1/2
tree/1/3
tree/1/1

using something like 6 levels and 6 directories per level would give us 55,986 total directories.

insert characters into string with sed

Posted on by
Reply 

# echo 20150429 | sed -e 's/\(.\{4\}\)\(.\{2\}\)/\1\/\2\//'
2015/04/29

Start with a single flat directory with thousands of log files…

# ls | head -5
db301.20140216.log.gz
db301.20140217.log.gz
db301.20140218.log.gz
db301.20140219.log.gz
db301.20140220.log.gz

Now move timestamped files into sorted directory by day

# for i in `ls`; do j=$(echo $i| cut -d . -f 2  | sed -e 's/\(.\{4\}\)\(.\{2\}\)/\1\/\2\//'); mkdir -p $j && mv $i $j; done;


check your work

# find . -type f | head -5
./2014/02/16/db301.20140216.log.gz
./2014/02/17/db301.20140217.log.gz
./2014/02/18/db301.20140218.log.gz
./2014/02/19/db301.20140219.log.gz
./2014/02/20/db301.20140220.log.gz

forgot to rename files

# for i in `find . -mindepth 3 -type d `; do pushd $i; for j in `ls`; do k=$(echo $j | sed -e 's/\(\.[0-9]\{8\}\)//' ); mv $j $k;done; popd; done;

check your work

# find . -type f | head -5
./2014/02/16/db301.log.gz
./2014/02/17/db301.log.gz
./2014/02/18/db301.log.gz
./2014/02/19/db301.log.gz
./2014/02/20/db301.log.gz

use eval to run commands generated by awk

Posted on by
Reply

Here’s one way to generate a set of commands with awk, and then run them in a loop with eval. 

# cat snippet
field1 /mnt/somedir/785/8785/948785 41 /mnt/somedir2/785/8785/948785 1 2
field1 /mnt/somedir/791/8791/948791 2 /mnt/somedir2/791/8791/948791 6 2
field1 /mnt/somedir/924/8924/948924 2 /mnt/somedir2/924/8924/948924 23 2
field1 /mnt/somedir/993/8993/948993 2 /mnt/somedir2/993/8993/948993 19876 2
field1 /mnt/somedir/3/9003/949003 8 /mnt/somedir2/3/9003/949003 273 2
field1 /mnt/somedir/70/9070/949070 341 /mnt/somedir2/70/9070/949070 6 2
field1 /mnt/somedir/517/4517/954517 2 /mnt/somedir2/517/4517/954517 14 2
field1 /mnt/somedir/699/4699/954699 210 /mnt/somedir2/699/4699/954699 1 2
field1 /mnt/somedir/726/4726/954726 1 /mnt/somedir2/726/4726/954726 6 2

Now use awk to get the output you want and generate commands. Use a forloop and eval to run them.

# for i in `awk '{if($3>$5) print "rsync -a --ignore-existing "$2"/ "$4}' left.compare.sorted  `; do echo $i; eval $i; done;
rsync -a --ignore-existing /mnt/somedir/70/9070/949070/ /mnt/somedir2/70/9070/949070
rsync -a --ignore-existing /mnt/somedir/699/4699/954699/ /mnt/somedir2/699/4699/954699
#

Vyatta create and update IP based ban lists from Spamhaus

Posted on by
2

You can use Spamhaus, or a number of other lists in a Vyatta firewall configuration. In this case we create a network group called ‘blocked’ from the Spamhaus blacklists. Then this network group can be used in firewalls to drop traffic. Use cron to update the list every day, or once a week.

#!/bin/bash
# FILE: /usr/local/sbin/updateBanList.sh
# AUTHOR: ForDoDone fordodone@fordodone.com
# DATE: 2013-10-01
# NOTES: Script to update IP ban list.  Run from cron, and integrate into firewall
# 

# variables
VERBOSE=0
DROPURL='http://www.spamhaus.org/drop/drop.txt'
EDROPURL='http://www.spamhaus.org/drop/edrop.txt'

# simple logger function
logger(){
  if [ "$VERBOSE" == "1" ]
  then
    echo "$@"
  fi
}

# set verbose flag if given
if [ "$1" == "-v" ]
then
VERBOSE=1;
fi

# create or truncate tmp file
>/tmp/block

# get drop file
wget -q $DROPURL -O - | grep ^[0-9] | sed -e 's/;.*//' >> /tmp/block
if [ $? -ne 0 ]
then
  logger "error getting drop file"
  logger "exiting..."
exit
fi

# get edrop file
wget -q "$EDROPURL" -O - | grep ^[0-9] | sed -e 's/;.*//' >> /tmp/block
if [ $? -ne 0 ]
then
  logger "error getting edrop file"
  logger "exiting..."
exit
fi
logger "received `wc -l /tmp/block | awk '{print $1}'` networks to block..."

logger "starting vyatta cmd wrapper"
/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper begin

# remove existing list, in case a network has been removed"
logger "deleting existing blocked network group"
/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper delete firewall group network-group blocked

# add each network to the block list
logger "building new blocked network group"
logger "this might take a while..."
for i in `cat /tmp/block`;
do
  /opt/vyatta/sbin/vyatta-cfg-cmd-wrapper set firewall group network-group blocked network $i
done;

# now commit the changes
logger "committing changes"
/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper commit

logger "ending vyatta cmd wrapper"
/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper end

# clean up
rm -rf /tmp/block >/dev/null 2>&1

ESXi add vlans

Posted on by
Reply

Using ESXi 4.1:

vm705:~ # esxcfg-vswitch --add-pg=vlan301 vSwitch0           
vm705:~ # esxcfg-vswitch --vlan=301 --pg=vlan301 vSwitch0

And this was to move a bunch of VLANs from one vSwitch to another:

vm705:~ # for i in `seq 221 235`; do esxcfg-vswitch -D vlan$i vSwitch0; done;
vm705:~ # for i in `seq 221 235`; do esxcfg-vswitch -A vlan$i vSwitch1; done;
vm705:~ # for i in `seq 221 235`; do esxcfg-vswitch -p vlan$i -v $i vSwitch1; done;

NetApp decode acp domain option

Posted on by
Reply

How does this option function to set a network? The acp.domain option is a convoluted decimal representation of the network portion of the IP address used for acp.

toaster*> options acp
acp.domain 65193
acp.enabled on
acp.netmask 65535
acp.port e0f

Take 65193 and convert it to binary: 1111111010101001. Then split it up into two (or more) octets: 11111110 10101001. Then convert each of the octets back to decimal: 254 169. Then reverse the order: 169 254. That is the acp network. The netmask portion is more straightforward. In this case our ACP network is 169.254/16.

You could hack a quick little one liner:

# for i in `echo "obase=2;65193" |bc | awk 'BEGIN{FS=""} {for(i=1;i<33;i++){printf $i; if(i==8)printf " ";}printf "\n"}'`; do echo "ibase=2;$i" |bc; done|tac | paste - - | sed 's/\t/./'
169.254
#