make large directory tree

I needed to create a large directory structure for some testing. I hacked together this quick script that makes a small or large directory tree easily. You define how many branch points there are and how many branches each branch has all the way from the trunk to the leaves.

#!/bin/bash

i=0;
l=0;
levels=2
dirsperlevel=3

rm -rf tree
mkdir tree && cd tree

while [ $i -lt $levels ];
do
  for j in `find . -mindepth $i -maxdepth $i -type d`
  do
    pushd $j > /dev/null 2>&1
    for k in `seq 1 $dirsperlevel`;
    do
      mkdir $k;
    done;
    popd > /dev/null 2>&1
  done;
  i=$(($i+1))
  l=`echo "($dirsperlevel^$i)+$l"|bc`
done;

echo "$l dirs created in ./tree"

using 2 levels, and 3 directories per level we get 12 total directories created like so:

# mktree.sh 
12 dirs created in ./tree
# find tree    
tree
tree/2
tree/2/2
tree/2/3
tree/2/1
tree/3
tree/3/2
tree/3/3
tree/3/1
tree/1
tree/1/2
tree/1/3
tree/1/1

using something like 6 levels and 6 directories per level would give us 55,986 total directories.

share:

insert characters into string with sed

# echo 20150429 | sed -e 's/\(.\{4\}\)\(.\{2\}\)/\1\/\2\//'
2015/04/29

Start with a single flat directory with thousands of log files…

# ls | head -5
db301.20140216.log.gz
db301.20140217.log.gz
db301.20140218.log.gz
db301.20140219.log.gz
db301.20140220.log.gz

Now move timestamped files into sorted directory by day

# for i in `ls`; do j=$(echo $i| cut -d . -f 2  | sed -e 's/\(.\{4\}\)\(.\{2\}\)/\1\/\2\//'); mkdir -p $j && mv $i $j; done;


check your work

# find . -type f | head -5
./2014/02/16/db301.20140216.log.gz
./2014/02/17/db301.20140217.log.gz
./2014/02/18/db301.20140218.log.gz
./2014/02/19/db301.20140219.log.gz
./2014/02/20/db301.20140220.log.gz

forgot to rename files

# for i in `find . -mindepth 3 -type d `; do pushd $i; for j in `ls`; do k=$(echo $j | sed -e 's/\(\.[0-9]\{8\}\)//' ); mv $j $k;done; popd; done;

check your work

# find . -type f | head -5
./2014/02/16/db301.log.gz
./2014/02/17/db301.log.gz
./2014/02/18/db301.log.gz
./2014/02/19/db301.log.gz
./2014/02/20/db301.log.gz

share:

use eval to run commands generated by awk

Here’s one way to generate a set of commands with awk, and then run them in a loop with eval.

# cat snippet
field1 /mnt/somedir/785/8785/948785 41 /mnt/somedir2/785/8785/948785 1 2
field1 /mnt/somedir/791/8791/948791 2 /mnt/somedir2/791/8791/948791 6 2
field1 /mnt/somedir/924/8924/948924 2 /mnt/somedir2/924/8924/948924 23 2
field1 /mnt/somedir/993/8993/948993 2 /mnt/somedir2/993/8993/948993 19876 2
field1 /mnt/somedir/3/9003/949003 8 /mnt/somedir2/3/9003/949003 273 2
field1 /mnt/somedir/70/9070/949070 341 /mnt/somedir2/70/9070/949070 6 2
field1 /mnt/somedir/517/4517/954517 2 /mnt/somedir2/517/4517/954517 14 2
field1 /mnt/somedir/699/4699/954699 210 /mnt/somedir2/699/4699/954699 1 2
field1 /mnt/somedir/726/4726/954726 1 /mnt/somedir2/726/4726/954726 6 2

Now use awk to get the output you want and generate commands. Use a forloop and eval to run them.

# for i in `awk '{if($3>$5) print "rsync -a --ignore-existing "$2"/ "$4}' left.compare.sorted  `; do echo $i; eval $i; done;
rsync -a --ignore-existing /mnt/somedir/70/9070/949070/ /mnt/somedir2/70/9070/949070
rsync -a --ignore-existing /mnt/somedir/699/4699/954699/ /mnt/somedir2/699/4699/954699
#
share:

Vyatta create and update IP based ban lists from Spamhaus

You can use Spamhaus, or a number of other lists in a Vyatta firewall configuration. In this case we create a network group called ‘blocked’ from the Spamhaus blacklists. Then this network group can be used in firewalls to drop traffic. Use cron to update the list every day, or once a week.

#!/bin/bash
# FILE: /usr/local/sbin/updateBanList.sh
# AUTHOR: ForDoDone fordodone@fordodone.com
# DATE: 2013-10-01
# NOTES: Script to update IP ban list.  Run from cron, and integrate into firewall
# 

# variables
VERBOSE=0
DROPURL='http://www.spamhaus.org/drop/drop.txt'
EDROPURL='http://www.spamhaus.org/drop/edrop.txt'

# simple logger function
logger(){
  if [ "$VERBOSE" == "1" ]
  then
    echo "$@"
  fi
}

# set verbose flag if given
if [ "$1" == "-v" ]
then
VERBOSE=1;
fi

# create or truncate tmp file
>/tmp/block

# get drop file
wget -q $DROPURL -O - | grep ^[0-9] | sed -e 's/;.*//' >> /tmp/block
if [ $? -ne 0 ]
then
  logger "error getting drop file"
  logger "exiting..."
exit
fi

# get edrop file
wget -q "$EDROPURL" -O - | grep ^[0-9] | sed -e 's/;.*//' >> /tmp/block
if [ $? -ne 0 ]
then
  logger "error getting edrop file"
  logger "exiting..."
exit
fi
logger "received `wc -l /tmp/block | awk '{print $1}'` networks to block..."

logger "starting vyatta cmd wrapper"
/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper begin

# remove existing list, in case a network has been removed"
logger "deleting existing blocked network group"
/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper delete firewall group network-group blocked

# add each network to the block list
logger "building new blocked network group"
logger "this might take a while..."
for i in `cat /tmp/block`;
do
  /opt/vyatta/sbin/vyatta-cfg-cmd-wrapper set firewall group network-group blocked network $i
done;

# now commit the changes
logger "committing changes"
/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper commit

logger "ending vyatta cmd wrapper"
/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper end

# clean up
rm -rf /tmp/block >/dev/null 2>&1

share:

ESXi add vlans

Using ESXi 4.1:

vm705:~ # esxcfg-vswitch --add-pg=vlan301 vSwitch0           
vm705:~ # esxcfg-vswitch --vlan=301 --pg=vlan301 vSwitch0

And this was to move a bunch of VLANs from one vSwitch to another:

vm705:~ # for i in `seq 221 235`; do esxcfg-vswitch -D vlan$i vSwitch0; done;
vm705:~ # for i in `seq 221 235`; do esxcfg-vswitch -A vlan$i vSwitch1; done;
vm705:~ # for i in `seq 221 235`; do esxcfg-vswitch -p vlan$i -v $i vSwitch1; done;
share:

NetApp decode acp domain option

How does this option function to set a network? The acp.domain option is a convoluted decimal representation of the network portion of the IP address used for acp.

toaster*> options acp
acp.domain 65193
acp.enabled on
acp.netmask 65535
acp.port e0f

Take 65193 and convert it to binary: 1111111010101001. Then split it up into two (or more) octets: 11111110 10101001. Then convert each of the octets back to decimal: 254 169. Then reverse the order: 169 254. That is the acp network. The netmask portion is more straightforward. In this case our ACP network is 169.254/16.

You could hack a quick little one liner:

# for i in `echo "obase=2;65193" |bc | awk 'BEGIN{FS=""} {for(i=1;i<33;i++){printf $i; if(i==8)printf " ";}printf "\n"}'`; do echo "ibase=2;$i" |bc; done|tac | paste - - | sed 's/\t/./'
169.254
#
share:

Linux rebuild software RAID1

Set the bash field separator to newline:

IFS="
"

See what disk and partitions are currently up, then generate commands to re-add the missing disk and partitions, then run them:

for i in `cat /proc/mdstat | grep md | cut -d [ -f1 | sed -e 's/\(md[0-9]\).*\(sd[a-z][0-9]\)/mdadm --add \/dev\/\1 \/dev\/\2/' | sed -e 's/sdb/sda/'`;  do eval $i; done;

TODO: make it determine which disk to add (/dev/sda or /dev/sdb)

share: