Monthly Archives: December 2013

use eval to run commands generated by awk

Posted on by
Reply

Here’s one way to generate a set of commands with awk, and then run them in a loop with eval. 

# cat snippet
field1 /mnt/somedir/785/8785/948785 41 /mnt/somedir2/785/8785/948785 1 2
field1 /mnt/somedir/791/8791/948791 2 /mnt/somedir2/791/8791/948791 6 2
field1 /mnt/somedir/924/8924/948924 2 /mnt/somedir2/924/8924/948924 23 2
field1 /mnt/somedir/993/8993/948993 2 /mnt/somedir2/993/8993/948993 19876 2
field1 /mnt/somedir/3/9003/949003 8 /mnt/somedir2/3/9003/949003 273 2
field1 /mnt/somedir/70/9070/949070 341 /mnt/somedir2/70/9070/949070 6 2
field1 /mnt/somedir/517/4517/954517 2 /mnt/somedir2/517/4517/954517 14 2
field1 /mnt/somedir/699/4699/954699 210 /mnt/somedir2/699/4699/954699 1 2
field1 /mnt/somedir/726/4726/954726 1 /mnt/somedir2/726/4726/954726 6 2

Now use awk to get the output you want and generate commands. Use a forloop and eval to run them.

# for i in `awk '{if($3>$5) print "rsync -a --ignore-existing "$2"/ "$4}' left.compare.sorted  `; do echo $i; eval $i; done;
rsync -a --ignore-existing /mnt/somedir/70/9070/949070/ /mnt/somedir2/70/9070/949070
rsync -a --ignore-existing /mnt/somedir/699/4699/954699/ /mnt/somedir2/699/4699/954699
#

find directories owned by root

Posted on by
1

Find the directories owned by root in a certain part of the tree:

# find . -depth -mindepth 1 -maxdepth 3 -type d -ls | awk '$5 ~ /root/ {print}'
  7930    0 drwxr-xr-x  12 root root      115 Oct 11 16:44 ./562
3805069    0 drwxr-xr-x   3 root root       20 Oct 11 16:44 ./562/8562
  7946    0 drwxr-xr-x   5 root root       46 Dec  8 23:52 ./563/6563
  7947    0 drwxr-xr-x   3 root root      21 Oct 21  2008 ./563/6563/456563
3464735    0 drwxr-xr-x   2 root root        6 Sep 26 17:29 ./563/6563/436563
4075144    4 drwxr-xr-x   2 root root     4096 Dec  9 00:39 ./563/6563/2366563

Change all the ownership to www-data:

# find . -depth -mindepth 1 -maxdepth 3 -type d -exec chown www-data: {} \;

You could do this:

# cd .. && chown -R www-data: dirname

But we only suspect the problem at a certain level in the tree, and it would be way slow to recursively chown hundreds of millions of files.

vyatta create firewall network group from geo ip table

Posted on by
Reply

The global Geo IP table is huge, and contains ranges of IP addresses that don’t necessarily fit into CIDR networks. Converting the IP ranges in the table into networks results in around 129,000 separate networks. Trying to define every network scope in the global table for inclusion in a firewall is well outside the realm of possibility. In the case of a few small countries, their ranges do actually fit into network groups. This is just a fun little exercise not to be used in production of course.

#!/usr/bin/perl
# FILE: netGroupCountry.pl
# AUTHOR: ForDoDone
# DATE: 2013-12-02
# NOTES: create a firewall network group for a specific country using a geoip table

use strict;
use Net::Netmask;

# download geoip table
# TODO: wget geoip table if doesn't exist, or if 1st wed of month

# set country here
# TODO: read from ARGV
# use Estonia for example it only has 236 IP ranges in table
my $country = 'EE';

# set geoip database CSV
my $geoIPDB='/var/lib/geoip/GeoIPCountryWhois.csv';

# open geoip table
open FILE, "$geoIPDB" or die $!;

# start the vyatta config wrapper
system("/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper begin");

# clear the old network group
system("/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper delete firewall group network-group $country");

# commit here optional
#system("/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper commit");

# setup commit counter for chunked commits
#my $i=0;
#my $chunk=100;

# read each line of the geoip table (~82k lines)
while(<FILE>){
  chomp;

  my @line = split(/"/,$_);

  # if country matches the one we want
  # convert the ip range to CIDR(s)
  # add the network to the network group
  if ("$line[9]" eq "$country" ){

    my @blocks = range2cidrlist("$line[1]","$line[3]");

        foreach(@blocks){
          my $setcmd = "set firewall group network-group $country network $_";
          system("/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper $setcmd");

          #print "set firewall group network-group $country network $_\n";

          # chunked commit
          #if ( $i % $chunk ){
          #  # do nothing
          #} else {
          #  print "committing after 100 entries... total: $i\n";
          #  system("/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper commit");
          #}
          #$i++;

        }

  }

}

# close geoip table
close FILE;

# run vyatta config wrapper to commit and end
system("/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper commit");
system("/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper end");

vyatta@vyatta:/usr/local/sbin$ time ./netGroupCountry.pl
Nothing to delete (the specified node does not exist)

real	1m3.515s
user	0m22.660s
sys	0m27.650s
vyatta@vyatta:/usr/local/sbin$

This definitely won’t work for larger networks, and who knows how it actually affects performance, I’ve never actually tried using the network group in the firewall.