monitor Apache memory usage

When looking at a webserver for memory usage, it’s important to consider the VSZ and RSS memory usage.

This little one liner gets the Total and Average VSZ and RSS usage as well as thread count, and prints those statistics every 5 seconds:

# while true; do ps auxfww | grep apache | grep -v -e cronolog -e grep | awk '{ vsum+=$5; rsum+=$6 } END { print "VSZ:", vsum, "(", vsum/NR, ") RSS:", rsum, "(", rsum/NR, ") Procs:", NR }'; sleep 5; done;
VSZ: 9896272 ( 341251 ) RSS: 1716216 ( 59179.9 ) Procs: 29
VSZ: 9547608 ( 340986 ) RSS: 1650100 ( 58932.1 ) Procs: 28
VSZ: 9546328 ( 340940 ) RSS: 1649044 ( 58894.4 ) Procs: 28
VSZ: 9861976 ( 340068 ) RSS: 1687968 ( 58205.8 ) Procs: 29
VSZ: 9868632 ( 340298 ) RSS: 1694496 ( 58430.9 ) Procs: 29
VSZ: 9853272 ( 339768 ) RSS: 1679112 ( 57900.4 ) Procs: 29
VSZ: 9853272 ( 339768 ) RSS: 1679264 ( 57905.7 ) Procs: 29
^C
#

So there are around 29 threads running right now on this server. The threads are using an average of 340MB per thread VSZ, and 59MB per thread RSS. The total of around 1.7GB of RSS looks good, on a machine with 8G physical memory.

use smbclient to connect to CIFS share from linux

To connect to a Windows network drive, or CIFS share, you can use a simple Linux utility called smbclient. This is much faster than trying to map a network drive through Windows. You can use this simple tool to test that the file server is working correctly.

# smbclient --user="DOMAIN\fordodone" //10.107.0.101/share_name
Enter DOMAIN\fordodone's password:
Domain=[DOMAIN] OS=[Windows 5.0] Server=[Windows 2000 LAN Manager]
smb: \> 

You can use commands like cd, ls, cp, etc. to interact with the share (similar to FTP).

Linux rebuild software RAID1

Set the bash field separator to newline:

IFS="
"

See what disk and partitions are currently up, then generate commands to re-add the missing disk and partitions, then run them:

for i in `cat /proc/mdstat | grep md | cut -d [ -f1 | sed -e 's/\(md[0-9]\).*\(sd[a-z][0-9]\)/mdadm --add \/dev\/\1 \/dev\/\2/' | sed -e 's/sdb/sda/'`;  do eval $i; done;

TODO: make it determine which disk to add (/dev/sda or /dev/sdb)

check if a directory is a nfs mount

Use stat to display the file system status, following links, and output the format as the human readable file system type:

#stat -f -L -c %T /tmp
ext2/ext3
#
# stat -f -L -c %T /mnt/fileserver/volume
nfs
#

As suspected “/tmp” is a regular ext file system, but the path “/mnt/fileserver/volume” is an nfs mount. What happens if we unmount the nfs mount:

# umount /mnt/fileserver/volume
#
# stat -f -L -c %T /mnt/fileserver/volume
ext2/ext3
#

It reports properly that the directory is just an unmounted directory in a regular ext file system.

Alternatively you can use the mountpoint command:

# mountpoint /tmp
/tmp is not a mountpoint
#
# mountpoint /mnt/adfs40/vol1
/mnt/adfs40/vol1 is a mountpoint
#
# umount /mnt/fileserver/volume
# mountpoint /mnt/adfs40/vol1
/mnt/adfs40/vol1 is not a mountpoint
#

change SSH listen address

If you have servers with internal and external interfaces, you may want to disable ssh on the external side. In this case we just get the internal IP address and tell sshd to only listen on that address:

sed -i "s/#ListenAddress 0.0.0.0/ListenAddress `grep address /etc/network/interfaces | grep 10.229 | awk '{print $2}'`/" /etc/ssh/sshd_config

Do it to many hosts:

for i in `seq 313 364`; do ssh ftp$i "sed -i \"s/#ListenAddress 0.0.0.0/ListenAddress \`grep address /etc/network/interfaces | grep 10.229 | awk '{print \$2}'\`/\" /etc/ssh/sshd_config"

And restart SSH:

for i in `seq 313 364`; do ssh ftp$i "service ssh restart"; done;

drop messages in mailqueue from single sender

Drop all messages from the sender ‘nagios’:

# for i in `mailq | tail -n +2 | awk  'BEGIN { RS = "" } { if ($7== "nagios") print $1}'`; do postsuper -d $i; done;
postsuper: B60BF9FB69: removed
postsuper: Deleted: 1 message
postsuper: C3B429FB6F: removed
postsuper: Deleted: 1 message
postsuper: 0306C9FB87: removed
postsuper: Deleted: 1 message
postsuper: E3BC79FB7E: removed
postsuper: Deleted: 1 message
postsuper: B32EA9FB65: removed
(many more lines)

Gets the mailqueue, starts the output on line 2, skipping the first header line, if the sender equals ‘nagios’ then print the first field, which is the message id. Then use postsuper to drop the message identified by it’s id.