# for i in `cat servers`; do echo "$i: " ; ssh $i "awk '\$4~/(^|,)ro($|,)/' /proc/mounts; done;www1:
www2:
www3:
/dev/mapper/vg01-tmp on /tmp type ext4 (ro)
www4:
find read-only mounts
Reply
Category Archives: Troubleshooting
wget monitor website download speed
# while true; do date | tr '\n' '-' | sed -e 's/-/ --- /'; wget http://testsite.com/fancy.pdf -O /dev/null 2>&1 | grep saved | awk -F"[()]" '{print $2}'; sleep 1s; done;
Thu Oct 30 15:18:26 PDT 2014 --- 1.25 MB/s
Thu Oct 30 15:18:28 PDT 2014 --- 1.20 MB/s
Thu Oct 30 15:18:29 PDT 2014 --- 958.95 KB/s
Thu Oct 30 15:18:31 PDT 2014 --- 1.36 MB/s
Thu Oct 30 15:18:32 PDT 2014 --- 873.98 KB/s
Thu Oct 30 15:18:33 PDT 2014 --- 1.38 MB/s
Thu Oct 30 15:18:35 PDT 2014 --- 261.90 KB/s
Thu Oct 30 15:18:37 PDT 2014 --- 1.38 MB/s
Thu Oct 30 15:18:38 PDT 2014 --- 360.14 KB/s
Thu Oct 30 15:18:40 PDT 2014 --- 1.37 MB/s
Thu Oct 30 15:18:42 PDT 2014 --- 427.06 KB/s
Thu Oct 30 15:18:44 PDT 2014 --- 1.37 MB/s
Thu Oct 30 15:18:45 PDT 2014 --- 397.54 KB/s
ftp to dev null to test bandwidth
When testing bandwidth, and troubleshooting bottlenecks, I prefer to use iperf. If you insist on testing bandwidth with FTP, it’s important NOT to use regular files. If you transfer actual files, the transfer could be limited by disk i/o, due to reads and writes. To eliminate this you can FTP from /dev/zero to /dev/null. It sounds super easy, but you have to use FTP in a special way to get it to read and write to special devices.
Here’s a little script. Be sure to replace the destination IP address, username and password with actual values:
# cat ftp_dev_null.sh#!/bin/bash /usr/bin/ftp -n <IP address of machine> <<END verbose on user <usernanme> <password> bin put "|dd if=/dev/zero bs=32k" /dev/null bye END
# ftp_dev_null.sh
Verbose mode on.
331 Password required for fordodone
230 User fordodone logged in
Remote system type is UNIX.
Using binary mode to transfer files.
200 Type set to I
local: |dd if=/dev/zero bs=32k remote: /dev/null
200 PORT command successful
150 Opening BINARY mode data connection for /dev/null
^C
send aborted
waiting for remote to finish abort
129188+0 records in
129187+0 records out
4233199616 bytes (4.2 GB) copied, 145.851 s, 29.0 MB/s
226 Transfer complete
4233142272 bytes sent in 145.82 secs (28350.0 kB/s)
221 Goodbye.
#In this case I was getting around 230Mbits per second (over an IPSec tunnel) between my client and the FTP server. Not too bad.
dig spf and domainkeys txt records
# dig txt fordodone.com @4.2.2.2
;; ANSWER SECTION:
fordodone.com. 900 IN TXT "v=spf1 mx -all"
fordodone.com. 900 IN TXT "spf2.0/mfrom mx"
# dig txt mx._domainkey.fordodone.com
;; ANSWER SECTION:
mx._domainkey.fordodone.com. 3472 IN TXT "p=MIGfMA0GCSqGSIb3DQEBAJFHEBCY5DCBiQKBgQC1zZ4Mc7HiXNM0Cu3GZxwcI9vs4nIIcAIlzefnB9MbXrmQJF0m75BsKj8xer23oS7apleInob0RjneHbi+bwvEAa/NN7AjxTxla42dLoXOM+1B0Wf6taG3Vma/FS3bOdqWKo1J4hP8masXJ+PdkMy+LWqSp66fvJtG9U6/eQ4sJwIDAQAB\;"quick memcache testing and troubleshooting
Here’s a quick way to test your new memcached setup. At first it’s a bit confusing to understand that memcache has no knowledge of what keys it stores, just whether or not it has a particular key. Unlike relational databases, you can’t query memcache and get all of the values it has. You have to know the key that you want to fetch before you ask memcache about it. Let’s ask it the value for the key foo:
telnet memcacheserver 11211
Trying 10.131.215.181...
Connected to 10.131.215.181
Escape character is '^]'.
get foo
ENDIt returns nothing, so it doesn’t have any value for that key. The key foo is unset. Let’s set it:
set foo 0 0 3
bar
STOREDWhen you set a key like this, follow the syntax “set <keyname> <flag> <ttl> <storebytes>”. In this case our key is foo, we have no important flags (0), there is no expiration of the key value pair (0), and the data we are about to store is 3 bytes (aka 3 characters). Let’s fetch it now:
get foo
VALUE foo 0 3
bar
ENDIt returns the value of key foo as bar. Now delete it:
delete foo
DELETEDNow another, this time the key is “foobar”, and the data is a 12 byte string “barbarbarbar”:
set foobar 0 0 12
barbarbarbar
STORED
get foobar
VALUE foobar 0 12
barbarbarbar
END
delete foobar
DELETED
^]
telnet> close
Connection closed.scan range of IP addresses
nmap is an amazing utility. With all of it’s flags and options it really gives you the power to know what is out in a network.
I used this to do a simple ping scan of my home network:
# nmap -nsP 192.168.1.0/24
Starting Nmap 5.00 ( http://nmap.org ) at 2013-08-22 09:06 MST
Host 192.168.1.1 is up (0.0051s latency).
MAC Address: 00:18:39:4E:82:60 (Cisco-Linksys)
Host 192.168.1.90 is up.
Nmap done: 256 IP addresses (2 hosts up) scanned in 5.51 seconds
#Hosts that allow icmp echo requests will show up. I can see my router and my workstation, but no other hosts (no other pingable hosts at least). I’m looking for a RPi that I recently plugged into the network. It doesn’t look like it’s up. Time to drag out the spare monitor and keyboard…
get current client IP addresses from web farm
To see what common IPs are connecting to your web farm, ssh to all of the servers and get a list of clients. Then sort it until you see most busy clients.
# for i in `seq 401 436`; do ssh www$i "netstat -natp | grep EST | grep apa | grep ":80 "| awk '{print \$5}' | cut -d : -f1"; done | sort | uniq -c | sort -nk1 | tail
3 10.0.0.1
3 10.0.0.10
3 10.245.34.2
4 10.29.45.89
5 10.111.111.111
5 10.239.234.234
5 10.1.1.1
5 10.2.2.2
6 10.3.3.3
10 10.100.100.100
#The list shows the number of connections, and the client IP.
monitor host for slow ping times
When there is intermittent network latency to a host, it’s important to monitor a it for a pattern. Using ping can help narrow down what is causing the latency. VMWare load, bandwidth limitations, employee work patterns, backups, and many other sources could be the cause of the latency.
while true; do j=`ping <slowhost> -i1 -c1 2>&1 | grep icmp_req | awk '{print $7}' | cut -d = -f2 | cut -d . -f1`; if [ $j -gt 30 ]; then date | tr '\n' ' '; echo $j; fi; sleep 1s; done;This does a ping every second, and if it’s over a threshold (30ms in this case) it is considered unacceptable and logged with date.
NetApp access NTFS CIFS share from Unix host via NFS
NTFS vs. Unix style volume settings have nothing to do with which hosts can mount the volume, they have to do with permissions. To access a NTFS volume via NFS, first allow rw or root mounting in /etc/exports (you do have your root vol mounted on your admin boxes right?):
# sed -i '/cifsshare/d' /mnt/toaster/vol0/etc/exports
# echo '/vol/cifsshare -sec=sys,rw,root=someadminhost:anotherlinuxbox,anon=0,nosuid' >> /mnt/toaster/vol0/etc/exports
# ssh toaster
toaster> exportfs -a
toaster> Connection to toaster closed by remote host.
Connection to toaster closed.
#Mount the volume on your administration host and list the directory:
# mkdir -p /mnt/toaster/cifsshare
# mount toaster:/vol/cifsshare /mnt/toaster/cifsshare
# cd /mnt/toaster/cifsshare
# ls
ls: .: Permission denied
#
# whoami
root
#So even though we are able to mount this share via NFS, the NTFS permissions do not let us see what’s there. Check the filer to see what permissions context it has for ‘root’.
toaster> wcc -u root
Tue Jul 16 09:11:57 PDT [toaster: auth.trace.authenticateUser.loginTraceMsg:info]: AUTH: LSA lookup: Lookup of account "DOMAINNAME\root" failed: STATUS_NONE_MAPPED (0xc0000073).
(NT - UNIX) account name(s): (DOMAINNAME\guest - root)
***************
UNIX uid = 0
user is a member of group daemon (1)
user is a member of group daemon (1)
NT membership
DOMAINNAME\Guest
DOMAINNAME\Domain Guests
DOMAINNAME\Domain Users
BUILTIN\Guests
BUILTIN\Users
User is also a member of Everyone, Network Users,
Authenticated Users
***************
toaster> Looks like the filer doesn’t recognize the user ‘root’ and sees it as a guest. This explains why we might not have permissions in the ‘cifsshare’ mount. The solution is to add a user mapping so that user ‘root’ is recognized as ‘administrator’ for the domain ‘DOMAINNAME’. Make an entry in usermap.cfg (you do have your root vol mounted on your admin boxes right?):
echo 'DOMAINNAME\administrator == root' >>/mnt/toaster/vol0/etc/usermap.cfgNow let’s see what user ‘root’ is seen as from the view of the filer:
toaster> wcc -u root
Tue Jul 16 09:12:30 PDT [toaster: auth.trace.authenticateUser.loginTraceMsg:info]: AUTH: LSA lookup: Located account "DOMAINNAME\administrator" in domain "DOMAINNAME"..
(NT - UNIX) account name(s): (DOMAINNAME\administrator - root)
***************
UNIX uid = 0
user is a member of group daemon (1)
user is a member of group daemon (1)
NT membership
DOMAINNAME\administrator
DOMAINNAME\Enterprise Admins
DOMAINNAME\Exchange Recovery Administrators
DOMAINNAME\Schema Admins
<a ton of other stuff here>
BUILTIN\Administrators
BUILTIN\Users
User is also a member of Everyone, Network Users,
Authenticated Users
***************
toaster>
Now we have all the privileges that the domain administrator has, and we can view, list, and alter files that the domain administrator has permissions for. In a production environment, you could just map a Linux admin jdoe to DOMAINNAME\jdoe assuming they had domain admin permissions.
check if a directory is a nfs mount
Use stat to display the file system status, following links, and output the format as the human readable file system type:
#stat -f -L -c %T /tmp
ext2/ext3
#
# stat -f -L -c %T /mnt/fileserver/volume
nfs
#As suspected “/tmp” is a regular ext file system, but the path “/mnt/fileserver/volume” is an nfs mount. What happens if we unmount the nfs mount:
# umount /mnt/fileserver/volume
#
# stat -f -L -c %T /mnt/fileserver/volume
ext2/ext3
#It reports properly that the directory is just an unmounted directory in a regular ext file system.
Alternatively you can use the mountpoint command:
# mountpoint /tmp
/tmp is not a mountpoint
#
# mountpoint /mnt/adfs40/vol1
/mnt/adfs40/vol1 is a mountpoint
#
# umount /mnt/fileserver/volume
# mountpoint /mnt/adfs40/vol1
/mnt/adfs40/vol1 is not a mountpoint
#