Category Archives: Troubleshooting

monitor host for connectivity

Posted on by
Reply

Sometimes, you want to be notified if a host goes up or down. Usually Nagios is perfect for this, but in this case I had an internet circuit, and all I cared about was knowing when the ISP deactivated it. Use ping in a loop, make 1 request every second, if ping doesn’t get a response, then send a text message (Verizon number) and stop the loop.

while true; do ping -nc 1 -W 1 5.6.7.8 | grep -q icmp; if [ "$?" == "1" ]; then echo "circuit is down" | mail <10-digit phone number no spaces>@vtext.com; break; fi; sleep 1s; done;

I also use the converse of this method when I want to know when a new circuit comes up.

get drive serial numbers from NetApp DS4243 shelves strapped to Linux server

Posted on by
Reply

When doing data recovery on a failed 3Ware RAID group, I utilized a couple spare NetApp DS4243 shelves. I put all the SATA drives into brackets, popped a quad port SAS card into a 1U server, and booted my rescue image from the network. The Debian OS was able to see each individual drive, at which point I could make ddrescue copies, and edit the 3Ware DCB metadata using hexedit. Keeping everything straight was quite a challenge. This one liner helped me find all the serials, and get the copies sorted out:

# for i in `ls /dev | grep sd | sed -e 's/[0-9]//' | sort -u`; do echo -n "/dev/$i " ; smartctl --all /dev/$i | grep "Serial number" | awk '{print " --- " $3}'; done;
/dev/sdaa  --- 9VS0R3T4
/dev/sdab  --- 9VS3SSJE
/dev/sdac  --- 9VS2K1DY
/dev/sdad  --- 9VS07FMW
/dev/sdae  --- 9VS402AA
/dev/sdaf  --- 9VS3V74N
/dev/sdag  --- 9VS388JX
/dev/sdah  --- 9VS2AQ9A
/dev/sdai  --- 9VS3THY9
/dev/sdaj  --- 9VS4EE3T
/dev/sdak  --- 9VS0FXWA
/dev/sdal  --- 9VS34DAD
/dev/sdam  --- 9VS45C9J
/dev/sdan  --- 9VS4L45S
/dev/sdao  --- 9VS2K9K1
/dev/sdap  --- 9VS2D631
/dev/sdaq  --- 9VS2L1BK
/dev/sdar  --- 9VS4DYDB
/dev/sdas  --- 9VS3T33R
/dev/sdat  --- 9VS3YE1K
/dev/sdc  --- 9VS1HWAM
/dev/sdd  --- 9VS1J9F3
/dev/sde  --- 9VS1L0FY
/dev/sdf  --- 9VS1H3RB
/dev/sdg  --- 9VS1JNPW
/dev/sdh  --- 9VS1GWGK
/dev/sdi  --- 9VS1DLZZ
/dev/sdj  --- 9VS1FSRD
/dev/sdk  --- 9VS3A8GZ
/dev/sdl  --- 9VS1L8ZZ
/dev/sdm  --- 9VS1JE7E
/dev/sdn  --- 9VS1CHE1
/dev/sdo  --- 9VS295R5
/dev/sdp  --- 9VS1HR8P
/dev/sdq  --- 9VS1EJQW
/dev/sdr  --- 9VS1A4V5
/dev/sds  --- 9VS1JGP8
/dev/sdt  --- 9VS1HPGB
/dev/sdu  --- 9VS1JAWZ
/dev/sdv  --- 9VS1JG8K
/dev/sdw  --- 9VS1JA51
/dev/sdx  --- WD-WMATV4441330
/dev/sdy  --- 9VS38GT3
/dev/sdz  --- 9VS4SQJX
#

troubleshoot UDP connectivity

Posted on by
1

When troubleshooting network connectivity issues the best method is to begin with the lower layers of the OSI model and work your way up until you eliminate the problem. It’s similar to the way some people troubleshoot helpdesk issues, where the first question is always “is it plugged in?”

Suppose you have 2 hosts, and you are trying to do an snmp query from one to the other. It doesn’t work. 

# snmpwalk -v2c -c public 10.210.0.142 
Timeout: No Response from 10.210.0.142

You try pinging one from the other and you get replies. At this point you know that everything is working through layer 3, but something is not right with the transport layer. If you were troubleshooting a TCP based application, you could just use ol’ telnet to test if the port will open up. To see if a webserver is running you could try to telnet to port 80. UDP is a connectionless protocol, and can’t be tested like TCP. Here is where you can use nmap to see if the UDP port in question is open. In this case we know that by default snmpd listens on UDP 161.

# nmap -sU -v -p161 10.210.0.142

Starting Nmap 5.00 ( http://nmap.org ) at 2013-04-24 11:27 MST
NSE: Loaded 0 scripts for scanning.
Initiating ARP Ping Scan at 11:27
Scanning 10.210.0.142 [1 port]
Completed ARP Ping Scan at 11:27, 0.03s elapsed (1 total hosts)
Initiating UDP Scan at 11:27
Scanning example.fordodone.com (10.210.0.142) [1 port]
Completed UDP Scan at 11:27, 0.07s elapsed (1 total ports)
Host example.fordodone.com (10.210.0.142) is up (0.00100s latency).
Interesting ports on example.fordodone.com (10.210.0.142):
PORT    STATE  SERVICE
161/udp closed snmp
MAC Address: 00:0C:29:A2:BE:4D (VMware)

Read data files from: /usr/share/nmap
Nmap done: 1 IP address (1 host up) scanned in 0.43 seconds
           Raw packets sent: 2 (70B) | Rcvd: 2 (98B)

Well, there you go. The port is closed. No wonder our queries were timing out. Remove the -p161 part to just scan for all open UDP ports. Also, if your host doesn’t allow icmp, you can use -P0 to just skip host discovery, and assume it’s up. Depending whether they are open, filtered, or closed, you can start to diagnose what’s going on, i.e. snmpd crashed, firewall is blocking it, etc.