Expect Ping/Reboot Script

Hello everyone. Long time no see.

We have some equipment at work that just seems to… stop working once in awhile. The fix from the manufacturer is “reboot it.” Lovely, right? I reached out to a helpful community to create a better solution. I asked a question on the Wisp Talk Facebook group a couple of weeks ago asking what the best way to monitor/reboot equipment via ping and telnet was.

We have some base stations that do what we call “traffic halts” where they will remain online but will not pass any traffic. These traffic halts happen randomly and seem to happen more frequently when it’s cold out. It’s a bad time of the year right now…

The BS has two IP addresses (one for data and one for management). When the traffic halts occur, the data IP address stops pinging but the management one is fine. Is there any software available that can detect if the data IP stops pinging, it can log in to the BS via SSH/Telnet and issue a reboot command?

A free solution is obviously preferred, but any suggestions would definitely help. Thank you all!

A few people suggested creating an Expect script. Since I never created an Expect script before, this was an adventure for me. After a few days of reading different information, here is what I created (however if you have any suggestions or improvements, please feel free to comment them below!) This script was created on a Raspberry Pi running Raspbian/Jessie 8.0.

First, create your file in your favorite text editor:

nano tower1.sh

Next, paste this script in and edit it as needed.

#!/usr/bin/expect
set timeout 30

spawn ping -c 5 10.10.xx.x

expect {
 "64 bytes from 10.10.xx.x" { send "^C" }
 "5 packets transmitted, 0 received, " {

spawn telnet 10.10.xx.y
expect "login: "
send "[user]\r"
expect "Password: "
send "[password]\r"
expect "node> "
send "en\r"
expect "node# "
send "reboot force\r"
expect "Do "
sleep 2
send "y\r"
send "exit\r"

spawn telnet [mail-server-ip] 25
sleep 2
expect "220 "
send "HELO [mail-user]\r"
sleep 2
expect "250 "
send "MAIL FROM:<[outages-email]>\r"
sleep 2
expect "250 "
send "RCPT TO:<[outages-email]>\r"
sleep 2
expect "250 "
send "DATA\r"
sleep 2
expect "354 "
send "Subject:Traffic Halt\r"
send "[Tower] had a traffic halt a few moments ago. We are rebooting it now and the customers should be back online in 5 minutes.\r"
send ".\r"
expect "250 "
sleep 2
send "quit\r"
expect "221 "
}
}

Save and close the file. You’ll need to change the following information:

  • If you’re familiar with Expect scripts, you’ll want to change the “expect” commands to work for your equipment.
  • 10.10.xx.x with the IP address you want to ping.
  • 10.10.xx.y with the IP address you want to Telnet/SSH in to.
  • [user]/[password] with your login credentials.

And that’s it for the reboot portion of the script.

If you would like to receive an email letting you know this script has ran, you will have to edit the second half of the script to have the following information:

  • Again, you’ll want to change the “expect” commands to work for your equipment.
  • [mail-server-ip] is the IP address of your mail server (if it supports sending email via telnet).
  • [mail-user] with the user you’d like to sign in as.
  • [outages-email] with the email you’d like to receive the email at.
  • [Tower] with the name of the tower you just rebooted.

After creating this script for as many pieces of equipment as you like, be sure to make the file executable by changing its permissions:

chmod 755 tower1.sh
chmod 755 tower2.sh
and so on...

Once these files are executable, you can create a cronjob to run the scripts every so often. To view/edit your cronjobs, type:

crontab -e

At the end of the file, add the following lines for each piece of equipment you would like to monitor:

*/10 * * * * /home/[server-user]/[script-directory]/tower1.sh
*/10 * * * * /home/[server-user]/[script-directory]/tower2.sh
and so on...

This command will allow the script to run every 10 minutes. We give it plenty of time as the equipment can sometimes take 3-8 minutes to come back online so it’s vital that it does not keep rebooting itself.

Overall, that’s the general idea of it. It works for our use, and it’s actually already come in handy!

traffic-halt.png

We have the emails feed into a Slack channel so everyone sees when it happens.

This script probably won’t be as easy as copying/pasting it and hoping it works, it will definitely need some tweaking, tuning, and TLC to fit your needs. But hopefully this can help someone out there as I had to start completely from scratch. If you have any questions, feel free to leave a comment below or email me at: jordan at jordohio dot net

Until next time.