Expect Ping/Reboot Script

Hello everyone. Long time no see.

We have some equipment at work that just seems to… stop working once in awhile. The fix from the manufacturer is “reboot it.” Lovely, right? I reached out to a helpful community to create a better solution. I asked a question on the Wisp Talk Facebook group a couple of weeks ago asking what the best way to monitor/reboot equipment via ping and telnet was.

We have some base stations that do what we call “traffic halts” where they will remain online but will not pass any traffic. These traffic halts happen randomly and seem to happen more frequently when it’s cold out. It’s a bad time of the year right now…

The BS has two IP addresses (one for data and one for management). When the traffic halts occur, the data IP address stops pinging but the management one is fine. Is there any software available that can detect if the data IP stops pinging, it can log in to the BS via SSH/Telnet and issue a reboot command?

A free solution is obviously preferred, but any suggestions would definitely help. Thank you all!

A few people suggested creating an Expect script. Since I never created an Expect script before, this was an adventure for me. After a few days of reading different information, here is what I created (however if you have any suggestions or improvements, please feel free to comment them below!) This script was created on a Raspberry Pi running Raspbian/Jessie 8.0.

First, create your file in your favorite text editor:

nano tower1.sh

Next, paste this script in and edit it as needed.

#!/usr/bin/expect
set timeout 30

spawn ping -c 5 10.10.xx.x

expect {
 "64 bytes from 10.10.xx.x" { send "^C" }
 "5 packets transmitted, 0 received, " {

spawn telnet 10.10.xx.y
expect "login: "
send "[user]\r"
expect "Password: "
send "[password]\r"
expect "node> "
send "en\r"
expect "node# "
send "reboot force\r"
expect "Do "
sleep 2
send "y\r"
send "exit\r"

spawn telnet [mail-server-ip] 25
sleep 2
expect "220 "
send "HELO [mail-user]\r"
sleep 2
expect "250 "
send "MAIL FROM:<[outages-email]>\r"
sleep 2
expect "250 "
send "RCPT TO:<[outages-email]>\r"
sleep 2
expect "250 "
send "DATA\r"
sleep 2
expect "354 "
send "Subject:Traffic Halt\r"
send "[Tower] had a traffic halt a few moments ago. We are rebooting it now and the customers should be back online in 5 minutes.\r"
send ".\r"
expect "250 "
sleep 2
send "quit\r"
expect "221 "
}
}

Save and close the file. You’ll need to change the following information:

  • If you’re familiar with Expect scripts, you’ll want to change the “expect” commands to work for your equipment.
  • 10.10.xx.x with the IP address you want to ping.
  • 10.10.xx.y with the IP address you want to Telnet/SSH in to.
  • [user]/[password] with your login credentials.

And that’s it for the reboot portion of the script.

If you would like to receive an email letting you know this script has ran, you will have to edit the second half of the script to have the following information:

  • Again, you’ll want to change the “expect” commands to work for your equipment.
  • [mail-server-ip] is the IP address of your mail server (if it supports sending email via telnet).
  • [mail-user] with the user you’d like to sign in as.
  • [outages-email] with the email you’d like to receive the email at.
  • [Tower] with the name of the tower you just rebooted.

After creating this script for as many pieces of equipment as you like, be sure to make the file executable by changing its permissions:

chmod 755 tower1.sh
chmod 755 tower2.sh
and so on...

Once these files are executable, you can create a cronjob to run the scripts every so often. To view/edit your cronjobs, type:

crontab -e

At the end of the file, add the following lines for each piece of equipment you would like to monitor:

*/10 * * * * /home/[server-user]/[script-directory]/tower1.sh
*/10 * * * * /home/[server-user]/[script-directory]/tower2.sh
and so on...

This command will allow the script to run every 10 minutes. We give it plenty of time as the equipment can sometimes take 3-8 minutes to come back online so it’s vital that it does not keep rebooting itself.

Overall, that’s the general idea of it. It works for our use, and it’s actually already come in handy!

traffic-halt.png

We have the emails feed into a Slack channel so everyone sees when it happens.

This script probably won’t be as easy as copying/pasting it and hoping it works, it will definitely need some tweaking, tuning, and TLC to fit your needs. But hopefully this can help someone out there as I had to start completely from scratch. If you have any questions, feel free to leave a comment below or email me at: jordan at jordohio dot net

Until next time.

Followup: Teach-N-Share in 2015

R.I.P. Teach-N-Share. It’s dead. Nobody used it.

R.I.P. T.N.S.

Ever since 2011, I had been working on Teach-N-Share off and on as a side project. Most of the teachers at my school did not care for it, although there were a few that always gave me hope. (Thanks, Mrs. Parkins, Mr. Heckler, and Mr. Acton!) Most of the people who walked by my makeshift booth at the eTech 2015 Conference didn’t care much for it. Most of the schools around my area, who were interested at one point, lost interest pretty quickly due to mandatory state testing. Thanks, Ohio.

As of today: Teach-N-Share is officially dead. The server that the forum software was running on kept crashing and freezing up. With every crash came more database corruptions. Some were fixable and some made the website run on its bare bones. After looking through the database, there’s too much corruption to attempt any sort of fix at this point.

And you know what? I’m okay with that. Teach-N-Share, just like any other project I’ve ever really worked on, needed to die. It seems to be a waste of my time at this point. I strongly believed this would be the year it would take off. Eh. Oh well.

Maybe it’ll return in the future if I hand the project off to one of my friends. Maybe I’ll get funding to actually deploy it to the state of Ohio. Maybe the world will never see it again. If I’ve learned anything over the last few months, it is to expect nothing. Things happen and change so quickly in my life.

The Teach-N-Share domain will eventually be used as a proof-of-concept website to show that free, easy-to-use teacher collaboration tools are needed in the education sector. For now it’ll be redirecting to Jordohio.net.

Until next time.

Teach-N-Share in 2015

Teach-N-Share Logo

Logo created by Dean Harvey

In late 2011, one of my prior teachers stopped me in the hallway and said, “I have an idea. Talk to me when you can.” I was so curious that I stopped him right there and we talked. For about ten minutes, my teacher had been discussing about a way for teachers to share ideas with one another – online. He wanted a way to upload his PowerPoint document so that another teacher from another school could view it, download it, suggest edits to it, and use it in his class.

“I call it… Teach And Share. What do you think?”

— Mr. Adam Heckler, 2011

I liked it.

I liked it so much that I’ve actually been working to try and get this “online sharing system” up and running since early 2012. I almost abandoned Teach-N-Share in late 2012/early 2013 because it seemed like people just weren’t interested. I let the domain just redirect to Jordohio.net for months. But then, at the start of the 2013/2014 school year, my tech coordinator approached me saying that someone had stolen my idea! They were trying to create their own version of Teach-N-Share to roll out to all of the school districts.

Nope, not today. This is my baby.

For the last year I’ve been tinkering with different forum software and Linux distros to figure out what works best. After a few hiccups (and one really big hiccup – more on that later), we’ve settled in with Simple Machines Forum 2.x running on Debian 7. I started deploying this out at the start of my 2014/2015 school year and we had a functioning server…for the most part. There were issues with pages not loading and, if you tried uploading a file, the web server crashed. (Fun Fact: Do NOT use Apache!)

Teach-N-Share Screenshot

Screenshot of Teach-N-Share

But then the fun really happened: The NWOET Conference. I had stayed up the night before preparing the website and a mailing list for people to stay updated. When I had arrived at school, the website didn’t load. The website, hosted in that same building, was not loading. Nothing worked. Long story short, it was an outside issue with our Internet Provider. The unfortunate part is that I made people think I didn’t know what I was doing. Everyone said, “It’s a good idea!” but nobody showed interest in it. Except one person. A very important person: My Superintendent.

To kind of sum everything up, she began showing this to other districts and they all loved the idea. They actually want it. So I rebuilt it from the ground up and am ready to deploy it to schools. Teach-N-Share is back, and we’re about to set the rockets on full throttle. I haven’t been this happy in a long time. Thank you to everyone who pushed me to finally create this, I owe you one.

Until next time.