User Tools

Site Tools


technical_logs:st._lawrence_market

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
technical_logs:st._lawrence_market [2013/07/05 12:44]
admin
technical_logs:st._lawrence_market [2014/10/01 19:16] (current)
gabe
Line 1: Line 1:
 +====== Oct 1, 2014 ======
 +Cloudtrax reported 3 nodes down last Monday, and Nesrin emailed reporting an outage. She wrote that she tried using the doodad to powercycle it, but no go. I went by this morning and it seems very much like the Scheffler'​s router is unplugged again. Makes sense -- someone was doing work up there on Monday and either did it on purpose or unknowingly jostled it. I didn't have time this morning to find a ladder to go up and check it out.
 +
 +Went back in the evening: yeah, it was unplugged. Back up now.
 +
 +====== Jun 13, 2014 ======
 +The network from .76 (downstairs north) and downstream went offline at some point this week. Similar problem to last time, which didn't really make sense to me. Nesrin tried powercycling it, but it didn't do anything. I finally had time to check it out today. When I got on the ladder, the router was off. The DC power plug wasn't all the way in. It seems like the router slid down the post it's attached to, and because the power cord has little slack, it got unplugged. (Unplugged by gravity!)
 +
 +I slid the router back up the post, and wedged it above a pipe. If no one touches it, it'll stay in place. And plugged in.
 +
 +====== Apr 25, 2014 ======
 +Nothing'​s changed. I connected to cd:e4, and could ping the 3 south routers, but high packet loss. I SSHed into schefflers to reboot it, and 
 +packet loss went down (though ping times were still highly variable).
 +
 +Connecting to 4d:a5 (downstairs-north),​ I can ping the router, but nothing else. I sshed in and rebooted it; no change.
 +
 +I got a ladder and plugged my laptop into the ethernet coming from the gateway router. I could ping it fine. 
 +I powercycled the downstairs-north router, and everything came back up.
 +Argh! I don't know why it's working now. Was it a loose cable? Why did it start working when I physically unplugged and plugged, but didn't work when I rebooted from the commandline?​
 +
 +I dunno, but it's working now.
 +
 +
 +====== Apr 24, 2014 ======
 +The problem is that everything'​s showing as down besides the gateway router.
 +The gateway router is up and looks fine. It's got an Ethernet link to the downstairs-north router.
 +I can connect by wifi to the downstairs-north router, and SSH in. I couldn'​t ping anything else from it.
 +I rebooted it. No change.
 +
 +I can connect to the wifi on the 0:​23:​69:​b3:​cd:​e4 router.
 +I can ping mustachios, schefflers and downstairs-south,​ but the ping times are very long: 1-3 seconds.
 +I can't SSH into the mustachios router: the connection is rejected or times out.
 +I can ssh into the schefflers router. When I ping the mustachios router from there, the pingtimes are good.
 +
 +I rebooted the Schefflers router and pingtimes suddenly got better. I guess that's what I was connected to. 
 +(Is the wifi on the mustachios router down?)
 +
 +When I ssh into schefflers, and ssh into mustachios from there, I can get in. 
 +I guess the mustachios router is configured to not allow ssh from wifi.
 +
 +Regardless: there'​s an obvious connectivity issue between schefflers and downstairs-north.
 +AND between downstairs-north and gateway.
 +
 +
 +====== March 21, 2014 ======
 +Nesrin emailed a few days ago to say that the south part of the network was down. I checked remotely, and it seemed like the 86 router was offline. I finally had time today to go by. The router had been unplugged. It also suffered some trauma: one of the antennas is missing, connector and all. I plugged it back in and everything came back up. I got a stepladder by knocking on the door of the security office. The nice fellow there didn't know where to find one; I suggested the boiler room. There was, indeed, one there.
 +
 +====== February 22, 2014 ======
 +Nesrin emailed to say that the wifi wasn't working. I checked this morning on cloudtrax and everything looked fine. When I checked cloudtrax again this afternoon, the whole network was reporting as down. I went by, could connect to the network but couldn'​t get an ip address. The door to the kitchen was open, so I restarted the Linksys and Bell routers. Soon after, the 3 north routers were up. I powercycled the .86 router, but that didn't help. I had to powercycle both the downstairs south routers to get them to come back up. (To get a ladder, I talked to a security guard, who put me in touch with a maintenance guy, Rocco.)
 +
 +====== August 9, 2013 ======
 +I got an email from Nesrin today saying that she's tried everything she can, and the network'​s still down. Cloudtrax reports that it's been down for 8 days. 
 +
 +Upon arrival, I could see the wifi network, could connect and get an IP, but not get online. I plugged into the ethernet on the gateway router, but couldn'​t get online. I sshed into the router, and it didn't have a DHCP address from the Bell modem/​router. Weirdly, the gateway router was reporting only 8 minutes uptime. ​
 +
 +I plugged my laptop'​s ethernet into the Bell modem/​router,​ and didn't get a DHCP address. I powercycled the Bell modem/​router,​ and got a DHCP address on my laptop. I plugged the ethernet back into the router and powercycled it, and everything came back up.
 +
 +I'll stick around for a bit to keep an eye on the gateway router.
 +
 ====== July 5, 2013 ====== ====== July 5, 2013 ======
 The whole network had been reporting as down in Cloudtrax. I was able to connect to all the routers remotely, and I traced the problem to that the Cloudtrax URL that they were using had changed. (Previously they were connecting to checkin.open-mesh.com;​ it's not checkin.cloudtrax.com.) I edited /​etc/​crontab,​ then restarted cron -- on each router. They'​re now checking in. (Though the mustachios router still isn't because it needs to be reflashed/​replaced.) The whole network had been reporting as down in Cloudtrax. I was able to connect to all the routers remotely, and I traced the problem to that the Cloudtrax URL that they were using had changed. (Previously they were connecting to checkin.open-mesh.com;​ it's not checkin.cloudtrax.com.) I edited /​etc/​crontab,​ then restarted cron -- on each router. They'​re now checking in. (Though the mustachios router still isn't because it needs to be reflashed/​replaced.)
technical_logs/st._lawrence_market.1373046280.txt.gz ยท Last modified: 2013/09/28 16:06 (external edit)