User Tools

Site Tools


technical_logs:st._lawrence_market

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
technical_logs:st._lawrence_market [2012/06/04 16:37]
gabe
technical_logs:st._lawrence_market [2014/10/01 19:16] (current)
gabe
Line 1: Line 1:
 +====== Oct 1, 2014 ======
 +Cloudtrax reported 3 nodes down last Monday, and Nesrin emailed reporting an outage. She wrote that she tried using the doodad to powercycle it, but no go. I went by this morning and it seems very much like the Scheffler'​s router is unplugged again. Makes sense -- someone was doing work up there on Monday and either did it on purpose or unknowingly jostled it. I didn't have time this morning to find a ladder to go up and check it out.
 +
 +Went back in the evening: yeah, it was unplugged. Back up now.
 +
 +====== Jun 13, 2014 ======
 +The network from .76 (downstairs north) and downstream went offline at some point this week. Similar problem to last time, which didn't really make sense to me. Nesrin tried powercycling it, but it didn't do anything. I finally had time to check it out today. When I got on the ladder, the router was off. The DC power plug wasn't all the way in. It seems like the router slid down the post it's attached to, and because the power cord has little slack, it got unplugged. (Unplugged by gravity!)
 +
 +I slid the router back up the post, and wedged it above a pipe. If no one touches it, it'll stay in place. And plugged in.
 +
 +====== Apr 25, 2014 ======
 +Nothing'​s changed. I connected to cd:e4, and could ping the 3 south routers, but high packet loss. I SSHed into schefflers to reboot it, and 
 +packet loss went down (though ping times were still highly variable).
 +
 +Connecting to 4d:a5 (downstairs-north),​ I can ping the router, but nothing else. I sshed in and rebooted it; no change.
 +
 +I got a ladder and plugged my laptop into the ethernet coming from the gateway router. I could ping it fine. 
 +I powercycled the downstairs-north router, and everything came back up.
 +Argh! I don't know why it's working now. Was it a loose cable? Why did it start working when I physically unplugged and plugged, but didn't work when I rebooted from the commandline?​
 +
 +I dunno, but it's working now.
 +
 +
 +====== Apr 24, 2014 ======
 +The problem is that everything'​s showing as down besides the gateway router.
 +The gateway router is up and looks fine. It's got an Ethernet link to the downstairs-north router.
 +I can connect by wifi to the downstairs-north router, and SSH in. I couldn'​t ping anything else from it.
 +I rebooted it. No change.
 +
 +I can connect to the wifi on the 0:​23:​69:​b3:​cd:​e4 router.
 +I can ping mustachios, schefflers and downstairs-south,​ but the ping times are very long: 1-3 seconds.
 +I can't SSH into the mustachios router: the connection is rejected or times out.
 +I can ssh into the schefflers router. When I ping the mustachios router from there, the pingtimes are good.
 +
 +I rebooted the Schefflers router and pingtimes suddenly got better. I guess that's what I was connected to. 
 +(Is the wifi on the mustachios router down?)
 +
 +When I ssh into schefflers, and ssh into mustachios from there, I can get in. 
 +I guess the mustachios router is configured to not allow ssh from wifi.
 +
 +Regardless: there'​s an obvious connectivity issue between schefflers and downstairs-north.
 +AND between downstairs-north and gateway.
 +
 +
 +====== March 21, 2014 ======
 +Nesrin emailed a few days ago to say that the south part of the network was down. I checked remotely, and it seemed like the 86 router was offline. I finally had time today to go by. The router had been unplugged. It also suffered some trauma: one of the antennas is missing, connector and all. I plugged it back in and everything came back up. I got a stepladder by knocking on the door of the security office. The nice fellow there didn't know where to find one; I suggested the boiler room. There was, indeed, one there.
 +
 +====== February 22, 2014 ======
 +Nesrin emailed to say that the wifi wasn't working. I checked this morning on cloudtrax and everything looked fine. When I checked cloudtrax again this afternoon, the whole network was reporting as down. I went by, could connect to the network but couldn'​t get an ip address. The door to the kitchen was open, so I restarted the Linksys and Bell routers. Soon after, the 3 north routers were up. I powercycled the .86 router, but that didn't help. I had to powercycle both the downstairs south routers to get them to come back up. (To get a ladder, I talked to a security guard, who put me in touch with a maintenance guy, Rocco.)
 +
 +====== August 9, 2013 ======
 +I got an email from Nesrin today saying that she's tried everything she can, and the network'​s still down. Cloudtrax reports that it's been down for 8 days. 
 +
 +Upon arrival, I could see the wifi network, could connect and get an IP, but not get online. I plugged into the ethernet on the gateway router, but couldn'​t get online. I sshed into the router, and it didn't have a DHCP address from the Bell modem/​router. Weirdly, the gateway router was reporting only 8 minutes uptime. ​
 +
 +I plugged my laptop'​s ethernet into the Bell modem/​router,​ and didn't get a DHCP address. I powercycled the Bell modem/​router,​ and got a DHCP address on my laptop. I plugged the ethernet back into the router and powercycled it, and everything came back up.
 +
 +I'll stick around for a bit to keep an eye on the gateway router.
 +
 +====== July 5, 2013 ======
 +The whole network had been reporting as down in Cloudtrax. I was able to connect to all the routers remotely, and I traced the problem to that the Cloudtrax URL that they were using had changed. (Previously they were connecting to checkin.open-mesh.com;​ it's not checkin.cloudtrax.com.) I edited /​etc/​crontab,​ then restarted cron -- on each router. They'​re now checking in. (Though the mustachios router still isn't because it needs to be reflashed/​replaced.)
 +
 +====== December 18, 2012 ======
 +Cloudtrax reported that the -whole- SLM network went down sometime on Saturday. I went by today, straight to the Market Kitchen. I rebooted the gateway router, and everything came back up. I also hard-coded the opendns addresses into the gateway router. It's back up now.
 +
 +====== December 12, 2012 ======
 +Cloudtrax reported that the bottom half of SLM went down on Monday, in that same way that it often does. Yesterday Nesrin emailed to say that she tried rebooting it using the weird switch, but that it didn't do anything. This morning I went by, and found the router unplugged -- the power cable had been pulled out on the router side. My guess is that the adjacent Christmas decorations had been installed on Monday, and in the process someone accidentally unplugged the router. It's back up now.
 +
 +====== November 15, 2012 ======
 +On Tuesday (13th) afternoon, Cloudtrax reported that the schefflers router (86) went down. I went by this morning, powercycled it using the switch, and it came right back up.
 +
 +I also created a simplified how-to on resetting the schefflers router, and shared it with SLM tenants, so that they can do the reset themselves in the future. http://​wiki.wirelesstoronto.ca/​fixslm
 +
 +
 +====== November 3, 2012 ======
 +Cloudtrax reported a few days ago that the south three routers were down (a common occurrence). I went by this morning and restarted the Schefflers router using my hacky switch. The routers came right back up. (We still need to replace the Mustachios one, tho...)
 +
 +====== August 20, 2012 ======
 +I went by today to try to fix the problem. I went first to the mustachios router; the link light on the port connecting it to the schefflers router was on. I plugged that ethernet into my laptop, assigned a static IP address, and I couldn'​t ping. I went to the schefflers side, plugged the ethernet into my laptop, and I *could* ping the downstairs routers. I rebooted the schefflers router, and everything seemed to start working again. All it needed was a powercycle!?​ I should'​ve tried that last time I was here (since we can now do it from the ground!) I don't understand why the router in this location is always flaking out in different ways -- even having replaced the router *and* the powersupply,​ the router in this position is always flaky.
 +
 +====== August 7, 2012 ======
 +Cloudtrax reported a few days ago that the downstairs south router stopped checking in. I went by this morning, and my laptop was able to connect to the downstairs south router, using a static IP address. I was able to ping both the downstairs south router (.40) and the mustachios router (.10), but not the others. So the problem is the line running from the schefflers router to the mustachios router. Troubleshooting that (checking the link lights, and running a cable test) is best left for after-hours. ​
 +
 +====== July 17, 2012 ======
 +Cloudtrax reported that all routers went offline just before 8am on July 7th. Why won't this network stay up? I went to the kitchen today -- the ethernet between the gateway router and the modem was unplugged. I plugged it back in and everything came up. 
 +
 +
 +====== July 6, 2012 ======
 +I went by today to swap in a new powersupply,​ and install a remote switch which we can use to reboot the schefflers router without climbing up a ladder. (Olympic Cheese'​s handy red-topped stepladder is now being stored in the little space that the sliding gate folds into during the day.) To reset the power to this router from the ground: on top of the heater near the ceiling on the south wall just as before the outside doors there'​s a phone jack, with a wire looped back into itself. Unplug the wire to cut the power to the router. Plug it back in to restore power.
 +
 +After turning the schefflers router back on, the south part of the building can now get online. However, I can't ping the mustachios router. (The downstairs south router -- which is beyond it -- *is* online, so clearly the mustachios router isn't out entirely.)
 +
 ====== June 4, 2012 ====== ====== June 4, 2012 ======
 The south half of the market is down again. To recap the problem: Once we got the network up again (on May 23), we were having the occasional problem of the router near schefflers locking up, and no longer passing traffic. Consequently,​ the mustachios and south-downstairs routers go offline too. A simple powercycle brings the router back up, but you've got to get on a ladder to get access to the router. The south half of the market is down again. To recap the problem: Once we got the network up again (on May 23), we were having the occasional problem of the router near schefflers locking up, and no longer passing traffic. Consequently,​ the mustachios and south-downstairs routers go offline too. A simple powercycle brings the router back up, but you've got to get on a ladder to get access to the router.
technical_logs/st._lawrence_market.1338845823.txt.gz ยท Last modified: 2013/09/28 16:06 (external edit)