UPDATE: 5-11-22 @ 0710: At this time, most of the DMR services are without connectivity. All MMDVM servers should be operational as are the Tri-Cities and Bremerton c-Bridges. Unknown issue and unknown when it will be resolved.
Views: 633
PNW-E and other Spokane Based Services Down
This Post Has 12 Comments
Comments are closed.
Dale,
Seems there is a break in the routing, I noticed it about 30 minutes ago and reported it. This is a different breakdown and on our side of the wire.
The MM’s are still working and use callwatch on D rather than E for now.
UPDATE: 5-26-22 @ 2200 or so, the BGP was fixed by Lumen. It took a few hours once once KC7AAD got access to the ticketing system and was able to “escalate” the ticket. It may take another day or so to get all the temporary changes removed that were added to enable PNW-E to limp along and a few repeaters as well for the 15 days.
Thanks to you Mike and Rod for your efforts!
And please don’t shoot the messenger here but, just out of curiosity, is anyone else back to not being able to bring up PNW-E or their local repeater?
I know PNW-E and Wenatchee were working yesterday and this morning. Now, neither one are working for me again.
Thought you might want to know if you hadn’t found it already, sorry!
As of today, the PNW-E c-Bridge and 3 Spokane repeaters are still offline…but the word is that CenturyStink (now Lumen) may in fact be getting closer to fixing the routing issue. So the temporary path that RoD put in 2 weeks ago is now removed. We are standing by… You may “ping pnw-e.pnwdigital.net” to see if the route has been restored.
UPDATE 5-22-22 @ 1114: Most of the Eastern Washington/Idaho repeaters have been off network since May 11th due to a CentruyLink failure to correctly route our 44 net IP blocks as part of their Level3 responsibilities. We did not envision that 12 days later it still would not be fixed. They are not communicating and we still have no resolution for the fix.
A few days into this mess, Rod KC7AAD took advantage of the down time and moved the PNW-E c-Bridge to a better Internet location and routed around the CT break to some degree. It brought the the Krells and Lookout repeaters back onto the network and allowed some of the out of network talkgroups to work again.
So I went ahead and fired up another c-Bridge to act as a mirror of PNW-E. We have a few repeaters now on PNW-S. Once the routing issue is fixed, the repeaters and the PNW-E c-Bridge will automatically come back online and likely will work better due to the location change.
The new PNW-S will stay in service though and become part of the network. It is available now if any repeater owners wish to switch to it now to get back on the network rather than wait for when CT fixes the issue. If interested please write to me off-list for more detail. If you move, you may move back or stay put, it is your choice.
The new c-Bridge PNW-S Callwatch is: pnw-s.pnwdigital.net:42420/CallWatch
5-20-22 @ 0720: Still down. Seems CenturyStink “lost the ticket”. New ticket went in yesterday, now 9 days after the original breakdown during their upgrade. I recommend that you guys buy stock in Lumen (latest owner of this POS company).
Still no corrective action from CenturyStink and no ETA.
KC7AAD is moving the c-Bridge to a better location today during this down time.
While it can’t fix our upstream problem, it will help to improve our connectivity and should provide us lower loss rates overall…once we are reconnected.
By way of background, CenturyStink shows no disruptions in their services but still show the planned upgrade on Tuesday. CenturyStink has a bit of history mucking up their BGP as well.
Hi Mike,
Thanks for this post. Any idea which repeaters might or might not work on the network?
I’ve been trying Wenatchee’s and I can’t see that it is functional as of yesterday.
Dale, Only Krells and Lookout Pt. Repeaters on D and A are fine, including Tri-Cities and Yakima. But nothing else on the East side that resides on PNW-E.
No Update as to when that Upstream provider will have this fixed.
UPDATE: 1641: Rod, KC7AAD did some manual re-routing and now has the c-Bridge on network with a few repeaters that are on the Spokane HamWAN via their WiFI. Connectivity between c-Bridges is restored.
But the upstream connection is still broken and not serving our 44 Net block…and still no ETA of that restoration.
This was posted to our IO Group this morning:
Around 2230 last night, some upgrades were made by the Internet provider upstream from out ISP. They killed our BGP routing and have taken down all our 44 net connectivity.
This issue has caused no flow of services out of Spokane at this time. This includes the PNW-E c-Bridge, most repeaters on WA eastside and Idaho. It includes and any talkgroups that flow into Spokane from Brandmeister and other DMR organizations.
The service ticket went in this morning around 0830. They have no ETA for restoration of our services.
This was reported earlier on the home page of the web site via Post, which is first point of notifications for PNWDigital. Updates will continue to be made there.
UPDATE: Upstream Internet provider did some updates and now they are no longer passing our BGP announcements for our 44 net blocks. Ticket is in, no ETA on them fixing the issue. It is out of our hands and not a failure of our network, for whatever that is worth. 🙂