Topics

Widespread Cell Phone Service Disruptions During Power Shut Downs

Alan Thompson
 

Widespread Cell Phone Service Disruptions During Power Shut Downs
 
Some stories I've been following over the last few days:
 
https://www.sfchronicle.com/california-wildfires/article/Without-cell-service-Bay-Area-fears-emergency-14572616.php
 
https://www.sfchronicle.com/business/article/Extremely-dangerous-Cell-outages-during-14546298.php
 
https://www.sfchronicle.com/california-wildfires/article/California-blackouts-Cell-service-improving-but-14576293.php
 
Alan - W6WN

Nathan Chilton - K6NDC
 

These widespread outages have provided an answer to the question of why some of us bother with ham radio, when cellular phones have made two-way radio "obsolete".  Now, many people have a recent experience where the cellular infrastructure has failed and they've been cut off.  I've had opportunities to explain how our community was able to maintain communications and relay information to/from people who had no other means of communication.

Jan Woldseth
 

Hey, Alan-

What really gets me is that our landlines went down within a couple of hours of the outage(s).  I would have thought them to be more reliable.  Cell phones in the Peardale/Chicago Park area of Nevada County remained operable. 

Jan
KB6FMZ


From: sfarc@w6ek.groups.io <sfarc@w6ek.groups.io> on behalf of Alan Thompson <alan@...>
Sent: Thursday, October 31, 2019 10:08 AM
To: sfarc@w6ek.groups.io <sfarc@w6ek.groups.io>
Subject: [from W6EK Groups.io] Widespread Cell Phone Service Disruptions During Power Shut Downs
 
Widespread Cell Phone Service Disruptions During Power Shut Downs
 
Some stories I've been following over the last few days:
 
https://www.sfchronicle.com/california-wildfires/article/Without-cell-service-Bay-Area-fears-emergency-14572616.php
 
https://www.sfchronicle.com/business/article/Extremely-dangerous-Cell-outages-during-14546298.php
 
https://www.sfchronicle.com/california-wildfires/article/California-blackouts-Cell-service-improving-but-14576293.php
 
Alan - W6WN

Greg D
 

Hi Jan,

Yeah, kind of makes you wonder sometimes.  I actually had the reverse happen.  For the past several years, every power outage I noticed no dial tone on our AT&T wired land-line phone, but the cell service (T-Mobile) seemed to stay up.  The first PSPS outage, where we went down, the land line was fine, but cells went down within 2-3 hours. 

Of course, we long ago gave up dial-up Internet in favor of Cable.  Assuming I could even find a modem, I'm not sure I remember what to do with it, or if anything would answer on the other end if I did.

Certainly pays to have two independent paths to get important services.  Sometimes you need three.

Greg  KO6TH


Jan Woldseth wrote:

Hey, Alan-

What really gets me is that our landlines went down within a couple of hours of the outage(s).  I would have thought them to be more reliable.  Cell phones in the Peardale/Chicago Park area of Nevada County remained operable. 

Jan
KB6FMZ


From: sfarc@w6ek.groups.io <sfarc@w6ek.groups.io> on behalf of Alan Thompson <alan@...>
Sent: Thursday, October 31, 2019 10:08 AM
To: sfarc@w6ek.groups.io <sfarc@w6ek.groups.io>
Subject: [from W6EK Groups.io] Widespread Cell Phone Service Disruptions During Power Shut Downs
 

Alan Thompson
 

Just goes to show that one of our commonly shared assumptions about the resiliency of land line service is being quietly undermined.  I'm convinced that AT&T is trying to quietly kill landline telephone service, and this is just one way they're doing this.

I believe they've adopted "benign neglect" as a policy. "Change out the backup batteries? Let's just ignore that for now."

The more people who become fed up with inferior service and disconnect their land line in favor of using just a cell phone, the happier AT&T is. Much cheaper to deliver phone service through a single-point-of-failure cell tower than 50 miles of aging copper wire.

Then, they can go back th the FCC and claim, " Consumers no longer want land line service. Release us from our obligation to provide it."

We hear it every week from people in El Dorado County who tried to order DSL or Land line service from AT&T and were told, "It's no longer available at your address."

Jan Woldseth
 

Actually, I think you have a point, and I was thinking the same thing.  I'm not the conspiracy-theory type, but there have been some things in the past few years that have made me wonder. 

I think there have been enough complaints about this that Nevada County put out an online form to fill for those who used landlines and had them fail with these outages.  There are lots of people out in the hills/valleys that don't get cell service, and end up communicatively stranded when landlines fail. 

"Benign Neglect" is well put. 


From: sfarc@w6ek.groups.io <sfarc@w6ek.groups.io> on behalf of Alan Thompson <alan@...>
Sent: Thursday, October 31, 2019 8:52 PM
To: Jan Woldseth <jwoldseth@...>; sfarc@w6ek.groups.io <sfarc@w6ek.groups.io>
Subject: Re: [from W6EK Groups.io] Widespread Cell Phone Service Disruptions During Power Shut Downs
 
Just goes to show that one of our commonly shared assumptions about the resiliency of land line service is being quietly undermined.  I'm convinced that AT&T is trying to quietly kill landline telephone service, and this is just one way they're doing this.

I believe they've adopted "benign neglect" as a policy. "Change out the backup batteries? Let's just ignore that for now."

The more people who become fed up with inferior service and disconnect their land line in favor of using just a cell phone, the happier AT&T is. Much cheaper to deliver phone service through a single-point-of-failure cell tower than 50 miles of aging copper wire.

Then, they can go back th the FCC and claim, " Consumers no longer want land line service. Release us from our obligation to provide it."

We hear it every week from people in El Dorado County who tried to order DSL or Land line service from AT&T and were told, "It's no longer available at your address."

Jim - N6MED
 

Jan and Alan --

Re the loss of dial tone for POTS service: Used to be "back in the day" when all subscriber connections were copper pairs back to the CO (central office), you would virtually always get dial tone when you took your phone off-hook. Exceptions were when a line break occurred (of course!) or a disaster of large enough magnitude that more than 5-7% of subscribers went off-hook at the same time (e.g., during the immediate aftermath of the '89 quake n S'Cruz County). Many folks might not be aware that dial tone service was built with the expectation that only 5-7% of subscribers would want to place a call at any one time.

More recently (past 15 years?), the local operating companies have begun using "remote terminals" to provide "last mile" copper service loops to subscribers, then multiplex those subscriber lines onto fiber for back haul to the CO. Whereas a copper pair (loop) all the way between a subscriber and the CO would always have "Battery" (-48V) power literally because the CO had a battery plant to power their equipment and lines (albeit that the batteries in the plant would be continually float charged until a public power utility went out.

I'm going to take a SWAG that remote terminals don't have Plant Battery supplied to them nor have power backup. So much for what was once virtually guaranteed service for POTS.

Something re cell service I learned from some AT&T guys upgrading a tower with 5G equipment near me: I noted that there were no back-haul antennas on the tower and puzzled how they connected with the MTSO (mobile telephone switching office). Seems they used, locally anyway, Comcrash fiber for back haul! So, it appears that, even though a cell sight is undamaged in a disaster but a fiber owner provides back haul, the loss of service might not be the fault of the cell provider.
Noteworthy from a one of the many programs about the Camp Fire, Verizon had something like 17 cell towers in Paradise, only one of which was damaged by the fire. But, Verizon service went down due to the loss of back haul over fiber.

-- Jim

Jef - N5JEF
 

For a contrasting data-point:

During the two of three PSPS events when our house power went down, our landline phone and DSL Internet remained available.  A 12V battery on the modem, and another 12V battery on our home router in another room kept us alive for the duration.-- With the decrease in cellular service, we might have starved otherwise.  ;-)

[This is the first time I've ever said anything relatively positive about our DSL service.]

- Jef  N5JEF



Brian Gohl - AI6US
 

I have an order with AT&T to reconnect the office to business DSL service on Monday. Slower than Suddenlink Cable Internet, but it has remained up during power outages here in Meadow Vista. 

Our AT&T cell/data remained active throughout the 4 PSPS events.

Best Regards! 
Brian Gohl - AI6US 
(916) 770-7751 cell

-------- Original message --------
From: Jef - N5JEF <jef@...>
Date: 11/1/19 8:30 AM (GMT-08:00)
To: Jim - N6MED <n6medjim@...>
Cc: Jan Woldseth <jwoldseth@...>, sfarc@w6ek.groups.io
Subject: Re: [from W6EK Groups.io] Widespread Cell Phone Service Disruptions During Power Shut Downs

For a contrasting data-point:

During the two of three PSPS events when our house power went down, our landline phone and DSL Internet remained available.  A 12V battery on the modem, and another 12V battery on our home router in another room kept us alive for the duration.-- With the decrease in cellular service, we might have starved otherwise.  ;-)

[This is the first time I've ever said anything relatively positive about our DSL service.]

- Jef  N5JEF




--
Brian- AI6US

Skip - K6DGW
 

I believe that "Benign Neglect" of their equipment on PG&E's part is what led to PSPS in the first place. [:-)

Alan Thompson
 

Here's another story:

https://www.mercurynews.com/2019/10/29/why-is-your-landline-phone-dead/

Bears out what Jim (N6MED) wrote about the reasons for the current, reduced reliability of POTS.

I hope the CPUC does something significant in January to force these guys to make their infrastructure more resilient: "OK, you guys want to eliminate POTS? Then make your Cell and IP Networks bulletproof."

Jim - N6MED
 

Gotta add, for me, a final anecdotal story: back in 2015 I was assigned to the Red Cross Shelter in the beautiful downtown cosmopolitan metropolis of Hayfork. It must have been thriving at one time enough to support a drug store, but it was closed and out of business when I was there.
Hayfork had a single provider for dial-up POTS, cell, and Internet with a single microwave back haul out of the small valley where Hayfork is to a relay vault on a nearby mountain top.
The fire took out the PG&E to the vault. The vault did (probably still does) have a propane-powered back up gen that ran just fine until it ran out of fuel. The telephone provider (Verizon) had to use a helicopter to bring in fuel to get the generator back on-line. That lasted until the gen was plastered with red stuff by a retardant drop.
Side-bar: Before telephone service was totally interrupted by the above, the Disaster Services Technology guy I tried to get interested in the local hams who were chomping at the bit to help out told me, as he flailed around with his Cat 5 cables, that he'd "call the hams when I need 'em."

Alan Thompson
 

I keep getting more stories sent to me. Here's another one:

https://www.sfchronicle.com/california-wildfires/article/Why-cell-phones-failed-in-PG-E-outages-and-how-14806460.php

It looks like this issue is finally going mainstream.

Jim - N6MED
 

WARNING, opinion ahead!

Cellular service itself has morphed a “convenience” technology into, by virtual of the ubiquitous demand and use of its subscribers, to a necessity where POTS was sold as a reliable service. (As discussed, POTS no longer reliable due to the changes in the last mile and long-haul fiber infrastructure.) From apparent evidence, carriers, seeing the immense profit in wireless communications, has been reluctant to continually maintain/improve their existing dial-up networks, AT&T and others move to “fiber to the curb” notwithstanding. Previously (and still currently?), carrier regional operating companies have been bound by public utility “tariffs” (law and regulations) that required minimums of service, which cell providers are bound by the FCC and the FTC, neither of which require any minimum level of service.

“Emergency service” (my words) has been mandated from the cellular providers in the way of 911 calls even if the phone was doesn’t have an active subscription to a carrier and E911. But the service comes at a price by way of a charge on subscribers’ bills, i.e., the cell companies don’t provide the emergency “service” for free.

For a service level that is becoming a demand as a result of recent fires, disaster notification broadcasts and subscriber calls TO emergency services are complicated by several factors:

Misunderstanding of the telephone subscriber at large, neither POTS nor the cellular network can possibly provide service to anything near its total number of subscribers. As I mentioned in a previous post, only about 5-7% of POTS subscribers going off-hook at the same time will get dial tone. If there is similar information available for cell subscribers, I’m guessing that the providers keep the information close to the breast. In my own experience when I used to commute from south San Jose, trying to get service at 5PM was a total loss.

I would SWAG it would take a lot of bucks (from us or the gummint) before cell providers would invest in infrastructure to provide the network capacity desired during a disaster.

As discussed in the SF Chron, I agree that it will likely take the push of a regulatory to get cell providers to harden their sites against fire and other disasters, whether local hardening or via network redundancy. But you know damned well the providers won’t do it without someone else paying for it. Then there’s the problem of those in local gummint planning committees who view the aesthetics of cell sites to be more important than disaster hardening.

‘Nuther opinion: In areas of relatively dense populations, regardless whether urban or on the wild land interfaces, folks are overly dependent on someone else being there to take care of them when the crap hits the fan.  I think that folks need to take personal responsibility to prepare for disasters – and that includes communications. I’m impressed with the organization Shingletown (near the Shasta Ham radio wildfire of a couple of weeks ago) has for alerting folks of issues via ham radio. But getting folks to organize to that degree in larger communities is akin to herding cats.

‘k. I’m oughta gas and done for now 😉

 

Jim - N6MED
 

One the subject: anyone have any idea whether or not Comcast has power back up to their head ends and repeaters? Question relates to expectations of continued service if public power goes out, but one has battery backup to their cable modem and VoIP interface.

Alan Thompson
 

Comcast upstream from the modems went down almost immediately with both of the local power shut offs in Placerville.

Fortuately, we also have HughesNet as a backup.

Brian Gohl - AI6US
 

Hello Jim - N6MED,

Cable providers require power insertion every 3-10 miles (depending on the design. Most cable companies have no back-up power and when a single power insertion point is lost, the daisy chain is broken and no signal is passed even though upstream insertion and amplifiers are working.

--
Best Regards!
Brian Gohl - AI6US
(916) 770-7751


-------------------------

On Mon, 04 Nov 2019 13:48:01 -0800, Jim - N6MED <n6medjim@...> wrote:

One the subject: anyone have any idea whether or not Comcast has power back up to their head ends and repeaters? Question relates to expectations of continued service if public power goes out, but one has battery backup to their cable modem and VoIP interface.




--
Brian- AI6US

Nathan Chilton - K6NDC
 

I don't know about Comcast, but I know that with Wave Broadband, someone (KO6TH?) tried powering up the cable modem while the grid power was down, and the modem was not able to connect to the Internet.  I have Wave, but I didn't even bother trying (I just had my phone share its connection to my other devices).

Greg D
 

Yep, that was me.  No carrier to the modem, so no chance to get to the next outage point farther upstream.

Interestingly, the 2nd and 3rd outages (where I did not lose power), cable and Internet remained up.  So, apparently the route of the cable happens to be along the (powered) Highway 49 corridor, and managed to cross the outage area at Luther Rd.  After enough of these outages, we should be able to map out where all the wires go based on what stays up and what goes down.

Greg  KO6TH


Nathan Chilton - K6NDC wrote:

I don't know about Comcast, but I know that with Wave Broadband, someone (KO6TH?) tried powering up the cable modem while the grid power was down, and the modem was not able to connect to the Internet.  I have Wave, but I didn't even bother trying (I just had my phone share its connection to my other devices).

Alan Thompson
 

Here's another one: 
 
From the LA Times: https://www.latimes.com/california/story/2019-11-05/mass-cellphone-outages-during-blackouts-and-fires-are-a-grim-preview-of-life-after-a-major-earthquake
 
I'm convinced the sudden attention is due to the coincidence of four key factors that have pushed this to the forefront:
 
    Our recent increase in catastrophic fires due to climate change
    More people than ever just using cell phones
    AT&T's largely clandestine efforts to kill/cost-cut copper wire land-line service
    PG&E's widespread power shut downs (likely the single, biggest factor that has pushed this to the forefront)
 
The problem won't get fixed at the National level as the FCC has been rendered essentially impotent by telecom and cable money.
 
It may eventually be mitigated at the state level but that will take some time. My wager is that local municipalities are where this will get addressed first through actions such as requiring cell phone service providers to harden their sites - especially during the approval process for new sites.
 
In the meantime, GMRS "Radio For The Rest of Us" is at least as an interim solution to keep the public safer until this is fixed. 
 
Alan