Government Technology
By Bill Schrier: Making technology work for a city government.

FirstNet Public Safety Network - Lessons from the School of Hard Knocks

July 1, 2012 By Bill Schrier

Even Babies have Access to LTE, but First Responders Don'tFriday, June 29th was the second and final day of the National Governors’ Association sponsored meeting of chief information officers (CIOs), statewide interoperability coordinators (SWICs) and other government officials from 49 states and territories. We’re discussing the States’ role in building the new Nationwide Public Safety wireless Broadband Network (NPSBN). The First Responder Network Authority (FirstNet), an independent agency inside the federal government, will be responsible for the planning, procurement and spending up to $7 billion to create the network. But FirstNet won't be constituted until August, 2012.  (Note:  I blogged about the first day of this event in FirstNet: Cats and Dogs Living Together).

Friday we heard from Stacey Black of AT&T, Don Brittingham of Verizon and Rishi Bashkar of Motorola. Each of those companies have practical, on-the-ground, experience building these 4th generation, long-term-evolution (LTE) networks. And their advice: basically LTE is a "horse of a different color" from our traditional public safety voice land-mobile-radio (LMR) networks.

Here are some specifics they talked about based upon their experience:

  • The timing is urgent. AT&T, Verizon and Motorola all agreed that the core of FirstNet could be up and running in 12 months. But, because of Federal procurement laws, the requirement to allow States to review the plans, time to issue RFPs and a whole variety of other factors, it may be 5 to 7 years before the network carries traffic. In their words "that is a bad plan" – that timeframe is just too long to meet the need, which is urgent today. Yet just getting all the various governments and agencies to "sing from the same sheet of music" in support of FirstNet may take years - see my blog post from yesterday for more about that.
  • Users - that is public safety responders - will need to be judicious in their use of the network. If responders view a number of high speed video feeds while at an incident scene, the network at that scene could be brought to its knees. This fact also underscores the need for local management of the network – so that some human being closely affiliated with a city or a county can "tweak" the network during major incidents to make sure it does not become overloaded. Similarly, if 200 cops walk onto the street during shift change and all of them boot up their computers, the network in that small, concentrated, area will be overloaded and slow.
  • Stacey emphasized – and Don agreed – that the 4G LTE network will NOT replace the existing voice radio networks for the foreseeable future. The standards for public safety mission-critical voice over LTE don’t exist. Public safety voice networks require individual radios (aka "cell phones") to talk to each other, without the radio traffic going through a tower. No commercial network anywhere in the world does that. Indeed, "push to talk", where a dispatcher or individual officer pushes a button on a radio and then talks to a whole precinct of cops or a whole battalion of firefighters – does not exist on LTE networks. Governments must continue to invest in upgrading and expanding their existing voice networks.
  • Don explicitly stated that success will depend upon several critical factors, including collaboration between FirstNet, States and industry. We all will need to work together. Next, there has to be a viable business plan. He offered a great example: Verizon does not try to build its network in every last square inch of the country. In many places Verizon will offer its spectrum and access to its nationwide network and core switches, to local carriers in return for their build out of the network within their footprint. FirstNet could do the same – offer access to its 700 MHz public safety spectrum to a private carrier in return for a joint build-out (wisely Congress specifically authorized this arrangement).
  • Verizon and AT&T also had testbeds – pilot locations such as Seattle and Boston and Baltimore – where they tested their LTE networks before rolling them out nationwide. Similarly, Rishi emphasized the need for testbeds for FirstNet.
  • Rishi reminded us that, while the Spectrum Act allocated $7 billion for this network, and only $2 billion immediately, the Federal government had $328 million more available today in BTOP (Broadband Technology Opportunity Program) grants to seven jurisdictions to build these networks. Those build-outs are on hold until FirstNet is created, but could serve as valuable testbeds for the larger network’s construction.
  • Finally, everyone on the panel agreed that the "opt-in opt-out" decision for states is a red herring (and I apologize for all the animal analogies!). Even if a state "opts out" they are still commiting to build the network themselves inside their state. So essentially they’ve "opted in" to the nationwide interoperable network. My comment on that:  the ultimate "opt out" decision is left up to each individual city and county government, police agency and fire district and electric/water utility. If the per-user price for FirstNet is too high, or if local agencies simply don’t want to use FirstNet, they can "opt out" and either use a commercial service or build their own network (IF they can find spectrum).

All in all, great lessons from three partners who have "been there, done that, got the Tee Shirt" in constructing 4G wireless networks.

The question is - will FirstNet and its local, state and federal partners - heed those lessons?

Leave a comment

FirstNet - Cats and Dogs Living Together

June 28, 2012 By Bill Schrier

FirstNet - Public Safety WirelessI’m attending the “Preparing for Public Safety Broadband” workshop hosted by the National Governors’ Association outside of Washington DC today. We’re discussing the States’ (and cities and counties) role in constructing the Nationwide Public Safety wireless Broadband Network (NPSBN), authorized by Congress in February and funded with $7 billion from sale of spectrum. More background on the network is here.

This workshop has about 200 participants with 49 states are represented and quite a number of chief information officers, but also police chiefs, fire chiefs and coordinators of the more traditional statewide land-mobile radio networks used by responders.

Chuck Dowd, Deputy Chief of Communications for NYPD talked on a panel about how remarkable this is – that Mayors and Governors, police chiefs and fire chiefs, agree on the importance of this network. And they all worked together with the Obama Administration and Congress to get the Spectrum Act passed earlier this year.

But, in many senses, the most difficult part of constructing the NPSBN is still ahead. Mistrust between government agencies and functions are historically rampant. The budget crisis of the Great Recession has exacerbated his mistrust, as every agency’s budget has been squeezed.

States don’t trust City and County governments, who may have only their own individual interests in mind. Rural areas don’t trust urban areas. Departments within State governments don’t trust each other – every department often has its own computer servers and applications and even email systems. Cities and counties, in turn, mistrust their States who, they feel, are always trying to take money and dictate unfunded mandates.

Individual government departments have often built their own networks for communications. Transportation departments have their fiber and wireless voice networks separate from State police separate from Natural Resources and often separate from the State’s information technology departments. And in most states, cities, counties and local agencies have usually built their own voice networks for public safety separate from the State’s. Often local agencies even have multiple networks on their own – transit and transportation and public safety and public works. In some locations, even police and fire departments have separate networks because of historical animosities between those responders. Agencies mistrust each other for many reasons – utilities resent that public safety has priority access to spectrum and higher priority on joint networks. Information technology departments are often seen as the “department of NO”, with high rates and poor customer service – “call the help desk and open a ticket”.

And everyone in State and local agencies mistrusts the Federal government. There is some fear FirstNet will build a network with minimum consultation, in a top down fashion.

Everyone at this conference agrees: if this mistrust continues, the NPSBN is doomed. CIOs must recognize the progress made by statewide interoperability coordinators (SWICs) to advance coordination of communications networks. SWICs and traditional “radio guys” need to recognize the expertise of CIOs in building and operating large-scale infrastructure and supporting all responders with technology.

Cops and firefighters recognize their co-dependence, especially after the disaster of 9-11 where police received the order to evacuate but firefighters did not. Now public safety officials must acknowledge the responder role of electric and water utilities, transportation and transit. It’s hard to fight fires without water; it’s hard to evacuate before hurricanes without transportation.

State governments need to recognize that most 911 calls are handled by cities and counties. But cities and counties must acknowledge that they need resources from across a state to respond to even small disasters like a windstorm or riot.

The federal Department of Commerce needs to understand FirstNet cannot build this network without public-public partnerships - cities, counties, states and federal agencies, all who bring assets such as radio sites and fiber networks to reduce the cost of construction. And all the government officials need to acknowledge we will need private-public partnerships with telecommunications companies and other private companies to use their sites and build the applications to make the network a reality.

We need to recognize where the dollars are coming from – they are not coming from the Federal government, but, rather, from private industry buying spectrum and consumers and businesses who buy cell phone service. These funds are “our” funds, not “theirs”.

Finally, everyone involved needs to recognize the good intentions and leadership of the National Telecommunications Information Administration (NTIA), charged to stand up FirstNet. They’ve said they will collaborate with local and state governments. Let’s take them at their word and move forward together.

Yes, cats and dogs will be living together. And, amazingly, they might build a true nationwide interoperable wireless network for all responders.

Leave a comment

Does Walla Walla (or any City) Need its own Domain Name?

June 15, 2012 By Bill Schrier

Walla Walla's Unique Product - OnionsThis week ICANN announced they had received 1930 applications for new top-level Internet domains.

ICANN is the Internet Corporation for Assigned Names and Numbers, usually pronounced “I can” and with no direct relationship to “Yes We Can”.

Today the Internet has just 22 such domains, including the familiar dot-com and dot-gov (which we all know and love). ICANN knew more were needed, especially non-English ones and some using non-Latin characters. So, for a mere $185,000 each, anyone was invited to apply for new domains, with no guarantee they’d be granted. The full list is here, and includes a number of familiar and quite a few unique proposals.

For city names, ICANN rules specified the City government had to at least acquiesce to the application. There are just a few proposals for City names, such as dot-NYC, dot-Boston, dot-Miami and dot-Vegas, plus some overseas ones such as dot-Paris (which could, I suppose, be contested by Paris, Kentucky). And only one of the United States applications appears to be from an official City government, dot-NYC.  See more details on this at Government Technology news here.

Do cities really need their own domain names?

The City of Seattle was approached by at least one company seeking to apply for dot-Seattle. Their proposal was, basically, to put up the cash for the application, and then manage the sale of the names, presumably to individuals and companies who wanted the brand, such as microsoft.seattle or schrier.seattle . The City would receive a portion of any income beyond the cost to administer the domain name.

We didn’t pursue the opportunity for several reasons. Chiefly, I didn’t see how anyone would want to type microsoft.seattle when was shorter and easier. How much would Microsoft pay for that domain?

This logic would definitely be true for individuals or small business as well. Certainly some businesses might want a dot-seattle brand, but would there be $185,000 of such sales?

Furthermore, in order to pursue this, the City of Seattle really would have to issue an RFP and give other companies an opportunity to manage dot-seattle for us.

Hey, RFP’s are a lot of work.

Do Cities need their own domain names? Well, even at $185,000 plus management costs, dot-NYC makes sense. Maybe dot-Vegas will work. Dot-minneapolis or dot-wallawalla ? Naw, I don’t think so.

Leave a comment

Overpromise and Underdeliver

March 18, 2012 By Bill Schrier

The project mantra is clear: "scope, schedule, budget". But how we actually do the planning, estimating and getting approval to start a project … well that's the horse of a different color.

We promise the moon - "Project Widget will be the best thing for this department since sliced bread - it not only will slice bread, but will knead the dough and grow the yeast and self-bake itself". Then, of course, instead of delivering sliced bread we might end up delivering half-a-loaf, or maybe an electric knife or perhaps a chopped salad. This problem: getting the project’s scope right.

Then there is schedule. Of course every project is a "priority". We're going to get it done in the "next nine months". Why "nine” months? Because that's less than a calendar and budget year, but it is longer than saying it will be done tomorrow, which is patently ludicrous. But nine months is also ludicrous for anything other than incubating a baby - and even babies usually take years of planning and preparation. Furthermore, in the public sector almost every procurement has to be done by RFP, and preparing a request for proposals alone, plus contract negotiations with a successful vendor, cannot be done in less than a year. And the schedule needs to include minor components such as business process discovery and the work of executing on the project.

Then there's budget. Generally we'll make a pretty good estimate of the actual real cost of the project. The usual mistake is for someone (fill-in-the-blank - "department director", "Mayor", "county commissioner", "state legislator", "grand phooba") to say "we only have x number of dollars". So, as the next step, the project budget shrinks to the magic budget number, while scope and schedule are left unchanged. And generally the "magic budget number" is determined by some highly scientific means such as the amount of money left over in a department budget at the end of a fiscal year, or the amount of money the City of Podunk Center spent on a similar project, or the size of a property tax increase which voters might be reasonably persuaded to pass.

Why do we plan projects this way in the public sector?

First, we are largely transparent and accountable in government. That’s really good news, because we – government – are stewards of taxpayer and ratepayer money. Oh, I suppose we can hide some small boondoggles, but there are too many whisteblowers and too much media scrutiny to hide a major failure. That's not true in the private sector, where projects costing tens or hundreds of millions of dollars are failures or near failures, often hidden from public or shareholder view, with wide-ranging and sometimes near catastrophic economic effects. Some public examples include Boeing's 787 Dreamliner or the Microsoft Courier tablet (gee, will anyone every produce a Windows tablet?) The federal government’s project failuresare paramount examples of both poor project planning/execution and admirable transparency with an eye to reform.

Here are my top reasons for project mis-estimation:

  • Lack of rigorous project management. Project managers are crucial. But good project managers are also expensive. Government doesn’t grow or pay PMs well. Too often we assign project managers as the "last man standing" - whoever is left over when everyone else is doing the work. And we are woefully short on training - usually training and education are the first things cut from public budgets during recessions.
  • Eagerness to please. Everyone in a project is eager to please a boss – the County Executive, the Governor, a legislator, a department director. How often do we invoke “the Governor is really interested in … (fill in the blank related to the current project)”. Projects need to stand on their own for business value, as well as be of interest to the elected official presently in office.
  • Jadedness. Knowing the budget process described above, we'll often pad estimates - make the budget larger and the schedule longer, knowing they will be cut. Then, of course, decision-makers and leaders can also play that game, figuring there is padding, and therefore cut all the deeper.
  • The constraints of the budget and election cycles. Typical budget cycles in government are one year. Election cycles can be two years, and at most four years. Unless elected officials and department directors really take a long-range view, these facts lead to short-range think and results, just as stock price and quarterly profits drive the private sector.

And here are my top cures:

  • Hiring professional project managers. Frankly, this means, for large projects, we should usually hire professional PMs or firms from outside government. As a side benefit, such outside firms can also help train, mentor and grow government employees so they become good PMs.
  • Good executive sponsorship. The executive sponsor for a project needs to be the government official with “skin in the game” – the owner of the “business”, whose job may be on the line if the project fails, as well as the official owning the business. A smartgrid project’s sponsor will be the electrical utility’s director of distribution and generation networks. A computer-aided dispatch system’s sponsor will be the Assistant Fire or Police Chief in charge of the 911 center and dispatch.
  • External quality assurance. QA is essentially a “watchdog” on projects, speaking truth to power, and highlighting areas of risk and opportunities for improvement as the project proceeds.
  • Small projects and quick wins. If possible, any large project should really be a series of small projects with quick wins. In the case of a computer-aided dispatch system, for example, the smaller projects could include installing computers in police vehicles, implementing automated vehicle location, implementing a records management system, and implementing the CAD software itself. Of course there is no way to build a sewage treatment plant serving a city of half-a-million as a series of small projects, but most IT projects can be decomposed.
  • Transparency. Perhaps the most important component is openness and honesty for everyone involved – project managers being honest with sponsors, technical staff being honest about their workloads, department directors looking at their portfolio of projects and putting the lower priority ones on hold so the higher priority ones have the resources for success. The federal government leads the way in transparency, with its public “dashboard” for information technology spending and projects.

What's amazing is that, despite everything I've said above, we get an amazing amount of great projects completed. At the City of Seattle, we’ve tracked all our major projects. Since 2006, we’ve tracked 77 project through 2,071 project dashboard reports. We’ve found that, when they are completed, 75% of them are within budget. Of those 77 projects, 32% have been on time and 57% have delivered the scope they promised (i.e. a whole loaf of sliced bread). Clearly this record reflects our priorities – budget is the most important consideration, with scope second, and schedule lowest.

Not a bad record when compared with Standish group project failure statistics, but plenty of room for improvement.

Leave a comment

Tech Lessons from the Seattle Snowstorm

January 21, 2012 By Bill Schrier

Seattle EOC ActivatedThe Seattle area and I just went through a four day snow/ice storm event. The City of Seattle’s emergency operations center (EOC) was activated and coordinated the City government’s response. That response received high marks from the public and media for a variety of reasons (see Seattle Times editorial here), including the leadership of Mayor Michael McGinn.

I was able to personally observe that response and lead the technology support of it. Information technology materially contributed to the improved response, nevertheless I see a number of further potential enhancements using technology . And that’s the purpose of this blog entry.

GIS GIS GIS (Maps) Every city, county and state is all about geography and maps. Maps are the way we deploy resources (think “snowplows”). Maps are the way we understand what’s happening in our jurisdiction.

Everyone who has lived and traveled inside a city can look on a map and instantly visualize locations - what the “West Seattle bridge” or any other street, infrastructure or geographical feature (think “hill”) looks like.

Winter Response (Snowplow) MapFor this storm, we have some great mapping tools in place, especially a map which showed which streets had been recently plowed and de-iced. This map used GPS technology attached to the snowplow trucks. That same map had links to over 162 real-time traffic cameras so people could see the street conditions and traffic. (Other cities, like Chicago, have similar maps.)

SCL Outage MapAnother useful map is the electrical utility’s system status map, which shows the exact locations of electrical system outages, the number of outages, the number of customers affected and the estimated restoration times. This is really useful if you are a customer who is affected – at least you know we’ve received your problem and a crew will be on the way.

What could we do better? We could put GPS on every City government vehicle and with every City crew and display all that information on a map. That way we’d immediately know the location of all our resources. If there was a significant problem – let’s say a downed tree blocking a road or trapping people – we could immediately dispatch the closest resources. In that case we’d typically dispatch a transportation department tree-clearing crew. But that crew might have to travel across the City when a parks department crew with the proper equipment might be a block away.

This same sort of map could show a variety of other information – the location of police and fire units, which streets are closed due to steep hills and ice, where flooding is occurring, blocked storm drains, as well as water system and electrical outages.

This “common operating picture”, across departments, would be enormously useful – as just one example, the fire department needs water to fight fires, and it needs good routes to get its apparatus to the fire and perhaps it would need a snowplow to clear a street as well.

Obviously we wouldn’t want to show all of this information to the public – criminals would have a field day if they knew the location of police units! But a filtered view certainly could be presented to show the City government in action.

Perceptions and Citizen Contact

A lot of media descended on Seattle this week. Partly that was due to the uniqueness of the storm – it doesn’t snow much in this City. And perhaps it was a slow news week in the world. A lot of news crews filmed inside the EOC. The Mayor and other key department spokespeople were readily available with information. This is quite important – the television, radio and print/blog media are really important in advising the public on actions they should take (“public transit to commute today, don’t drive”) and actions they should avoid (“don’t use a charcoal grill to cook when you are without power”). Our joint information center (JIC) was a great success.

Mayor McGinn’s family even contributed to this – his 11 year old son filmed him in a public service announcement about how to clear a storm drain of snow and ice which is now posted on the Seattle Channel.

What could we do better? We need better video conferencing technology, so the Mayor and senior leaders can be reached quickly by news media without sending a crew to the EOC. This video conferencing would also be quite useful in coordinating action plans between departments with leaders in different locations. In a larger, regional, disaster, such capability would allow the governor, mayors and county executives to rapidly and easily talk to each other to coordinate their work. It is much easier for anyone to communicate if they can see the visual cues of others on the call.

Also, Seattle, like many cities, is a place of many languages and nationalities. We need to have translators available to get communications out in the languages our residents speak. This might include a volunteer-staffed translation team but at least could include recording and rapidly distributing written, video and audio/radio public service announcements in multiple languages.

Commuting and Telecommutnig 

In these emergencies, many people elect to use public transit – buses and trains for commuting. (I actually took my “boat” – the water taxi - to work twice this week.) Yet snowstorms are also the times when buses jackknife or get stuck in snowdrifts and going up hills. In this emergency, the coordination between the transit agency (“Metro”) and the City was quite improved, because we had people – liaisons – from each agency embedded with the other. This allowed snowplows to help keep bus routes clear and help clear streets near trapped buses.

And, with recent technology advances and sorta-broadband networks, many workers can now telecommute. Seattle had few outages of Internet service this week, although in suburban areas trees and snow brought down not just power lines, but telephone and cable lines as well causing more widespread Internet issues.

What could we do better? The easiest and most useful advance, I think, would be GPS on every bus and train and water taxi boat. That, combined with real-time mapping, would allow people to see the location of their rides right on their smartphones. If we deployed it right, such technology might also show how full the bus is and the locations of stuck buses. This sort of technology would be useful every day for public transit users – but is especially important during snow emergencies.

Another huge necessity – which I’ve advocated often and loudly – is very high speed fiber broadband networks. With fiber broadband – and Gigabit (a billion bits per second), two way, telecommuting and tele-education becomes really possible. Kids could continue their school day with video classes even when schools are closed, you could visit your doctor, and of course citizens would have access to all that emergency information and maps described above, real time and two-way. I could go on and on about this – and I have – read it here.

Crowdsourcing and Two-Way Communications, Cell Phones

This area is the most ripe for improved technology to “weather the storm”. In any emergency – even a minor disaster like a major fire or a pile-up collision – just obtaining and distributing information early and often will have a significant result in managing the problem.

On-duty at any time, the City of Seattle may have 200 firefighters, 350 police officers and several hundred to several thousand other employees. Yet we also have 600,000 people in the City, each one of which is a possible source of information. How could we get many of them, for example, to tell us the snow and ice conditions in their neighborhoods? Or perhaps to tell us of problems such as clogged storm drains or stuck vehicles? The Seattle Times actually did this a bit, crowdsourcing snow depths from Facebook.

How can we “crowd source” such information? I’m not exactly sure. Perhaps we could use Facebook apps or Twitter (although not a lot of people use Twitter). Two-way text messages are possible. Any one of these solutions would present a whole mass of data which needs to be processed, tagged for reliability, and then presented as useful analytics.

Eventually, of course, there will be whole armies of remote sensors (“the Internet of things”) to collect and report the information. Perhaps everyone’s cell phone might eventually be a data collector (yes, yes, I’m well aware of privacy concerns).

In the meantime, we should have some way citizens can sign up for alerts about weather or other problems. Many such systems exist, such as the GovDelivery-powered one used by King County Transportation. I’m not aware of such a system being used two-way, to crowd-source information from citizens. There are also plenty of community-notification or “Reverse 911” systems on the market. The Federal government is developing CMAS, which would automatically alert every cell phone / mobile device in a certain geographical area about an impending problem or disaster.

Furthermore, during this Seattle snowstorm, many City of Seattle employees – including police and fire chiefs and department heads, used text messages on commercial cellular networks to communicate with their staff and field units. This continues a tradition of use of text messaging during emergency operations which first came to prominence during Hurricane Katrina.

All of these solutions depend, of course, on reliable cellular networks. We know during disasters commercial cellular networks can easily be overloaded (example: 2011 Hurricane Irene), calls dropped and cell sites can drop out of service as power outages occur and backup batteries at the sites run out of juice. Yet, for people without power or land-line Internet, a smartphone with internet is a potential lifesaver and at least a link to the outside world.

I’d like a way to easily collect this information – privately – from the carriers so emergency managers would know the geographies where mobile networks are impacted.

This leads me, of course, to my final point – that we need a nationwide public safety wireless broadband network. Such a network would be built using spectrum the Congress and the FCC have set aside for this purpose. It would only be used by public safety, although – as our Seattle snowstorm underscored, “public safety” must be used broadly to include utilities, transportation and public works – even building departments. And it would be high speed and resilient, with 4G wireless technology and backup generators, hardened cell sites.

These are a few of my thoughts on better management, through technology, of future snowstorms and other disasters, large and small, both daily and once-in-a-lifetime ones. What have I missed?

Leave a comment

Featured Papers