As if that wasn't enough the state auditor said computer logs showed some employees had visited pornographic Web sites or viewed pornographic images on Transportation Cabinet machines. Finally 33 routers and switches used by the Cabinet were running without password protection, and the state auditor said malicious hackers used these open doors to enter the Cabinet's network and install software tools to ferret out system administrator passwords.
Kentucky's situation may have been extreme, but officials there weren't alone in coping with information security issues. It was not a pleasant summer for a lot of CIOs. A host of worms plagued state and local government IT systems, paralyzing networks and forcing some state agencies to temporarily close their offices.
The events gave policy-makers and IT professionals ample reason to reconsider the importance of security policies and enforcement issues.
Although recent events help underscore the danger, one problem with information security is its vagueness. It's difficult to convince lawmakers to approve significant spending on security when nothing is going wrong. Allocating money for information security means those dollars won't be spent elsewhere, and absent a visible crisis, lawmakers will likely choose the path of least resistance.
Besides securing their computing infrastructure, states must deal with the human side of security. Where should acceptable-use policies originate? Who should enforce those policies? How much latitude should there be?
Not Out of Mind
In Kentucky, CIO Aldona Valicenti weathered a storm over the state's well publicized security nightmares, and the hullabaloo over the unfortunate situation is enlightening on many fronts.
At the end of July, Kentucky's auditor of public accounts sent out a press pack airing the Transportation Cabinet's dirty laundry.
The press pack contained a letter to the secretary of the Transportation Cabinet detailing that French hackers had been distributing pirated material and hosting a chat room on the Transportation Cabinet's servers since early April 2003.
Local newspapers reported later that cyber-attackers from two other countries, Croatia and Canada, also joined in the fun.
The auditor's office also said it had "documented evidence that approximately 30 Transportation Cabinet computers were used 6,000 times within a four-day period to browse pornographic Web sites, images or other materials."
The auditor's press release also blamed the Governor's Office for Technology (GOT) -- led by Valicenti -- and accused the Transportation Cabinet and the GOT of being "asleep at the switch while state computers have been used for illegal purposes."
The spar made for juicy headlines, and though problems in government are sometimes sensationalized out of proportion, there appears to be a fire behind all the smoke. In August, Valicenti confirmed that the FBI seized 11 PCs from the Transportation Cabinet for investigation of potential child pornography.
"For three years," said the auditor in the release, "I have been warning those who manage state systems to make computer security job one."
Cynics may have dismissed the auditor's tactics as an attempt to politicize a nonpolitical issue, but certainly something went wrong in the Transportation Cabinet. Perhaps unfair is that the auditor's attack glued two entirely different security issues -- violations of acceptable-use policies and network infrastructure security -- into one supersecurity problem.
Shooting the Messenger
The GOT is indeed the agency responsible for promulgating a statewide information security policy, and its new Enterprise Network Security Architecture Policy took effect in January 2003.
The GOT has also written a bevy of other security policies, and the state's CIO Advisory Council is working on an enterprise solution for Web filtering, spam and anti-virus protection that will manage Internet access, with the goal of improving the security of Kentucky's IT network. The GOT said a pilot project was completed and an RFP to solicit a software product for implementation is near completion.
In a statement released to the media, Valicenti said, "The Governor's Office for Technology has worked very hard to put in place statewide policies and practices for IT security. That includes policies to help prevent hacking incidents and policies related to viewing pornographic material by state workers.
"We're disappointed those policies were not followed by the Transportation Cabinet, and that this led to a misrepresentation of our efforts by the State Auditor's Office," Valicenti said. "Most issues identified in the auditor's report are related to IT management and procedures, which are the responsibility of each state agency. They are not necessarily pure technology issues. Each agency in state government, including the Transportation Cabinet, has been delegated the responsibility to adhere to state government-wide policies and procedures."
A Management Issue
The GOT can issue policies ad infinitum, but that doesn't necessarily mean the state's networks will be safe. It all comes down to making sure the policies are followed, and therein lies the problem. Who should have enforcement responsibility?
"In Kentucky, the enforcement [of GOT policies], we consider a management issue," Valicenti said.
"Those are not my employees," she continued. "They are not the auditor's employees. They are employees of a certain Cabinet, a certain agency or a certain department that has a management structure. That philosophy is no different in the private sector."
If a department or agency manager suspects an employee is using IT resources inappropriately, she said, the GOT will monitor the employee's usage and provide the resulting information to the agency's management, so appropriate action can be taken.
Agency or division heads are fully aware of their roles, she said, and Cabinet secretaries are notified every time a security policy is updated, as are Cabinet CIOs. The GOT also asks secretaries and CIOs to discuss the updates in their agency management meetings. What may need to happen is even more education.
"I think we have a good policy in place," she said. "We need to make sure we're executing on the policy. Maybe we ought to strengthen the follow-up. We can put out a policy, and then do we go back and start collecting statistics by asking agencies, 'How many people did you reprimand?' I'm a strong believer in scorecards. We did a scorecard during Y2K for each Cabinet by asking, 'What do you need to do? How far are you?'"
A recurring security scorecard could do a lot to help managers keep security in the front of their minds, she said.
In addition, the GOT is reviewing its security policies to make sure they're adequate and discuss monitoring strategies. What needs to be avoided is having a Big Brother type organization, she said, because that approach isn't warranted based on the numbers of employees who don't misuse state IT resources.
"We've done a lot at the front end with the policy and making sure people understand it's there," she said. "Now we need to look at the back end. What is it we need to do to track and alert and monitor, when appropriate, and audit? We're looking at whatever outside guidance we can. We're looking at NIST's [National Institute of Standards and Technology] guidelines to find out what we need to do and where we can do a better job."
Practical Realities
It remains extremely difficult to stop electronic attacks, said Jim Ramsey, CIO of the Kentucky Transportation Cabinet, especially when knowledgeable security staff is in short supply, and with the costs in securing networks and the ease of getting caught up in everyday activities.
"We were paying attention to supporting some business processes going on, and had not paid enough attention to the security that apparently we needed," Ramsey said. "Security is not inexpensive, and it takes an inordinate amount of time. We've had numerous people reassigned. We've put in a lot of extra overtime.
"We are in the process of hiring additional staff dedicated to security," he continued. "We did have a security officer that had some shared responsibilities, but we didn't have a full-time staff concentrating on enforcing password authentications and those kinds of things. At this stage of the game, we're willing to dedicate a personnel slot to that. Realistically we probably need about six people doing that around the clock."
He said his IT staff is supporting approximately 6,000 employees across a statewide distributed network with about 150 servers and more than 4,000 desktop PCs. As a result of fallout over the Cabinet's incident, Ramsey said he and his staff will start running software and other diagnostic tools to probe for weaknesses at the desktop level.
"That is a very time- and labor-intensive effort," he said. "We now have to go out and crack everybody's password to make sure people are abiding by the password policies, and whether it takes 38 minutes to crack it or 40 seconds to crack it, we'll now know."
Ramsey said he's preparing reports to alert management of the percentage of employees not in compliance with policies because they're using passwords that appear in the dictionary or don't have enough characters. Even that will be an ongoing effort, he said, because without continued vigilance, users tend to revert back to old habits and use weak passwords. This effort, again, takes staff time and resources.
Juggling the myriad responsibilities in today's climate is hard for any CIO, and the situation Ramsey found himself in could happen to other agency CIOs across the country.
"There is, realistically, a certain amount of attention you should be providing to security, to network maintenance, to making sure your systems are safe," he said. "Trying to balance all of those with a reduced staff, with an insufficient funding level, is a real challenge. You're going to err somewhere, and it becomes necessary to make some decisions as to where that's going to be."
Ramsey said funding requests for security enhancements have been turned in to the Legislature for the last several years, but they haven't been granted. The nature of security is part of the problem.
"IT security is very nebulous," he said. "It's hard to show people that IT systems are safer when they say, 'Well, I wasn't really having problems with those systems six months ago, and even though you say they're safer now, I'm still not seeing a real difference.'
"When you start throwing millions of dollars to security and you don't have a visible return, it becomes difficult to justify," he said.
Education is Golden
Though enterprise security policies fortify architecture weaknesses against electronic attacks, network-connected desktop PCs offer many paths for malicious code to enter an enterprise network. A big part of enforcing IT policies is making sure employees know what not to do, which is what acceptable-use policies are meant to do.
Michigan's Department of Information Technology (DIT) just put the finishing touches on an updated, enterprise-wide acceptable-use policy, said Dan Lohrmann, the DIT's chief information security officer.
He said the DIT updated the policy to respond to increasing types of security vulnerabilities facing the state every day. Though viruses that cripple public- and private-sector networks, such as Sobig and Lovesan, cause their share of headaches, those attacks also do some good.
"You never want to hope for incidents, but I would say they reinforce the message that it's very important that we have proper processes and procedures in place should we get hit by viruses," Lohrmann said. "This isn't the first time, and we learned a lot of lessons from Code Red and Code Red II."
Education is one of those lessons, he said, and Michigan's new policy is designed to eliminate misconceptions employees might have relating to how they use their computers.
In the past, agencies wrote their own acceptable-use policies, leading to a crazy quilt of approaches. The agency-specific policies also contained outdated legal language, he said, and often were not communicated well to employees.
"You had policies all over the map," he said. "You had enforcement all over the map. You've got a whole different perspective on PC awareness training from agency to agency. It all goes back to agencies' culture. It's an ongoing cultural change issue."
Like Valicenti, Lohrmann said the DIT's role is not enforcement, and the education process in Michigan includes making managers aware that it's their job to enforce policies drafted at higher levels.
"I'm not going to be effective if the department directors don't send this message down through their management chains," he said. "One of the things we've encouraged is that the message be talked about at staff meetings. You don't have to dwell on it, but this is something that's not just one office or five people or 10 people, or even that the DIT is the police. The enforcement is really in the business area. We're the coordinators."
As part of its education campaign, the DIT will send an e-mail to all affected employees explaining the new policy, and Lohrmann said it's likely the e-mail will come from the governor and contain a link to the policy.
Getting to the point where Michigan agencies trust what DIT says and support the new enterprise-wide acceptable-use policy took some time and quite a few rounds of discussions.
"People didn't understand the risks," he said. "After several different briefings we gave them on incidents we're seeing, on the kinds of things that are going on -- the inappropriate use in some places -- they bought in very quickly. You have to steer the discussion and try get buy-in and develop the groundswell of opinion of the different agencies."
Not Fixing What's Broken
States do appear to be paying attention to security risks, but some argue that approach won't solve the problem since the attacks are made possible by vulnerable software.
In Louisiana, the recent Lovesan/MSBlast worm forced the state to temporarily shut down all 86 of its driver's license offices. Maryland, Kentucky and California fought other attacks, like the Sobig and Nachi worms. The worms also victimized local governments in Kentucky and Texas.
"You might say, 'Why didn't the security people do their jobs?'" said Alan Paller, director of research at the SysAdmin, Audit, Network, Security (SANS) Institute. But Paller said the only solution is to have security built in. "You've got to have it baked in when you buy it and as you use it," he said.
Security must be part of software because making end-users responsible for updating patches or other fixes isn't working, Paller said. Users find the process inconvenient, and because they're human, they put it off until it's too late.
Given this inescapable fact, he said, shouldn't vendors be made responsible for securing the products they sell? Should government take a more aggressive stance on procurement to perhaps force vendors to build in more security?
issues -- violations of acceptable-use policies and network infrastructure security -- into one supersecurity problem.
Shooting the Messenger
The GOT is indeed the agency responsible for promulgating a statewide information security policy, and its new Enterprise Network Security Architecture Policy took effect in January 2003.
The GOT has also written a bevy of other security policies, and the state's CIO Advisory Council is working on an enterprise solution for Web filtering, spam and anti-virus protection that will manage Internet access, with the goal of improving the security of Kentucky's IT network. The GOT said a pilot project was completed and an RFP to solicit a software product for implementation is near completion.
In a statement released to the media, Valicenti said, "The Governor's Office for Technology has worked very hard to put in place statewide policies and practices for IT security. That includes policies to help prevent hacking incidents and policies related to viewing pornographic material by state workers.
"We're disappointed those policies were not followed by the Transportation Cabinet, and that this led to a misrepresentation of our efforts by the State Auditor's Office," Valicenti said. "Most issues identified in the auditor's report are related to IT management and procedures, which are the responsibility of each state agency. They are not necessarily pure technology issues. Each agency in state government, including the Transportation Cabinet, has been delegated the responsibility to adhere to state government-wide policies and procedures."
A Management Issue
The GOT can issue policies ad infinitum, but that doesn't necessarily mean the state's networks will be safe. It all comes down to making sure the policies are followed, and therein lies the problem. Who should have enforcement responsibility?
"In Kentucky, the enforcement [of GOT policies], we consider a management issue," Valicenti said.
"Those are not my employees," she continued. "They are not the auditor's employees. They are employees of a certain Cabinet, a certain agency or a certain department that has a management structure. That philosophy is no different in the private sector."
If a department or agency manager suspects an employee is using IT resources inappropriately, she said, the GOT will monitor the employee's usage and provide the resulting information to the agency's management, so appropriate action can be taken.
Agency or division heads are fully aware of their roles, she said, and Cabinet secretaries are notified every time a security policy is updated, as are Cabinet CIOs. The GOT also asks secretaries and CIOs to discuss the updates in their agency management meetings. What may need to happen is even more education.
"I think we have a good policy in place," she said. "We need to make sure we're executing on the policy. Maybe we ought to strengthen the follow-up. We can put out a policy, and then do we go back and start collecting statistics by asking agencies, 'How many people did you reprimand?' I'm a strong believer in scorecards. We did a scorecard during Y2K for each Cabinet by asking, 'What do you need to do? How far are you?'"
A recurring security scorecard could do a lot to help managers keep security in the front of their minds, she said.
In addition, the GOT is reviewing its security policies to make sure they're adequate and discuss monitoring strategies. What needs to be avoided is having a Big Brother type organization, she said, because that approach isn't warranted based on the numbers of employees who don't misuse state IT resources.
"We've done a lot at the front end with the policy and making sure people understand it's there," she said. "Now we need to look at the back end. What is it we need to do to track and alert and monitor, when appropriate, and audit? We're looking at whatever outside guidance we can. We're looking at NIST's [National Institute of Standards and Technology] guidelines to find out what we need to do and where we can do a better job."
Practical Realities
It remains extremely difficult to stop electronic attacks, said Jim Ramsey, CIO of the Kentucky Transportation Cabinet, especially when knowledgeable security staff is in short supply, and with the costs in securing networks and the ease of getting caught up in everyday activities.
"We were paying attention to supporting some business processes going on, and had not paid enough attention to the security that apparently we needed," Ramsey said. "Security is not inexpensive, and it takes an inordinate amount of time. We've had numerous people reassigned. We've put in a lot of extra overtime.
"We are in the process of hiring additional staff dedicated to security," he continued. "We did have a security officer that had some shared responsibilities, but we didn't have a full-time staff concentrating on enforcing password authentications and those kinds of things. At this stage of the game, we're willing to dedicate a personnel slot to that. Realistically we probably need about six people doing that around the clock."
He said his IT staff is supporting approximately 6,000 employees across a statewide distributed network with about 150 servers and more than 4,000 desktop PCs. As a result of fallout over the Cabinet's incident, Ramsey said he and his staff will start running software and other diagnostic tools to probe for weaknesses at the desktop level.
"That is a very time- and labor-intensive effort," he said. "We now have to go out and crack everybody's password to make sure people are abiding by the password policies, and whether it takes 38 minutes to crack it or 40 seconds to crack it, we'll now know."
Ramsey said he's preparing reports to alert management of the percentage of employees not in compliance with policies because they're using passwords that appear in the dictionary or don't have enough characters. Even that will be an ongoing effort, he said, because without continued vigilance, users tend to revert back to old habits and use weak passwords. This effort, again, takes staff time and resources.
Juggling the myriad responsibilities in today's climate is hard for any CIO, and the situation Ramsey found himself in could happen to other agency CIOs across the country.
"There is, realistically, a certain amount of attention you should be providing to security, to network maintenance, to making sure your systems are safe," he said. "Trying to balance all of those with a reduced staff, with an insufficient funding level, is a real challenge. You're going to err somewhere, and it becomes necessary to make some decisions as to where that's going to be."
Ramsey said funding requests for security enhancements have been turned in to the Legislature for the last several years, but they haven't been granted. The nature of security is part of the problem.
"IT security is very nebulous," he said. "It's hard to show people that IT systems are safer when they say, 'Well, I wasn't really having problems with those systems six months ago, and even though you say they're safer now, I'm still not seeing a real difference.'
"When you start throwing millions of dollars to security and you don't have a visible return, it becomes difficult to justify," he said.
Education is Golden
Though enterprise security policies fortify architecture weaknesses against electronic attacks, network-connected desktop PCs offer many paths for malicious code to enter an enterprise network. A big part of enforcing IT policies is making sure employees know what not to do, which is what acceptable-use policies are meant to do.
Michigan's Department of Information Technology (DIT) just put the finishing touches on an updated, enterprise-wide acceptable-use policy, said Dan Lohrmann, the DIT's chief information security officer.
He said the DIT updated the policy to respond to increasing types of security vulnerabilities facing the state every day. Though viruses that cripple public- and private-sector networks, such as Sobig and Lovesan, cause their share of headaches, those attacks also do some good.
"You never want to hope for incidents, but I would say they reinforce the message that it's very important that we have proper processes and procedures in place should we get hit by viruses," Lohrmann said. "This isn't the first time, and we learned a lot of lessons from Code Red and Code Red II."
Education is one of those lessons, he said, and Michigan's new policy is designed to eliminate misconceptions employees might have relating to how they use their computers.
In the past, agencies wrote their own acceptable-use policies, leading to a crazy quilt of approaches. The agency-specific policies also contained outdated legal language, he said, and often were not communicated well to employees.
"You had policies all over the map," he said. "You had enforcement all over the map. You've got a whole different perspective on PC awareness training from agency to agency. It all goes back to agencies' culture. It's an ongoing cultural change issue."
Like Valicenti, Lohrmann said the DIT's role is not enforcement, and the education process in Michigan includes making managers aware that it's their job to enforce policies drafted at higher levels.
"I'm not going to be effective if the department directors don't send this message down through their management chains," he said. "One of the things we've encouraged is that the message be talked about at staff meetings. You don't have to dwell on it, but this is something that's not just one office or five people or 10 people, or even that the DIT is the police. The enforcement is really in the business area. We're the coordinators."
As part of its education campaign, the DIT will send an e-mail to all affected employees explaining the new policy, and Lohrmann said it's likely the e-mail will come from the governor and contain a link to the policy.
Getting to the point where Michigan agencies trust what DIT says and support the new enterprise-wide acceptable-use policy took some time and quite a few rounds of discussions.
"People didn't understand the risks," he said. "After several different briefings we gave them on incidents we're seeing, on the kinds of things that are going on -- the inappropriate use in some places -- they bought in very quickly. You have to steer the discussion and try get buy-in and develop the groundswell of opinion of the different agencies."
Not Fixing What's Broken
States do appear to be paying attention to security risks, but some argue that approach won't solve the problem since the attacks are made possible by vulnerable software.
In Louisiana, the recent Lovesan/MSBlast worm forced the state to temporarily shut down all 86 of its driver's license offices. Maryland, Kentucky and California fought other attacks, like the Sobig and Nachi worms. The worms also victimized local governments in Kentucky and Texas.
"You might say, 'Why didn't the security people do their jobs?'" said Alan Paller, director of research at the SysAdmin, Audit, Network, Security (SANS) Institute. But Paller said the only solution is to have security built in. "You've got to have it baked in when you buy it and as you use it," he said.
Security must be part of software because making end-users responsible for updating patches or other fixes isn't working, Paller said. Users find the process inconvenient, and because they're human, they put it off until it's too late.
Given this inescapable fact, he said, shouldn't vendors be made responsible for securing the products they sell? Should government take a more aggressive stance on procurement to perhaps force vendors to build in more security?
"The only organizations that have the buying power to provide the incentives to the vendors to bake it in are governments," Paller said. "So the traditional policy of kowtowing to the vendors has been another failure, and it's time for a switch."
He cited changes put into effect by the U.S. departments of Energy (DOE) and Transportation (DOT) that shift responsibility to the companies that sell the products, instead of the customer, as a glimpse into what the future of software procurement might hold. The DOE, he said, essentially forced Oracle to secure its database product before purchasing the software, while the DOT is focusing its efforts on monitoring software vulnerabilities and ensuring those vulnerabilities are repaired by the vendor.
"The shift comes only when the users put money there, not just requests," Paller said. "When the DOE delays procurement until Oracle delivered a safer system, then you begin to get action. At some point, the buyers -- and it's got to be government because it has to be a buyer with enough money -- have to say, 'We're mad as hell, and we're not going to take it anymore.'
"The way government should respond is not with regulation, but simply say, 'Look, there are three different database systems available. If you want to sell to us, we'd like to use your product, you sell it this way, you take responsibility for securing it and you take liability if you don't secure it,'" he said.
Procurement as a Weapon
Paller said SANS recommends a three-part program to the federal government to fight potential vulnerabilities by changing the rules. First is the procurement approach to software vulnerabilities, modeled after what the DOE is doing.
"The vendor is not only responsible for delivering a product safely, and that means configured according to something like the Center for Internet Security Benchmarks, but the vendor is responsible for delivering security patches and installing them on a reference version of their software at the customer's site," he said.
He said the DOE purchased additional software tools to distribute the reference version to all users and automatically issue updates.
The second part of the program is implementing a testing and reward program that measures the number of vulnerabilities per system for every computer owned by a government agency, Paller said, which follows what the DOT is doing.
"The DOT does constant scanning and rating of all the divisions of the DOT and makes the results of that scanning visible to all the managers of the department," he said. "You watch how fast people get better. Every one of us who leaves our system vulnerable makes it dangerous for everyone else."
The third aspect of this program is the creation of a 24/7 response network for agencies, in which every agency designates a contact person who is always available in case of an emergency or attack.
"There's more, but if a government did those three things, it would be way ahead of everyone else in the world," he said.
Security After the Sale
On Sept. 23, the DOE formally took the wraps off what could be a new trend in software contracts -- one that should interest state and local governments, said Tim Hoechst, senior vice president of technology for Oracle's Government, Education and Healthcare group.
From Oracle's perspective, the two most important aspects of the model contract center on how the DOE changed its procurement methods and what role the company will play in security after the procurement.
The issue of security is key, Hoechst said, because the company's customers -- especially large, enterprise customers -- have enough leverage with their aggregate buying power to ensure products and services mirror their security needs.
"What this contract focused on is making sure that not only do we build secure products, but that we promise to help the customer use them in a secure way and the most secure fashion as possible," Hoechst said. "We could make the best and strongest door locks, for example, but our customers still have to remember to lock the doors. This contract is about making sure we get involved in that end of the process as well, and making sure they use the technology to its fullest."
On the DOE's side, the department consolidated purchasing performed by various entities within the department and rolled that business into a larger, enterprise-level agreement. That approach, he said, gave the DOE leverage to ask for additional security measures while helping the company manage the cost of sale the contract represents.
"They save money; we save money," said Hoechst. "They get exactly what they want. We know exactly who our customer is, day in and day out. That sort of change is something we'll continue to see across government acquisitions."
The new security benchmark at the heart of the model contract is available to all governments online at the Center for Internet Security's (CIS) Web site. The document compiles more than 250 security installation and configuration recommendations identified by commercial organizations, federal government agencies and security specialists from Oracle, according to CIS.
When it comes to security, the software industry has been criticized for making vulnerable products. The DOE's model contract could go a long way toward getting people to change their minds, but no company is safe from this first impression.
"We have been getting grouped into the collection of 'software companies' that write insecure stuff and require tons of patches," Hoechst said. "While we aren't claiming to be perfect, we don't like being included in the same sentence with the other companies that have not prioritized security. That said, it is important for us to remain responsible providers of technology.
"We would love to see government continue to use its buying power to influence us and the rest of our industry to continue to focus on developing secure technology, and even going to the next step, which is what the DOE agreement does -- to make us responsible not only for delivering secure technology, but in helping ensure it is used in the most secure fashion."
