Biz & IT —

Why the “biggest government hack ever” got past the feds

Inertia, a lack of internal expertise, and a decade of neglect at OPM led to breach.

Einstein, cubed and accelerated

A DHS Inspector General report diagram of how Einstein works.
Enlarge / A DHS Inspector General report diagram of how Einstein works.
Einstein, despite its name, is not exactly an intelligent system. Much like NSA's Xkeyscore surveillance system, it depends on its human DHS masters to tell it what exactly to look for.

Einstein began as strictly a network monitoring operation. In 2005, DHS's National Protection Programs Directorate (NPPD) started putting the first generation of Einstein "sensors" on federal agencies' Internet access points. Called Einstein 1, the system passively monitored .gov Internet traffic and collected "network flow records" in an attempt to establish baselines for normal network activity and detect potential attacks as they occurred.

DHS quickly discovered that agencies didn't even know how many Internet gateways they had, let alone what was going out over their ISP connections. In 2007, Einstein 1 monitoring became a mandatory part of DHS's Trusted Internet Connections initiative—an effort to pare down .gov networks' Internet gateways to a manageable, securable number. In August of that year, NPPD debuted Einstein 2—a more advanced intrusion detection and alert capability. It was deployed with the ISPs serving as trusted Internet gateways to generate alerts on "the presence of malicious network activity," according to a DHS Inspector General report. According to DHS officials, Einstein 2 is based exclusively on unclassified threat data—so it cannot be fed information from NSA.

But the Bush administration's cyber-strategists realized that intrusion detection wasn't enough. A January 2008 secret directive, National Security Presidential Directive 54 (NSPD 54), mandated that federal agencies increase network protection. This launched the Comprehensive National Cybersecurity Initiative (CNCI)—an effort the Obama administration would quickly escalate to be the centerpiece of a larger national cybersecurity policy. One of the immediate requirements of CNCI was a government-wide intrusion prevention system. The feds wanted something that would shoot down cyber-attacks before they reached their networks.

The provisions of President George W. Bush's NSPD-54 upping the ante on Einstein and calling for "active response sensors" to block and possibly counter network attacks.
Enlarge / The provisions of President George W. Bush's NSPD-54 upping the ante on Einstein and calling for "active response sensors" to block and possibly counter network attacks.

That capability would become known as Einstein 3. "Einstein 3 will draw on commercial technology and specialized government technology to conduct real-time full packet inspection and threat-based decision-making on network traffic entering or leaving these Executive Branch networks," members of the Obama Administration's policy team wrote in a 2009 overview of CNCI's action items. "The goal of Einstein 3 is to identify and characterize malicious network traffic to enhance cybersecurity analysis, situational awareness, and security response. It will have the ability to automatically detect and respond appropriately to cyber threats before harm is done, providing an intrusion prevention system supporting dynamic defense. EINSTEIN 3 will assist DHS US-CERT in defending, protecting and reducing vulnerabilities on Federal Executive Branch networks and systems."

By April 2012, the "specialized government technology" part of the Einstein 3 equation was bogging things down. It became apparent that Einstein 3 wasn't happening fast enough. DHS pivoted, reaching out to the industry for commercially available intrusion prevention tools under the program name "Einstein 3 Accelerated (E3A)" as a managed security service. The first service provider for E3A, Louisiana-based telecom company CenturyLink, got approved for operation and started in December 2014.

The main difference between Einstein 3's vision and E3A's delivered capability is that E3A still depends on identifying "known or suspected cyber-threats." According to CenturyLink's press release on the first E3A order, "EINSTEIN capabilities are provided through a combination of commercial off-the-shelf hardware and software, government-developed software, and commercially available managed security services that are enhanced by DHS-provided information."

That "DHS-provided information" is threat profile information created by DHS' US-CERT from analysis of existing attacks and threats. US-CERT and NPPD maintain local workstations at each trusted ISP running Einstein to pull recorded network traffic into DHS' Top Secret Mission Operating Environment, a shared network that allows US-CERT analysts to dive into suspicious traffic and share it with NSA. However, that sharing occurs only after information has been appropriately scrubbed of personal identifying information, according to the privacy provisions of DHS policy. When threats are identified, the traffic data is used to create rules for the Einstein system's sensors that will in turn trigger alerts and traffic blocking.

All that's great for stopping attacks if they have been seen before. It's not exactly great protection against "zero-day" attacks that disguise themselves as normal traffic. "Systems that are just based on detecting are a great compliment to other things," said Ammon. "But if you're betting the farm on that, it's not a winning strategy. [On the other hand,] two years ago we wouldn't have even known this was going on. That we can do something to get them out—that's not something we could have done before."

Defense out of its depth

Unfortunately, given the state of security at OPM and other federal agencies, this sort of post-attack forensics and remediation is about all Einstein will be good for in many cases. An Ars review of federal agency security audits found similar issues across the government with varying levels of severity. Even when security actions were taken, they were often misinformed—such as when the Economic Development Administration physically destroyed entire computers (including their keyboards and mice) when agency officials believed they were experiencing a malware outbreak in 2011.

Ammon believes the problems faced by the federal government are the same facing many organizations.

"What I see as the challenge here is that we spent 20 years accumulating a whole slew of bad practices, and we have issues with the underlying technology," he said. "New technology really comes with the pricetag of changing how you do business—organizations used to have a basement with 100 really smart sysadmins, and nobody questioned how they did things.

"That sort of loosely controlled world is a playground for these attacks," he continued. "So we have to overcome everything that we've accumulated from almost the advent of technology—this pile of bad practices and configurations—we have to unwind that and keep not only the hactivism guys out but well-funded state hackers out. It just takes time."

It also will take a cultural change in government, and this problem isn't only limited to federal government. State governments are an attractive target as well, especially to cyber-criminals looking to take advantage of antiquated tax systems and other setups laden with personal and financial data. State governments aren't as well-equipped to deal with the mitigation of attacks. Webroot's Milbourne noted that when he contacted State of Colorado's Taxation Division about his tax return—which their system this past week still insisted the agency hadn't received—he was told that because of a spike in fraud, all returns were being handled by hand. (Colorado's Department of Revenue has not yet responded to an Ars inquiry on the issue.)

The most unnerving part of the current state of government IT security is that there's no way to know the extent of breaches. The Obama administration doesn't seem inclined to necessarily inform the public of them based on the comments Caitlin Hayden, a spokesperson who talked with the New York Times. "The administration has never advocated that all intrusions be made public. We have advocated that businesses that have suffered an intrusion notify customers if the intruder had access to consumers' personal information. We have also advocated that companies and agencies voluntarily share information about intrusions."

The security community is paying attention to both the breaches and bureaucratic banter. "They came off as belligerent, didn’t acknowledge their fault in the breach, and provided very few details," Jeff Williams, CTO of Contrast Security, told Ars.. "How about the people whose sensitive information was in the e-QIP database, including me? What am I supposed to do now?"

Given the state of government IT, it's a question many more American citizens may be asking in the near future.

Channel Ars Technica