Why does President Trump suddenly care who wins the Pentagon’s $10 billion cloud-storage contract?
The president is not known for wading deep into information technology policy. But on July 22, Trump tweeted about a Fox News segment alleging that the Pentagon was involved in a crooked scheme to award Amazon the coveted Joint Enterprise Defense Infrastructure Program, or JEDI.
“It’s not just appropriate but vital that the president kills this contract” said Steve Hilton, a Fox News host.
The Pentagon was expected to choose between Amazon, the early favorite, and Microsoft. But on Thursday, a Pentagon spokesperson confirmed that new Defense Secretary Mark Esper put a hold on the contract award while he reviews the process. “No decision will be made on the program until he has completed his examination,” said Elissa Smith.
Esper could decide to throw out the bids and ask for new ones, or cancel the JEDI program entirely and break up its cloud efforts into smaller chunks. If that happens, according to two former senior defense officials familiar with the drafting of the contract requirements, the program could be delayed, costs could rise by billions of dollars, and troops may head into combat lacking critical information.
Opposition From the Start
When the Pentagon’s plan to award a single contract for enterprise-wide commercial cloud services leaked through the press in November 2017, it generated immediate opposition. Industry groups representing established defense contractors lobbied the Pentagon and Congress to consider awarding JEDIto multiple companies. Critics contended that certain requirements for JEDI, such as the Impact Level 6 security required to host classified information outside government facilities, were so stringent that only one or two companies that provide cloud services could meet them. Amazon Web Services, which already held a $600 million contract with the CIA to host secret and top-secret classified data, was the obvious early frontrunner.
In March 2018, the Pentagon announced it had received more than 1,000 comments from industry, many decrying the winner-takes-all plan. That same month, Congress began asking about JEDI. In its responses, the Pentagon justified its approach as faster and easier to manage. Whereas a single-award contract would result in one competition among industry to determine the best overall cloud service provider, a multiple-award contract would require countless competitions for specific task orders between companies. These competitions would be protestable, which could have led to significant delays that ultimately impact the U.S. troops awaiting the technology.
That’s when the competition got ugly. In early 2018, a misinformation campaign to discredit the JEDI procurement and influence decision-makers began to take shape. As reported last August by Defense One and Nextgov, a private investigative firm, RosettiStarr, began circulating to reporters a 100-page dossier of allegations of improper contact between Pentagon officials and Amazon, including Defense Secretary Jim Mattis and some of his staff. The same unsubstantiated allegations appeared in Fox News’ broadcast last month.
While it remains unclear who or what funded the dossier, similar allegations were made by Amazon’s rival Oracle in an unsuccessful lawsuit against the Defense Department and in a one-page document created by Oracle’s top Washington lobbyist that allegedly was shown to Trump in late July. The document includes a flow chart titled, “A Conspiracy to Create a Ten Year DoD Cloud Monopoly,” and alleges personnel in the Pentagon and Amazon conspired to create the JEDI procurement.
The allegations also attracted further attention from lawmakers.
In October 2018, Reps. Steve Womack, R-Ark., and Tom Cole, R-Okla., members of the House Appropriations Committee, askedthe Defense Department inspector general to investigate the beginnings of JEDI. Other Republican lawmakers have since criticized JEDI, citing allegations of impropriety between AWS and defense officials. In June, on Fox News’s “Tucker Carlson Tonight,” Rep. Mark Meadows, R-North Carolina, called the allegations against Amazon “incredible” and called for an investigation. Sen. Ron Johnson, of Wisconsin, who chairs the Homeland Security Committee, asked the defense secretary’s office in a late-June letter to delay awarding the JEDI contract until an investigation is completed. Womack also re-upped his concerns in a July 8 letter to Trump, asking for his “personal attention” in the matter. Sens. Marco Rubio, of Florida, and Chuck Grassley, of Iowa, also issued statements critical of JEDI.
Trump’s public disdain for Amazon and its founder, Jeff Bezos, is not lost on those attempting to influence JEDI’s outcome. The president has tweeted attacks on Amazon’s shipping business and referred to Bezos as “Jeff Bozo.” Trump gets along better with Oracle CEO Safra Catz, who served on Trump’s transition team and with whom Trump had a private dinner in April 2018. Bloomberg reported that Catz expressed concerns about the contract during that engagement.
On July 18, Trump finally weighed in, saying he would review JEDI after hearing “tremendous complaints” about the contract from “some of the great companies in the world,” including JEDIbidders Microsoft, Oracle, and IBM.
But the next day, four Republican lawmakers on the House Armed Services Committee wrote Trump asking him to ignore outside complaints and allow the Pentagon to award JEDI. The program is needed to keep pace with China and Russia, said Rep. Mac Thornberry, R-Texas, who is ranking member, and Reps. Elise Stefanik, R-N.Y., Michael Turner, R-Ohio, and Robert Wittman, R-Va.
“We believe that it is essential for our national security to move forward as quickly as possible with the award and implementation of this contract,” said the authors. “Our committee has conducted oversight of this contract from the beginning. As you know, the courts have upheld DOD’s handling of this competition. While it is understandable that some of the companies competing for the contract are disappointed at not being selected as one of the finalists, further unnecessary delays will only damage our security and increase the costs of the contract.”
In July, the court sided with the Pentagon, which summed up the decision in a statement: “The Court concurs with DOD’s extensive review – that the individuals at issue were ‘bit players,’ in the Court’s own words, and the alleged conflicts had no impact on the integrity of the procurement. DOD officials directly involved in the work of this procurement along with the senior leaders charged with making the critical decisions related to JEDI have always placed the interests of the warfighter first and have acted without bias, prejudice, or self-interest. The same cannot be said of all parties to the debate over JEDI.”
A Cure For the Status Quo
The JEDI program is intended to address a critical need that Pentagon officials claim they are not able to meet with current ITstructure. The U.S. military wants to move data quicker, including vital intelligence. There are too many disparate networks across bespoke weapons and systems and too little coordination in the way the Pentagon handles data now, the two former senior defense officials argue. The problem will only get worse as the military moves toward new concepts of highly networked warfare and embraces new tools like artificial intelligence, virtual reality for training, and more computer modeling for drafting its plans.
One example of the current problem is the Modernized Integrated Database, or MIDB, which contains information on enemy units, equipment, and the location of friendly or non-hostile elements — essentially, “a list of hospitals and schools and other things that we don’t want to drop a bomb on,” the second official said.
It’s also a bit of a mess. The first official called it “a series of disconnected databases that are…almost manually replicated across a series of networks and places.” It can take six months to incorporate new information, like the opening of a new hospital, and that can produce, say, airstrikes that go horribly awry.
“The Pentagon’s IT structure grew up one system at a time. It was a very decentralized structure” said Tom Spoehr, who leads the Heritage Foundation’s Center for National Defense. As a three-star general, he led the Army’s Office of Business Transformation. “So Fort Bragg, or DISA, they all went out and found their own cloud and contracted for it. No one had the central vision at the DOD level that they should be all connected. They were just happy to be in the cloud, as opposed to the server closet behind your office door.”
On July 25, the conservative think tank published a white paperarguing that Congress and the White House should step aside and let the JEDI award proceed. Spoehr believes the companies who challenged the JEDI approach are trying to change its requirements to more closely suit their current product line. And the former defense officials who spoke to Nextgov and Defense One said those smaller solutions, when pieced together, would only bring more of what JEDI is intended to fix.
“The vision of JEDI was: what if we were to have something that worked very well, that was secure, that moved the department out of doing activities and work that took it off missions, things like patching servers, things like maintaining software updates?” said the first official.
The official described traveling to a remote military base and encountering equipment that was intended to help the base improve its computing and IT capabilities, but no one knew what to do with it.
“’Why is the server not plugged in? Well, they don’t know what the server’s for. Why don’t they know what the server is for? Well, it took twelve months to get the server. What is the average rotation on the people who were there? Three or four months,” he said. “So, several generations ago, a person saw a need that required a server and ordered a server. That person left. The next person left. Eventually we show up, this person’s showing me a closet where the server is not plugged in, has no cables, has no blinking lights…they don’t know what to do with it.”
A massive enterprise-level cloud, the sort of infrastructure relied upon by streaming entertainment services like Netflix, offers an alternative less reliant on hardware and networks that the military has to stand up. The hope is for a solution that could be “always secure, up to date, and could scale nearly infinitely.”
The Pentagon believes one large cloud network would also be far more resilient than a hodgepodge of networks. In the event that one server cluster in a cloud environment stops performing, backup servers quickly take over the same tasks and the user doesn’t experience any interruption in service. It’s a safeguard known as “failover.” That ability to failover isn’t unique to enormous cloud environments of the sort that Amazon and Microsoft are competing to build for the Pentagon; but some enterprises do it better than others.
Netflix, for instance, has built failover into its operations so thoroughly that users across the world don’t experience interruptions even when entire geographic regions lose service. Service can be re-routed elsewhere. Netflix is able to do that, in part, by relying on massive enterprise web services provided by Amazon. The Pentagon doesn’t have anything that comes close.
“Every one of the commercial cloud providers is driven by requirements of tens, if not hundreds, maybe even millions, of other customers who are demanding failover and redundancy and scalability that far surpass anything that we have,” said the first official. “Every one of these cloud providers have failover that is so much more advanced than anything that exists at the Department of Defense. It seems like a dream to us.”
If opponents of the JEDI program force the Pentagon to adopt a kludge of smaller clouds patched together, matching Netflix is impossible, the former officials argue.
Commercial cloud services are often treated like a commodity, but there are very real and important technical differences, such as unique application programming interfaces, or APIs, which define communication computer components, or software development kits, or SDKs, the tools that allow for development of applications within a specific software framework. It’s a bit like trying to mix LEGO bricks with DUPLO bricks. You can push them together but because the bricks aren’t exactly the same size and shape, the structure you build will be far less attractive than if you had stuck with one brick type.
That’s not to say that so-called multi-cloud environments don’t work. And it doesn’t mean the Pentagon won’t continue to have them or build them. But a “multi-cloud environment” is the inadequate status quo that the Pentagon is trying to push past, the officials said.
JEDI is also meant to expand the Pentagon’s ability to accommodate massive amounts of data for future weapons and networks. The military is pursuing plans to integrate virtually every jet, ship, drone, satellite, and soldier on the battlefield in an enormous data web, part of a dream for the future, networked, multi-domain operations.
Spoehr says that vision is an essential part of competing with China, and deterring a potential war. The Pentagon’s current ITsystem is a huge barrier, he says. “We are trying to get to full joint interdependence, where an Army sensor sees a target and can call on the Air Force nearly instantaneously to deliver a munition on it. So if you are working with disparate IT systems and disparate clouds, you’ll never get there,” he says.
Those future needs also include distributing new software instantaneously to soldiers who need it near points of conflict.
“AI is the future of warfare,” said the second official, “Let’s say the Joint AI Center develops an algorithm to process sensor data and find mobile missile launchers. How does that algorithm get deployed on our existing platforms today?”
The answer is: manually, over and over again, at great cost and after much delay. If there isn’t enough computing power to handle the new algorithms at the bases or posts where it is needed most, then more must be purchased. The current process can take close to a year because each new server, whether real or virtual (a so-called server instance), must be certified or accredited. That’s much more cumbersome than drawing from a single, giant, accredited cloud, noted the first official.
“That is a capability that exists everywhere else but not in the Department of Defense.”