The Pentagon’s IT arm, the Defense Information Systems Agency (DISA), is expressing concerns about the current hype surrounding artificial intelligence (AI). During the agency’s annual Forecast to Industry event, new DISA director Lt. Gen. Paul Stanton emphasized the need for vendors to clearly explain how their AI features function and the data used for training.
“If vendors are incorporating artificial intelligence, they need to be able to explain the details of how and why they’re using it,” Stanton remarked at a roundtable with reporters. “AI requires context. You don’t just sprinkle AI on a problem.”
DISA’s chief technology officer, Steve Wallace, added, “Don’t do it for the joy of doing AI … just to say that you have it.” He noted that many vendors claim to offer AI solutions, but upon further inspection, these claims often lack substance.
Particularly, DISA is not interested in creating clones of ChatGPT. Wallace mentioned that the Air Force has already developed a chatbot for Department of Defense use, called NIPRGPT, thus eliminating the need to duplicate such capabilities: “We don’t need 60 GPTs running around the Department.”
Instead, DISA leaders, including Stanton and Wallace, stressed the need for AI solutions tailored to specific Defense Department missions. They also require an understanding of, and contractual rights to, the underlying data that train these AI systems.
„Being very, very careful and thoughtful about how we acquire, leverage, and analytically resolve the right data is important,” Stanton stated. “That’s at the heart of our business as an agency.”
Douglas Packard, DISA’s director of contracting and procurement, highlighted that the agency’s rights to data will be a significant topic in contract negotiations. “Don’t create AI without thinking I’m going to need the data rights to it,” he advised.
According to Packard, data rights are often the largest negotiation item because those rights could be needed to share data with other contractors to foster competition and avoid vendor lock-in.
DISA also wants to ensure that the data utilized by AI tools is reliable. The agency is looking for AI to be trained on relevant, authoritative, and well-curated data, rather than random posts from the internet. “We don’t want our data derived from the entire corpus of the internet; there’s a lot of bad data,” Stanton remarked.
Training AI on inaccurate or irrelevant data can lead to significant errors. For example, Wallace shared an experience where he asked a large language model how many directors DISA has had, receiving an absurd response of approximately 1,700, which would imply an impossibly frequent change in leadership since the agency’s founding in 1991.
Moreover, DISA discovered errors in an “AI concierge” tool designed to help staff navigate policies. For instance, the AI struggled to identify the most current policy when new ones superseded older ones, leading to contradictory responses based on how users phrased their questions.
To mitigate such risks, Wallace emphasized the importance of a well-trained workforce capable of questioning and validating the AI’s outputs. He noted that current trust in AI resembles the early days of GPS technology, where blind reliance led to navigation mishaps.
Industry must also focus on developing more reliable algorithms that do not hallucinate as frequently. Wallace and other officials suggested that large language models should generate answers based on verified official sources rather than general internet data. They advocated for techniques like Retrieval Augmented Generation (RAG) and the need for LLMs to provide traceable sources for their answers, rather than producing plausible but unverified content.
“There’s some ongoing research that is really intriguing” regarding such “traceable” answers, Stanton said. He acknowledged that requiring AI to cite sources may demand more processing power and data storage but could also lower costs by filtering out irrelevant data.
Stanton concluded, “Help us understand what data is relevant … so we leverage the right data. Or else we’re going to be stuck in data centers we can’t afford, cranking away on datasets that have no relevancy to the problem we’re trying to solve.”
