TL;DR: In our nine-person study, we assessed two AI systems (an Army-tailored one and a general one) and found that AI-assisted search of Army doctrine showed no statistically significant accuracy advantage over PDF search, and correct answers took longer when we considered the two AI systems together. Participants reached correct answers faster with the Army-tailored system than with the general AI system but were less accurate overall, so evaluating the tools required looking at both speed and accuracy.
Finding useful guidance in time
Army doctrine can be hard to search even when the needed documents are at hand. Guidance may be spread across publications, use unfamiliar terms, or apply to several military functions. Army staff have to find it in time to use it, and an incorrect answer can undermine the decision it informs.
Artificial intelligence (AI) might reduce that effort by searching across documents and recognizing related terms. We tested this possibility in a controlled study described in our 2020 technical report. We wanted to know whether an early Army-tailored AI search system could help people answer doctrine questions more efficiently than familiar PDF search.
Searching the same documents three ways
We compared three search methods using the same collection of Army doctrine. For PDF search, participants searched doctrine files stored in a computer folder. The general AI system was a commercial tool already trained on general knowledge. It indexed those same documents and gave participants a search interface.
The Army-tailored version incorporated Army-relevant language and had filters for categories such as warfighting function, echelon, and document type. Both AI systems used the same machine learning and natural language processing algorithms to analyze the doctrine and build searchable indexes. Natural language processing helps software work with human language. Army tailoring was meant to handle differences in terminology: a search for "military decision making process" could also retrieve references to "MDMP."
The screenshot above is the Army system's search display, reproduced from Figure 2 of the report. Participants had to use the information it returned to answer a question. We were measuring how people performed with the complete search tool, including its interface.
Nine participants tried all three methods
Eight retired Army officers and one Army civilian took part at the Mission Command Battle Lab at Fort Leavenworth, KS. Each person used every search method, so we could compare their performance across methods. Before testing, all nine attended a three-hour introduction to the AI systems and their search functions.
They then attempted to answer 90 questions across three sessions over two days, using one search method for each set of 30 questions. Active-duty and retired officers developed the questions around common mission command information needs. One asked, "What are the staff inputs to the Battle Update Brief?" We counterbalanced the order of the search methods so that each appeared equally often in each session position.
Custom software recorded answer time and confidence. Participants wrote their answers in a separate document. Two or three coders assessed accuracy and resolved disagreements through discussion. After each session, participants completed the NASA Task Load Index, a questionnaire about perceived workload, and the System Usability Scale, which asks users to assess a system's usability.
Computer problems left 295 of the 810 possible question-level observations unavailable; another 31 lacked accuracy ratings. That left 484 observations for the accuracy analysis. For timing, we included only the 311 correct answers. A quick response would not count as successful information gathering if the answer was wrong.
Faster with the Army system, less accurate overall
Our analyses accounted for differences among participants and questions. Participants found correct answers faster with PDF search than with the two AI systems considered together (p = .006). We attribute this difference to the slower general AI system search. The Army-tailored system search was faster than the general system search (p < .001).
After accounting for answer time, participants were less accurate with the Army-tailored system than with the general AI system (p < .001). The same analysis found no statistically significant accuracy advantage for the two AI systems together over PDF search (p = .311).
A speed score alone would have missed the Army-tailored system's lower accuracy. We could not establish why these outcomes differed or whether participants deliberately traded accuracy for speed.
Search method did not significantly explain answer confidence either. Participants' reports of workload and usability showed no significant differences among methods. Those ratings describe their experiences with the tools. They do not establish that the systems were equally usable or that performance was equivalent.
What a small study leaves open
These results apply to the systems and tasks we tested with this group. Nine participants gave us enough statistical power to detect large effects, but smaller benefits could have gone undetected. The group also differed from the intended users, active-duty soldiers. Experienced retirees may already have known where to find doctrine and had less need for help searching it.
Training and practice were limited. Participants might have needed more time to become proficient, and straightforward doctrine questions may have missed uses where AI would help more, such as complex searches with several possible answers. We also could not separate the AI's contribution from the interface people used to work with it.
Full Citation
Stothart, C. R., Burland, B. R., Strickland, H. C., Messina, F. D., Couch, D. S., & From, J. D. (2020). Can artificial intelligence systems improve information-gathering efficiency in Army mission command processes? (Technical Report 1382). U.S. Army Research Institute for the Behavioral and Social Sciences. Publication record.