{ "componentChunkName": "component---src-templates-publication-template-tsx", "path": "/publications/ai-information-gathering-army-mission-command", "result": {"data":{"markdownRemark":{"html":"
\n

TL;DR: In our nine-person study, we assessed two AI systems (an Army-tailored one and a general one) and found that AI-assisted search of Army doctrine showed no statistically significant accuracy advantage over PDF search, and correct answers took longer when we considered the two AI systems together. Participants reached correct answers faster with the Army-tailored system than with the general AI system but were less accurate overall, so evaluating the tools required looking at both speed and accuracy.

\n
\n

Finding useful guidance in time

\n

Army doctrine can be hard to search even when the needed documents are at hand. Guidance may be spread across publications, use unfamiliar terms, or apply to several military functions. Army staff have to find it in time to use it, and an incorrect answer can undermine the decision it informs.

\n

Artificial intelligence (AI) might reduce that effort by searching across documents and recognizing related terms. We tested this possibility in a controlled study described in our 2020 technical report. We wanted to know whether an early Army-tailored AI search system could help people answer doctrine questions more efficiently than familiar PDF search.

\n

Searching the same documents three ways

\n

We compared three search methods using the same collection of Army doctrine. For PDF search, participants searched doctrine files stored in a computer folder. The general AI system was a commercial tool already trained on general knowledge. It indexed those same documents and gave participants a search interface.

\n

The Army-tailored version incorporated Army-relevant language and had filters for categories such as warfighting function, echelon, and document type. Both AI systems used the same machine learning and natural language processing algorithms to analyze the doctrine and build searchable indexes. Natural language processing helps software work with human language. Army tailoring was meant to handle differences in terminology: a search for \"military decision making process\" could also retrieve references to \"MDMP.\"

\n

The screenshot above is the Army system's search display, reproduced from Figure 2 of the report. Participants had to use the information it returned to answer a question. We were measuring how people performed with the complete search tool, including its interface.

\n

Nine participants tried all three methods

\n

Eight retired Army officers and one Army civilian took part at the Mission Command Battle Lab at Fort Leavenworth, KS. Each person used every search method, so we could compare their performance across methods. Before testing, all nine attended a three-hour introduction to the AI systems and their search functions.

\n

They then attempted to answer 90 questions across three sessions over two days, using one search method for each set of 30 questions. Active-duty and retired officers developed the questions around common mission command information needs. One asked, \"What are the staff inputs to the Battle Update Brief?\" We counterbalanced the order of the search methods so that each appeared equally often in each session position.

\n

Custom software recorded answer time and confidence. Participants wrote their answers in a separate document. Two or three coders assessed accuracy and resolved disagreements through discussion. After each session, participants completed the NASA Task Load Index, a questionnaire about perceived workload, and the System Usability Scale, which asks users to assess a system's usability.

\n

Computer problems left 295 of the 810 possible question-level observations unavailable; another 31 lacked accuracy ratings. That left 484 observations for the accuracy analysis. For timing, we included only the 311 correct answers. A quick response would not count as successful information gathering if the answer was wrong.

\n

Faster with the Army system, less accurate overall

\n

Our analyses accounted for differences among participants and questions. Participants found correct answers faster with PDF search than with the two AI systems considered together (p = .006). We attribute this difference to the slower general AI system search. The Army-tailored system search was faster than the general system search (p < .001).

\n

After accounting for answer time, participants were less accurate with the Army-tailored system than with the general AI system (p < .001). The same analysis found no statistically significant accuracy advantage for the two AI systems together over PDF search (p = .311).

\n

\n \n \n \n \n \n \n \n \n

\n

A speed score alone would have missed the Army-tailored system's lower accuracy. We could not establish why these outcomes differed or whether participants deliberately traded accuracy for speed.

\n

Search method did not significantly explain answer confidence either. Participants' reports of workload and usability showed no significant differences among methods. Those ratings describe their experiences with the tools. They do not establish that the systems were equally usable or that performance was equivalent.

\n

What a small study leaves open

\n

These results apply to the systems and tasks we tested with this group. Nine participants gave us enough statistical power to detect large effects, but smaller benefits could have gone undetected. The group also differed from the intended users, active-duty soldiers. Experienced retirees may already have known where to find doctrine and had less need for help searching it.

\n

Training and practice were limited. Participants might have needed more time to become proficient, and straightforward doctrine questions may have missed uses where AI would help more, such as complex searches with several possible answers. We also could not separate the AI's contribution from the interface people used to work with it.

\n

Full Citation

\n

Stothart, C. R., Burland, B. R., Strickland, H. C., Messina, F. D., Couch, D. S., & From, J. D. (2020). Can artificial intelligence systems improve information-gathering efficiency in Army mission command processes? (Technical Report 1382). U.S. Army Research Institute for the Behavioral and Social Sciences. Publication record.

","tableOfContents":"","frontmatter":{"title":"Can Artificial Intelligence Systems Improve Information-Gathering Efficiency in Army Mission Command Processes?","description":"A small Army study found no overall search advantage from two AI tools. An Army-tailored AI tool was faster but less accurate than a general AI tool.","featuredImage":{"childImageSharp":{"gatsbyImageData":{"layout":"constrained","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAIAAAAmMtkJAAAACXBIWXMAAAsSAAALEgHS3X78AAAB8ElEQVQoz32SzU4UQRDH58GMSOLzCOrBq/ECiSdvHvQguPACJhDF9WCispsYVFhAHsAdenemu+ejq6qrqs3MIBIP/lLpdHf+lfrMVh/srqzv3F4brayNbt17vfbszcard49f7D15ubex9X5ze7y5Pd7YGj8dfXj0fP/O+mgQr97fuftwN5v+OD/8djY9vpgcX3w+Oh9PZ2+/nIynp+Pp2cHk9K8dzg4OZ5+OziffzybHP6cnFx+/zrKkkkRSj2piFvnz/AfiBKQxsqoOPxkANG0bmSMLICJ0x03FNYAEiMbBwkMDkZmztmkWiwUzp5SQ6HJpc1MsC+erqqrqgaaHmVUEACmKduKY1XXtvR/iEKEty7ZtAcJNt6ZtB+eU1Flry1I1iUiGiCEEEVFVAKybtg0YgJg5xt74ikikqs7Z+XweY+ycIYSmboaqEDDPTVHapg2dligSxf4yoKrWlsYY6cm6GohSSqJKSACEUSgKkADxYNc9k65muG5QhgDeV8zdeABwfmlK63xVu6p2vnK+dr6um7ovvBVh731RFKo6pA3GmBhjSoljJID0X8qyyPN8yKKLDIDDhgCQWdrLRfkrX+Zmaa313ltrnfOhk4GIOOdMP9reOXRc7RChd7bfFIAAIQTE7gSA2MMiiIRIzIIYfwNsf9Fz+wjDFwAAAABJRU5ErkJggg=="},"images":{"fallback":{"src":"/static/7a31d59636c29e064ca4a61aab75a50e/77ca0/army-ai-doctrine-search-interface.png","srcSet":"/static/7a31d59636c29e064ca4a61aab75a50e/5cf5a/army-ai-doctrine-search-interface.png 298w,\n/static/7a31d59636c29e064ca4a61aab75a50e/b6512/army-ai-doctrine-search-interface.png 595w,\n/static/7a31d59636c29e064ca4a61aab75a50e/77ca0/army-ai-doctrine-search-interface.png 1190w","sizes":"(min-width: 1190px) 1190px, 100vw"},"sources":[{"srcSet":"/static/7a31d59636c29e064ca4a61aab75a50e/932f5/army-ai-doctrine-search-interface.webp 298w,\n/static/7a31d59636c29e064ca4a61aab75a50e/091be/army-ai-doctrine-search-interface.webp 595w,\n/static/7a31d59636c29e064ca4a61aab75a50e/fe2dc/army-ai-doctrine-search-interface.webp 1190w","type":"image/webp","sizes":"(min-width: 1190px) 1190px, 100vw"}]},"width":1400,"height":929.4117647058823}}}}},"allStothartPublicationsJson":{"nodes":[{"title":"Overcoming Cognition-Limiting Heuristics in Mission Planning","summaryPath":"/publications/overcoming-cognition-limiting-heuristics-in-mission-planning","reportNumber":"ARI TR 1471","year":2025,"publisher":"U.S. Army Research Institute for the Behavioral and Social Sciences","publisherUrl":"https://www.dtic.mil/document;accessionNumber=AD1346348;type=DTICTR","downloadUrl":"https://www.researchgate.net/publication/399721342_Overcoming_Cognition-Limiting_Heuristics_in_Mission_Planning","podcast":null},{"title":"Thinking-in-Time","summaryPath":"/publications/developing-thinking-in-time-skills","reportNumber":"ARI-SSSR 2024-31","year":2024,"publisher":"U.S. Army Research Institute for the Behavioral and Social Sciences","publisherUrl":"https://www.dtic.mil/document;accessionNumber=AD1306281;type=DTICTR","downloadUrl":"https://www.researchgate.net/publication/394451644_Thinking_in_Time_A_Scenario-Based_Developmental_Method_for_Army_Officers","podcast":null},{"title":"Active Inference","summaryPath":"/publications/active-inference-decision-making-uncertainty","reportNumber":"Scientific Report 2024-09","year":2024,"publisher":"U.S. Army Research Institute for the Behavioral and Social Sciences","publisherUrl":"https://apps.dtic.mil/sti/html/trecms/AD1226410/index.html","downloadUrl":"https://www.researchgate.net/publication/394451646_Active_Inference_A_Competency_for_Making_Decisions_in_Uncertain_Situations","podcast":null},{"title":"Active Listening Skills Training","summaryPath":null,"reportNumber":"Scientific Report 2024-18","year":2024,"publisher":"U.S. Army Research Institute for the Behavioral and Social Sciences","publisherUrl":"https://apps.dtic.mil/sti/html/trecms/AD1230849/index.html","downloadUrl":"https://www.researchgate.net/publication/394451827_Active_Listening_Skills_Training_Improving_Officer-_Soldier_Relationships_in_Developmental_Counseling","podcast":null},{"title":"Can Artificial Intelligence Systems Improve Information-Gathering Efficiency in Army Mission Command Processes?","summaryPath":"/publications/ai-information-gathering-army-mission-command","reportNumber":"Technical Report 1382","year":2020,"publisher":"U.S. Army Research Institute for the Behavioral and Social Sciences","publisherUrl":"https://apps.dtic.mil/sti/citations/AD1107156","downloadUrl":"https://www.researchgate.net/publication/344405669_Can_Artificial_Intelligence_Systems_Improve_Information-Gathering_Efficiency_in_Army_Mission_Command_Processes","podcast":null},{"title":"Productive Discourse to Enhance Army Strategic Planning","summaryPath":null,"reportNumber":"Technical Report 1384","year":2020,"publisher":"U.S. Army Research Institute for the Behavioral and Social Sciences","publisherUrl":"https://apps.dtic.mil/sti/citations/AD1107160","downloadUrl":"https://www.researchgate.net/publication/344405821_Productive_Discourse_to_Enhance_Army_Strategic_Planning","podcast":null},{"title":"Human Factors Guidelines to Develop Educational Tip Cards for Aging Road Users","summaryPath":null,"reportNumber":"BDV30 977-15","year":2017,"publisher":"Florida Department of Transportation","publisherUrl":null,"downloadUrl":"https://www.researchgate.net/publication/319448916_Human_Factors_Guidelines_to_Develop_Educational_Tip_Cards_for_Aging_Road_Users","podcast":null},{"title":"Driving Simulator Studies of the Effectiveness of Countermeasures to Prevent Wrong-Way Crashes","summaryPath":null,"reportNumber":"BDV30-977-10","year":2015,"publisher":"Florida Department of Transportation","publisherUrl":null,"downloadUrl":"https://www.researchgate.net/publication/319448611_Driving_Simulator_Studies_of_the_Effectiveness_of_Countermeasures_to_Prevent_Wrong-Way_Crashes_BDV30-977-10","podcast":null},{"title":"Aging Road User Studies of Intersection Safety","summaryPath":null,"reportNumber":"BDV30 TWO 977-04","year":2014,"publisher":"Florida Department of Transportation","publisherUrl":null,"downloadUrl":"https://www.researchgate.net/publication/319448724_Aging_Road_Studies_of_Intersection_Safety_BDV30_TWO_977-04","podcast":null},{"title":"Aging Road User, Bicyclist, and Pedestrian Safety","summaryPath":null,"reportNumber":"BDK83 977-15","year":2013,"publisher":"Florida Department of Transportation","publisherUrl":null,"downloadUrl":"https://www.researchgate.net/publication/319448505_Aging_Road_User_Bicyclist_and_Pedestrian_Safety_Effective_Bicycling_Signs_and_Preventing_Left-Turn_Crashes_BDK83_977-15","podcast":null},{"title":"Aging Driver and Pedestrian Safety","summaryPath":null,"reportNumber":"BDK83 977-12","year":2012,"publisher":"Florida Department of Transportation","publisherUrl":null,"downloadUrl":"https://www.researchgate.net/publication/319448395_Aging_Driver_and_Pedestrian_Safety_Parking_Lot_Hazards_Study_BDK83_977-12","podcast":null},{"title":"Aging Driver and Pedestrian Safety","summaryPath":null,"reportNumber":"BDK83 977-09","year":2011,"publisher":"Florida Department of Transportation","publisherUrl":null,"downloadUrl":"https://www.researchgate.net/publication/281094963_Aging_Driver_and_Pedestrian_Safety_Human_Factors_Studies_BDK83_977-09","podcast":null},{"title":"Typicality Modulates the Visual Awareness of Objects","summaryPath":null,"reportNumber":null,"year":2022,"publisher":"consciousness and cognition","publisherUrl":"https://doi.org/10.1016/j.concog.2022.103314","downloadUrl":null,"podcast":null},{"title":"\"Satisfaction\" in Search","summaryPath":"/publications/satisfaction-in-search-expectations-predict-errors","reportNumber":null,"year":2019,"publisher":"attention perception and psychophysics","publisherUrl":"https://doi.org/10.3758/s13414-019-01790-9","downloadUrl":"https://www.researchgate.net/publication/333433720_Satisfaction_in_Search_Individuals'_Own_Search_Expectations_Predict_Their_Errors_in_Multiple-Target_Search","podcast":null},{"title":"What to Where","summaryPath":"/publications/what-to-where-right-attention-set-wrong-location","reportNumber":null,"year":2019,"publisher":"perception","publisherUrl":"https://doi.org/10.1177/0301006619854302","downloadUrl":"https://www.researchgate.net/publication/333224436_What_to_Where_The_Right_Attention_Set_for_the_Wrong_Location","podcast":null},{"title":"Semantic Associations Do Not Modulate the Visual Awareness of Objects","summaryPath":null,"reportNumber":null,"year":2019,"publisher":"the quarterly journal of experimental psychology","publisherUrl":"https://doi.org/10.1177/1747021818811411","downloadUrl":"https://www.researchgate.net/publication/328481618_Semantic_associations_do_not_modulate_the_visual_awareness_of_objects","podcast":null},{"title":"Predictors of Attitudes Towards Autonomous Vehicles","summaryPath":null,"reportNumber":null,"year":2018,"publisher":"frontiers in psychology","publisherUrl":"https://doi.org/10.3389/fpsyg.2018.02589","downloadUrl":"https://www.researchgate.net/publication/329806164_Predictors_of_Attitudes_Toward_Autonomous_Vehicles_The_Roles_of_Age_Gender_Prior_Knowledge_and_Personality","podcast":null},{"title":"Attention Capture, Processing Speed, and Inattentional Blindness","summaryPath":null,"reportNumber":null,"year":2018,"publisher":"acta psychologica","publisherUrl":"https://doi.org/10.1016/j.actpsy.2018.07.005","downloadUrl":"https://www.researchgate.net/publication/326356485_Attention_Capture_Processing_Speed_and_Inattentional_Blindness","podcast":null},{"title":"Satisfaction in Motion","summaryPath":null,"reportNumber":null,"year":2018,"publisher":"psychonomic bulletin and review","publisherUrl":"https://doi.org/10.3758/s13423-017-1300-1","downloadUrl":"https://www.researchgate.net/publication/316441769_Satisfaction_in_motion_Subsequent_search_misses_are_more_likely_in_moving_search_displays","podcast":null},{"title":"The Costs (or Benefits) Associated with Attended Objects Do Little to Influence Inattentional Blindness","summaryPath":null,"reportNumber":null,"year":2017,"publisher":"acta psychologica","publisherUrl":"https://doi.org/10.1016/j.actpsy.2016.12.012","downloadUrl":"https://www.researchgate.net/publication/311799778_The_costs_or_benefits_associated_with_attended_objects_do_little_to_influence_inattentional_blindness","podcast":null},{"title":"Clarifying the Role of Target Similarity, Task Relevance and Feature-Based Suppression During Sustained Inattentional Blindness","summaryPath":null,"reportNumber":null,"year":2016,"publisher":"journal of vision","publisherUrl":"https://doi.org/10.1167/16.15.13","downloadUrl":"https://www.researchgate.net/publication/311734195_Clarifying_the_role_of_target_similarity_task_relevance_and_feature-based_suppression_during_sustained_inattentional_blindness","podcast":null},{"title":"Using Mechanical Turk to Assess the Effects of Age and Spatial Proximity on Inattentional Blindness","summaryPath":null,"reportNumber":null,"year":2015,"publisher":"collabra","publisherUrl":"https://doi.org/10.1525/collabra.26","downloadUrl":"https://www.researchgate.net/publication/282425519_Using_Mechanical_Turk_to_Assess_the_Effects_of_Age_and_Spatial_Proximity_on_Inattentional_Blindness","podcast":null},{"title":"The Attentional Cost of Receiving a Cell Phone Notification","summaryPath":"/publications/attentional-cost-cell-phone-notification","reportNumber":null,"year":2015,"publisher":"journal of experimental psychology: human perception and performance","publisherUrl":"https://doi.org/10.1037/xhp0000100","downloadUrl":"https://www.researchgate.net/publication/279457726_The_Attentional_Cost_of_Receiving_a_Cell_Phone_Notification","podcast":"https://open.spotify.com/episode/72ClqMYneG8qwNiIpc9BnS?si=0J6oZaOiSDmc_DIzC_wxRQ"},{"title":"Is the Effect of Aerobic Exercise on Cognition a Placebo Effect?","summaryPath":null,"reportNumber":null,"year":2014,"publisher":"PLOS ONE","publisherUrl":"https://doi.org/10.1371/journal.pone.0109557","downloadUrl":"https://www.researchgate.net/publication/266625747_Is_the_Effect_of_Aerobic_Exercise_on_Cognition_a_Placebo_Effect","podcast":null},{"title":"The Pervasive Problem with Placebos in Psychology","summaryPath":"/publications/placebos-active-control-groups-expectations","reportNumber":null,"year":2013,"publisher":"perspectives on psychological science","publisherUrl":"https://doi.org/10.1177/1745691613491271","downloadUrl":"https://www.researchgate.net/publication/258179760_The_Pervasive_Problem_With_Placebos_in_Psychology_Why_Active_Control_Groups_Are_Not_Sufficient_to_Rule_Out_Placebo_Effects","podcast":null},{"title":"Improving the Safety of Aging Road Users","summaryPath":null,"reportNumber":null,"year":2013,"publisher":"gerontology","publisherUrl":"https://doi.org/10.1159/000354212","downloadUrl":"https://www.researchgate.net/publication/256289389_Improving_the_Safety_of_Aging_Road_Users_A_Mini-Review","podcast":null},{"title":"Age Effects on Inattentional Blindness","summaryPath":null,"reportNumber":null,"year":2016,"publisher":"Springer International Publishing","publisherUrl":null,"downloadUrl":"https://www.researchgate.net/publication/304189658_Age_Effects_on_Inattentional_Blindness_Implications_for_Driving","podcast":null},{"title":"Ensuring the Safety and Accessibility of Transportation for an Aging Population","summaryPath":null,"reportNumber":null,"year":2016,"publisher":"Springer International Publishing","publisherUrl":null,"downloadUrl":"https://www.researchgate.net/publication/304189962_Ensuring_the_Safety_and_Accessibility_of_Transportation_for_an_Aging_Population","podcast":null},{"title":"Active Listening Learning Resource","summaryPath":null,"reportNumber":"Research Product 2023-04","year":2023,"publisher":"U.S. Army Research Institute for the Behavioral and Social Sciences","publisherUrl":"https://apps.dtic.mil/sti/citations/AD1189571","downloadUrl":"https://www.researchgate.net/publication/394451659_Active_Listening_Learning_Resource_Self-Paced_Learning_Modules_to_Improve_Your_Ability_to_Communicate_Effectively","podcast":null},{"title":"Strategies for Stimulating Discussion Handbook","summaryPath":null,"reportNumber":"Research Product 2019-02","year":2019,"publisher":"U.S. Army Research Institute for the Behavioral and Social Sciences","publisherUrl":"https://apps.dtic.mil/sti/citations/AD1067297","downloadUrl":"https://www.researchgate.net/publication/394451664_Strategies_for_Stimulating_Discussion","podcast":null},{"title":"Using AI to Assist Commanders with Complex Decision-Making","summaryPath":null,"reportNumber":null,"year":2018,"publisher":"Author Published","publisherUrl":null,"downloadUrl":null,"podcast":null}]}},"pageContext":{"id":"f942b960-6d01-5fa9-b712-bafceb5da6ed","slug":"ai-information-gathering-army-mission-command","markdownPath":"/publications/ai-information-gathering-army-mission-command/index.md","seo":{"type":"article","title":"Can Artificial Intelligence Systems Improve Information-Gathering Efficiency in Army Mission Command Processes?","description":"A small Army study found no overall search advantage from two AI tools. An Army-tailored AI tool was faster but less accurate than a general AI tool.","image":"/static/7a31d59636c29e064ca4a61aab75a50e/army-ai-doctrine-search-interface.png","imageAlt":"Preview image for Can Artificial Intelligence Systems Improve Information-Gathering Efficiency in Army Mission Command Processes?","imageWidth":1190,"imageHeight":790}}}, "staticQueryHashes": ["3600524341"]}