Lightweight LLMs decrease incident response time using decision theoretic planning

Researchers from the University of Melbourne and Imperial College London have developed a novel method for enhancing incident response planning using Large Language Models (LLMs), with a particular emphasis on minimising the risk of hallucinations. Their approach utilises a smaller, fine-tuned LLM in conjunction with retrieval-augmented generation and decision-theoretic planning. The method addresses the prevalent issue of incident response being predominantly manual, slow, and reliant on expert-configured playbooks, which can lead to prolonged recovery times for organisations. Kim Hammar, one of the authors, highlighted that the system is designed to integrate seamlessly into existing workflows without necessitating additional software or modifications to current systems. The method accepts log data and threat information in raw textual form, eliminating the need for specific syntax or formatting.

The incident response planning method comprises three key steps. First, the team fine-tunes a 14-billion-parameter LLM on a dataset of 68,000 historical incidents, each paired with corresponding response plans and reasoning steps. This process aligns the model with the phases and objectives of incident response while maintaining flexibility across various scenarios. Second, the system retrieves relevant threat intelligence and vulnerability data based on indicators from system logs, enabling it to adapt to emerging threats. Finally, rather than executing the first suggested action, the system generates multiple candidate actions and simulates potential outcomes using the LLM. It selects the action predicted to yield the fastest recovery, filtering out responses that do not contribute to progress. Hammar noted that this method functions as a more adaptive playbook, guiding security operators to validate suggested actions against available evidence rather than treating them as definitive solutions. The paper also presents a probabilistic analysis demonstrating that the likelihood of hallucination can be minimised, offering a formal basis for the method’s reliability compared to prompt-only frontier LLMs.