In brief

Cisco Talos Incident Response describes lessons from its AI Tiger Team’s experiments with using large language models to draft cybersecurity reports. The team found that while LLMs can produce polished long-form technical writing, they often introduce inconsistencies in sourcing, conclusions, structure, and context management. These problems arise because models generate output probabilistically, may rely on changing or unclear source material, and can suffer from context drift or contamination when too much unrelated work is handled in the same session.

To reduce these risks, Talos tested prompt-specialization, strict source constraints, detailed output-format requirements, and template-guided prompting in a tabletop exercise report case study. The approach produced clear benefits, including an estimated 50% reduction in drafting time, stronger recommendations, and writing quality that passed blind review. However, the article stresses that production use requires careful data handling, model selection, input quality control, and human ownership of the final report. Talos also found that AI editing of full reports was not reliable enough for production, especially because grammar-checking results varied significantly between runs.

Source: Cisco Blogs

‍

‍