# Towards a standardized methodology and dataset for evaluating LLM-based digital forensic timeline analysis

Canonical URL: https://markscanlon.co/publications/LLM-based-Digital-Forensic-Timeline-Analysis

CSL-JSON: https://markscanlon.co/publications/LLM-based-Digital-Forensic-Timeline-Analysis.csl.json
BibTeX: https://markscanlon.co/publications/LLM-based-Digital-Forensic-Timeline-Analysis.bib
RIS: https://markscanlon.co/publications/LLM-based-Digital-Forensic-Timeline-Analysis.ris

Authors: Hudan Studiawan; Frank Breitinger; Mark Scanlon
Venue: Forensic Science International: Digital Investigation
Year: 2025
DOI: https://doi.org/10.1016/j.fsidi.2025.301982
PDF: https://markscanlon.co/publications/LLM-based-Digital-Forensic-Timeline-Analysis.pdf
Full text: https://markscanlon.co/publications/LLM-based-Digital-Forensic-Timeline-Analysis.full.md

## Contribution Summary

This paper addresses the need for a standardized approach to evaluate the performance of Large Language Models (LLMs) in digital forensic timeline analysis tasks. The proposed methodology includes a dataset, timeline generation, and ground truth development, and recommends the use of BLEU and ROUGE metrics for quantitative evaluation. The paper presents a case study using ChatGPT and demonstrates the effectiveness of the proposed methodology in evaluating LLM-based forensic timeline analysis. The study also discusses the limitations of applying LLMs to forensic timeline analysis and highlights the importance of maintaining the 'AI-assisted investigation' and 'human-in-the-loop' mantras when using LLMs in digital forensics.

## Abstract

Large language models (LLMs) have widespread adoption in many domains, including digital forensics. While prior research has largely centered on case studies and examples demonstrating how LLMs can assist forensic investigations, deeper explorations remain limited, i.e., a standardized approach for precise performance evaluations is lacking. Inspired by the NIST Computer Forensic Tool Testing Program, this paper proposes a standardized methodology to quantitatively evaluate the application of LLMs for digital forensic tasks, specifically in timeline analysis. The paper describes the components of the methodology, including the dataset, timeline generation, and ground truth development. In addition, the paper recommends the use of BLEU and ROUGE metrics for the quantitative evaluation of LLMs through case studies or tasks involving timeline analysis. Experimental results using ChatGPT demonstrate that the proposed methodology can effectively evaluate LLM-based forensic timeline analysis. Finally, we discuss the limitations of applying LLMs to forensic timeline analysis.