Utilização de um LLM local com RAG para auxiliar a fase de extração de dados de uma revisão sistemática da literatura

Abstract

Systematic Literature Review (SLR) is a research methodology that follows specific protocols and is widely used in academic work to summarize and synthesize evidence on a given topic of study, with its application growing in the field of Software Engineering. However, conducting an SLR is laborious, as it requires significant time and human resources. With recent advances in Artificial Intelligence, tools such as Large Language Models (LLMs), Generative Pre-trained Transformer (GPT), for example, and Retrieval-Augmented Generation (RAG) offer opportunities to reduce the manual effort in conducting these reviews. This study aims to investigate whether the use of a local LLM augmented with RAG can assist the data extraction phase of a systematic review. To this end, the Llama 3.2 model was used to extract data from a systematic mapping study containing 22 SLR articles whose contents were provided to the LLM using the RAG technique, and the responses generated by the model were compared with those already extracted by the authors of the mapping. The local LLM augmented with RAG achieved approximately 42% correct answers, demonstrating that it offers limited assistance to the researcher in relation to the data extraction phase of the RSL; however, most of the correct answers concerned bibliographic data from the articles, suggesting that the model can be used to obtain this data more easily.