Análise comparativa de modelos em sistemas multi-agentes para geração de documentos analíticos.

Abstract

The evolution of Large Language Models has driven significant advances in the automation of intellectual tasks. However, the reliability of their responses and their tendency to produce hallucinations remain relevant limitations for their use in professional contexts. This work presents a detailed comparative analysis of the performance of three models: GPT-4o-mini, Llama 3–8B, and Llama 3–70B. These models were applied to two distinct case studies: (i) the validation of an academic text with a focus on detecting inconsistencies and hallucinations, and (ii) the comparative financial analysis of multiple balance sheets. To conduct the experiments, a multi-agent system was developed using the CrewAI framework to orchestrate autonomous agents responsible for generating, reviewing, and validating documents. The methodology employs a structured verification pipeline based on manually created checklists, allowing the comparison of model outputs against the original documents and enabling the assessment of their accuracy. The results reveal significant differences among the three models in terms of content fidelity, textual clarity, analytical consistency, and hallucination rates, demonstrating that proprietary and open-source models exhibit distinct behaviors depending on the nature of the task.