ESG Report Analyzer by Multimodal AI Agent
An automated ESG analytics AI Agent powered by OpenAI's o1 multimodal model, delivering superior, unbiased sustainability evaluations beyond human capability.
About
The problem it solves Environmental, Social, and Governance (ESG) reports are crucial for evaluating corporate sustainability and social responsibility. However, these documents often **contain dense, multi-format information** making extraction, analysis, and interpretation challenging and resource-intensive. This project addresses the complexity by implementing a Retrieval-Augmented Generation (RAG) pipeline using OpenAI’s multimodal model, o1. It automates the extraction and interpretation of critical ESG data from PDF reports by converting pages into images, summarizing content, and generating meaningful insights through advanced semantic understanding. The key innovation lies in effectively combining keyword-based and semantic search methods, refining results through reranking with Cohere’s model, and validating relevance visually with GPT-4o. This approach significantly reduces manual effort, increases accuracy, and provides rapid, scalable insights into corporate sustainability activities, making ESG data more actionable and accessible.
How it's made
Challenges we ran into Multimodal Data Complexity: Converting infographic PDF reports into embedding vector and accurately interpreting dense graphical and textual data required significant fine-tuning of the image-to-text summarization pipeline. I overcame this by iteratively adjusting the prompts and evaluating summaries through multiple rounds of feedback and testing. Optimizing Retrieval Accuracy: Balancing keyword-based search with semantic search initially produced inconsistent results. Implementing Cohere's reranker significantly improved accuracy but introduced additional latency. I resolved this by optimizing retrieval parameters and fine-tuning the reranking model thresholds, achieving a good balance between speed and relevance. Relevance Assessment: Evaluating relevance visually through GPT-4o occasionally led to misinterpretations due to ambiguous visual contexts. Addressing this involved enhancing the visual context provided to GPT-4o and incorporating additional context-checking mechanisms to improve consistency and reliability in relevance assessments.