Extending the Rogo Big Finance Benchmark to Financial Data

Rogo’s Big Finance Bench (‘BFB’) was built to define what good looks like in financial AI: the questions financial professionals actually ask, and the standard a correct answer must meet.

We are now extending that framework to financial data, optimizing for quality & cost in two dimensions. At the source level, we refine each integration around the information and functionality that matter for real financial workflows, removing extraneous fields and encoding the structures models need to use the data correctly. At the ecosystem level, we evaluate how sources work together: which source is authoritative for a given task, how to resolve overlapping information, and what should enter the model’s context in the first place.

The objective is to move beyond data connectivity toward data intelligence: a system that is optimized for quality, speed, and token efficiency for our customers.

The Data Problem

Two years ago, one of the central challenges in financial AI was simply getting models access to financial data. Valuable information sat behind terminals, proprietary databases, APIs, and bespoke integrations, and connecting each source required meaningful engineering work.

That is changing quickly. APIs have become easier to integrate, while MCP gives data providers a common protocol for making their information available to LLMs. Instead of building a bespoke integration for every application, providers can increasingly expose their data across general-purpose AI tools, internal systems, and vertical platforms like Rogo. This is an important step forward, but in finance, access to data is not the same as using that data well.

Financial data creates two distinct problems for AI systems: First, the data itself is unusually sensitive to context. Second, as more sources become available, the system should be able to decide which data should enter the model’s context at all.

Rogo has spent the last several years working on both.

Financial data needs interpretation, not just access

Consider something as simple as revenue. A single provider might contain many versions of the same revenue figure: different fiscal periods, currencies, reported and adjusted values, restatements, originally filed values, provider-specific calculations, and so many more. And, while a financial professional navigates these idiosyncrasies with ease, a model has to be taught which distinction matters for the task at hand.

The interfaces exposed by financial data providers often preserve much of this underlying complexity. Tool definitions, metadata, filters, fields, and provider-specific structures are necessary for the underlying database, but they are not necessarily the right interface for an agent to surface.

This is why provider APIs or MCPs are not a plug and play.

For each source, our data team works with financial professionals to determine which information matters for the workflows our customers actually perform. We build evaluations around those workflows, run tests across all the models available in Rogo as well as various data integration configurations, to make sure the newly integrated data works for every user. For a credit dataset, for example, we evaluate the integration against the work a credit professional actually performs rather than simply testing whether the model can call the API successfully.

That evaluation is rigorous. We encode financial definitions explicitly, normalize periods, currencies, and scale, and test the integration across real financial workflows. The results show us precisely where answer quality breaks down, allowing us to improve how Rogo retrieves, interprets, and applies each source, regardless of the underlying model.

The impact is measurable. Across the sources we’ve optimized, we see 75% fewer seriously flawed answers, 77% fewer false “no data” responses, and 40% faster answers.

This benefits both sides of the ecosystem. Customers are more likely to get the right data when they need it, while data providers are more likely to have their information surfaced in the workflows where it is actually valuable.

The data entropy problem

Improving individual connectors solves only part of the problem. The next challenge appears as the number of available sources grows.

One connector is relatively simple. Ten begin to overlap. A financial institution can eventually have hundreds of sources spanning market data, filings, research, CRM systems, internal files, deal data, email, and other proprietary information. Not every question requires all of them.

If every connected system exposes every available tool whenever a user asks a question, the model has to spend an increasing share of its context understanding what information is available before it can reason about the information that actually matters. We think of this as the data entropy problem.

We have solved this through Felix, Rogo’s agent harness, which dynamically determines which tools should be available to the model for a particular task. If someone is analyzing a public company’s historical financials, the model does not need to carry every CRM, email, file-storage, and deal-data tool while performing that analysis. Those sources remain accessible, but they do not consume context unless the task requires them.

The efficiency gains can be substantial. In one evaluation, optimizing Rogo’s integration of an MCP reduced the tokens consumed by tool definitions by roughly 4x. That context could instead be used for the real ‘thinking’ required to complete the task.

More importantly, this architecture scales - which is increasingly important as our customers move from a handful of integrations to hundreds.

What comes after connectivity

The first generation of financial AI infrastructure was concerned with access: Can the model reach the data? The next generation has a harder problem: Can the model consistently find the right data, interpret it correctly, and use it efficiently?

As we expand Rogo’s financial data ecosystem, we are optimizing around two constituencies.

For our customers, the objective is to deliver the right information for the task with high fidelity, low latency, and efficient context & token usage. For our data partners, the objective is equally important: ensure valuable proprietary information is surfaced when it is genuinely relevant, rather than buried inside an ever-growing universe of competing tools and sources.

As data access scales, Rogo’s advantage lies in the financial expertise applied to integrating, structuring, and evaluating that data for real financial workflows.