Asking a procurement database in plain language, and getting the evidence back
A multi-agent assistant that answers procurement questions from a live transactional database and an internal document library, with the evidence attached to every answer.
Client details removed
This engagement is covered by a non-disclosure agreement. The client, the product, the vendors and anything that could identify them have been intentionally removed or generalised. Everything described about the problem, the engineering and the outcome is real.
Executive summary
A heavy manufacturing group's buyers needed answers that lived in two incompatible places: a transactional procurement database and a large pile of internal documents. We built a conversational assistant on a multi-agent orchestration layer. One specialist translates a question into a validated read-only query, another retrieves from a hybrid document index, and a top-level agent decides which internal sources a given question actually needs. Every answer carries its evidence, whether that is the executed query and the rows it returned or the document page a passage came from. The same backend serves two front ends: a full internal console with analytics and knowledge-base management, and a chat-only surface for staff who simply want to ask.
Business context
The client is a heavy manufacturing group whose margin is set in large part by what it pays for raw material inputs. Buying decisions depend on facts spread across two systems that do not talk to each other. The first is a transactional database holding purchase records, supplier offers and inbound receipts, accumulated over years and named accordingly. The second is an unstructured library of specifications, internal procedures and vendor correspondence. Answering a question of the form "what did this input cost us, from whom, and does that match what the vendor committed to" meant querying the first and reading the second, and almost nobody in the organisation could do both. That gap, rather than any shortage of data, is what the engagement set out to close. It also fixed the hard constraint on the design: for a procurement team, an assistant that produces a confident number nobody can trace is worse than no assistant at all, because a number that looks authoritative is a number that gets used.
The challenge
Procurement questions were easy to ask and slow to answer. "What did we pay for this input last quarter, and to whom" meant someone opening a reporting tool, writing SQL against tables whose names encoded a decade of history, then cross-checking the result against offers and receipts that existed only as documents. The people who knew where the numbers lived were not the people who needed them, so requests queued behind an analyst. Self-service reporting had been tried and had not stuck, for a reason that turned out to matter later: the schema was not self-describing, and two of the most obvious-looking date columns meant entirely different things.
Objectives
- Let buyers ask procurement questions in plain language, in Indonesian or English
- Answer from the transactional database and the document library in the same turn
- Attach evidence to every answer: the executed query, or the page a passage came from
- Keep database access strictly read-only and schema-scoped, enforced in code
- Let staff add internal documents themselves, with no engineer in the loop
Solution architecture
We treated the question, not the data source, as the unit of work. A top-level agent decides which internal sources a question needs, then delegates: a query specialist that must profile the tables it intends to use before it writes anything, a retrieval specialist over the document index, and a summariser that assembles the final answer. Nothing the model writes reaches the database unchecked. Queries are read-only, scoped to a single schema, validated against both the original question and a stated query plan, and capped before execution. The assistant is also allowed to say that a question is not answerable from the exposed data, which turned out to be the behaviour that earned it trust.
End-to-end workflow
- User
A buyer asks a question
- AI
The assistant chooses its sources
Internal questions default to the transactional database and the document indexes together. An explicit restriction from the user overrides that, and the assistant may use only the relevant subset.
- AI
Tables are profiled before any query is written
The specialist reads the exposed schema and a reviewed business dictionary, then profiles one to three candidate tables. Profile samples are redacted and used only to choose tables, never to answer.
- System
The query is validated against the plan
A separate validation step checks the query against the original question and the stated plan. An invalid query gets one revision, and only then a second validation.
- System
Read-only execution under a row cap
Only select statements run, only inside one schema, and never above the row cap. This is enforced by the execution tool itself, not by asking the model to behave.
- AI
Documents are retrieved alongside
Passages come back with their document, section and page range, so a claim in the answer can be traced to the page it was read from.
- AI
The answer is assembled with its evidence
A table of rows, a short analyst note grounded only in those rows, and the sources. Where a chart helps, it is generated from the returned data rather than described in prose.
- System
The response streams to the surface that asked
Tokens are piped through without buffering, and the presentation policy of the calling surface decides how much of the machinery is shown to that particular audience.
Engineering challenges and decisions
Confident queries against a plausible but wrong table
The schema had grown over a decade, and several tables looked like reasonable homes for the same concept. Given only table and column names, the model would pick one that read plausibly, write a clean query against it, and return a well-formatted answer that was simply about the wrong thing. Nothing failed. The query executed, the numbers formatted, and the only signal that anything was wrong was someone who already knew the answer noticing.
Decision
We stopped expecting the model to infer meaning from names. A reviewed business dictionary now states what each table is for, and the specialist must profile its candidate tables before writing anything, then pass the query through a validation step that checks it against the original question and a written query plan. Choosing a table became an evidenced decision rather than a guess.
Trade-off
A single question now costs several model round trips before a row is read, which is slower and more expensive per answer. For a procurement team that was an easy trade: the cost of a wrong number reaching a purchasing decision is not paid in tokens.
A loading timestamp that read like a business date
One column recorded when a row was inserted by the loading process. It was well named, populated on every row, and completely wrong as an answer to "when did this happen". The model reached for it constantly, because it was the most reliably present date in the schema, and every trend it produced from that column was a chart of the loading job rather than of the business.
Decision
The column is now named and forbidden in the specialist's instructions, and the schema the assistant is shown exposes business date columns explicitly. When a question needs a date the schema does not carry, the assistant says so and asks, instead of substituting the nearest available column.
Trade-off
Some questions that used to return a chart now return a clarification. That felt like a regression in demos and was the opposite in practice, because an admitted gap is actionable and a plausible wrong trend is not.
Bulk document ingestion against a fixed embedding quota
Staff do not upload one document, they upload a folder. A large upload generated thousands of embedding calls at once, hit the provider rate limit within seconds, and the first version treated a rate limit like any other error: the document failed, and the user was told to try again with no idea whether trying again would help.
Decision
Indexing moved into a durable orchestration where extraction, chunking, embedding and index upsert are separate checkpointed steps. Embedding calls are batched, and a rate limit is retried with backoff inside the step instead of failing the document. A busy provider became a longer wait rather than a lost upload.
Trade-off
Indexing is no longer synchronous with the upload, so the interface needed an explicit per-document status and a way to see why something is still pending. That surface did not exist before and had to be built.
Retrieved fragments that arrived without their context
Fixed-size chunking cut procedures in half and split tables away from the headings that gave them meaning. A retrieved passage would be topically correct and practically useless, and the figures that carried much of the information in these documents were invisible to a text index entirely, so a question whose answer was a chart on page nine simply found nothing.
Decision
Chunking became page-aware and boundary-aware, keeping section titles and table structure with the text they belong to, and retrieval now expands into neighbouring chunks before answering. Figures are extracted during ingestion, stored as cropped images, indexed separately, and rendered inline in the conversation through an authenticated route.
Trade-off
Two indexes over one document set means two things that can drift apart, so deletion had to become a lifecycle rather than a delete: mark, remove from both indexes, then remove the stored file.
One answer, two audiences
The internal console and the chat-only surface share a backend, and for a while they shared everything else too. Showing the executed query is exactly right for the console, where it is the evidence, and exactly wrong on the surface aimed at buyers, where a block of SQL reads as noise at best and as something to copy at worst.
Decision
The calling surface passes a presentation policy that overrides the agent's default output rules. The reasoning, the tools and the evidence gathering are identical; only what is rendered differs. Answer quality stayed one thing to improve rather than two.
Trade-off
Behaviour now depends on a caller-supplied policy, which is one more thing that can be set wrongly. It is worth it: the alternative was two prompt sets drifting apart, one of them always the stale one.
Results and lessons
- 9Specialised agents
- 33Registered tools
- 3Retrieval indexes
- 2Delivery surfaces
Buyers ask in their own language and get a table back with the executed query or the source page attached, in one turn rather than as a request to an analyst. Internal documents became self-served: staff upload a specification or a procedure and it becomes searchable inside the same conversation once indexing finishes. Delivery ran from April to July 2026 across roughly thirty merged changes, and most of those changes tightened answer quality rather than adding new surface.
The lasting result is not the chat window. It is that the meaning of the procurement schema now lives in a reviewed artefact instead of in two people's heads: which table means what, which date column is a business date, which one is a by-product of the loading process. That artefact is what makes the assistant correct, and it keeps its value whether or not a conversation stays the interface.
Lessons learned
- A schema that does not describe itself has to be described before a model can query it
- Validating a query before running it is cheaper than a confident wrong answer
- Read-only and single-schema are code boundaries; an instruction in a prompt is not one
- Let the assistant admit a gap, and people start trusting the answers that do come back
Production considerations
Authentication is handled by the corporate directory at the edge, and the browser never holds a model or database credential: the front end talks only to its own backend for frontend, which injects the service credential inward and streams the response back. Database access from the assistant is read-only and confined to one schema, enforced in the execution tool rather than requested in a prompt. Uploaded documents carry an explicit lifecycle, so removing one marks it, clears it from both indexes, and only then deletes the stored file. Usage events are recorded per conversation so answer quality can be reviewed against real questions rather than against a demo script, and every agent is a versioned prompt file, which means a change in behaviour is a reviewable diff.
What we would do next
The clarification requests are the most useful unread dataset in the system. Every time the assistant says a question is not answerable from the exposed schema, that is a precise statement of a concept the business cares about and the data model does not represent. Collected over a quarter, they would form a better backlog for the data warehouse than any workshop. The second thread is the business dictionary itself: it is maintained by hand today, and the profiling step already produces most of what it would take to propose entries automatically and have a human confirm them. Third, the evidence tables and the retrieved passages currently answer the question that was asked, and much less often the follow-up the answer implies.
Technologies
10- Azure OpenAI
- Microsoft Agent Framework
- Azure AI Search
- Azure Durable Functions
- Azure Document Intelligence
- FastAPI
- Next.js
- Cosmos DB
- RAG
- Text-to-SQL