Open Clinical AI
Menu

ChartSearchAI

Answer paths and their evaluation

ChartSearchAI can answer a clinician’s question about a patient chart with its own built-in pipeline, or it can pass the question to Med Agent Hub. This page explains how the two approaches differ, how our evaluations configure them so that the comparison is fair, and which differences in the results reflect genuine differences in capability.

Inspect published evaluations

Architecture

ChartSearchAI answer paths A clinician's question reaches the ChartSearchAI module in OpenMRS, which sends it either to the built-in pipeline or to Med Agent Hub. Both read patient records from QueryStore and call a language model; the Hub also uses WHO guidance and checks dates, citations and drug safety. A default installation runs the model inside the OpenMRS server; our evaluations run it on one shared server for both paths. A clinician asks about a patient ChartSearchAI module in OpenMRS Checks access, keeps the conversation and sends the question to one answer path Built-in ChartSearchAI pipeline Med Agent Hub Retrieve the chart from QueryStore Build one prompt Call the language model once Apply drug-safety checks and cite records Retrieve records and WHO guidance Call the language model one or more times Check dates, citations and drug safety Return the answer, confidence and In-Depth QueryStore patient records Language model Where the model runs: a default installation starts it inside the OpenMRS server (Gemma 4 E4B). In our evaluations both paths call one shared model server with a graphics chip, and every request is recorded.
The two answer paths, the shared record source and where the language model runs.

When ChartSearchAI answers a question with its built-in pipeline, it retrieves the patient’s records, writes a single prompt for a language model, sends that prompt once, applies its drug-safety checks to the reply and returns an answer that cites the records it used.

When ChartSearchAI uses Med Agent Hub, it passes the question to the Hub and displays the Hub’s response. The Hub retrieves the records itself, may call the model several times and checks the answer before and after it is shown. ChartSearchAI’s own answer pipeline is not involved.

In both cases OpenMRS controls which users may ask about which patients, keeps the conversation history and records which path produced each answer.

Model hosting

Where the built-in pipeline’s language model runs is a separate setting from which path answers. By default, ChartSearchAI starts the model inside the OpenMRS server; the recommended model is Gemma 4 E4B, a file of about 5 GB. A site can instead direct ChartSearchAI to a separate model server. This changes only where the single model call is sent. The answer is still produced by ChartSearchAI’s own pipeline.

ChartSearchAI’s Docker installation runs the model on the computer’s processor, because, as its documentation notes, Docker cannot use a graphics chip, even on a Mac. Its macOS installation runs natively and uses the Mac’s graphics chip.

Evaluation configuration

ChartSearchAI runs with its default settings, as an OpenMRS site receives it, including sending the whole patient chart (about 12,200 input tokens on our longitudinal HIV records) with every question. The only setting we change in engine-parity runs is the location of its language model.

Both paths use the same language model on the same model server, which runs on a computer with a graphics chip, so neither path benefits from a stronger model or faster hardware. Every Server-Sent Event frame is timestamped as it streams from OpenMRS, separating Time to First Token (TTFT) and initial answer completion (answer_done) from post-answer validation and In-Depth completion (turn_done).

Every request that either path sends to the model is recorded through an HTTP wire tap (engine-tap), so that we can verify exactly what prompt, sampler settings and cached tokens each path used. Because ChartSearchAI’s built-in LocalLlmEngine enables a repetition penalty (dry_multiplier = 0.8) whenever no drug-reference block is present in the prompt, the harness also supports an explicit out-of-the-box simulation mode (--simulate-local-engine-dry) to measure how that sampler setting interacts with long fullChart prompts separately from shared-engine parity.

Because the model runs on a shared server in parity runs, those runs isolate pipeline behavior from local CPU process startup; see the hardware and concurrency sizing guide for CPU vs. single-GPU vs. hospital GPU server benchmarks.

Capability differences

With both paths on their shipped settings, the following differences remain. They reflect what each path is able to do, not how we configured it.

CapabilityBuilt-in ChartSearchAI pipelineMed Agent Hub
Multi-turn conversation memoryBundledClinicalAnswerProvider sends only the current question to searchStreaming and drops prior turns, so follow-up questions in a conversation lose prior context.Retains prior clinical turns across the conversation session so follow-up questions resolve pronouns and prior findings accurately.
Awareness of the current dateThe prompt does not state the current or reference date, so questions about upcoming visits or the past 6 months can treat historical chart dates as current.The model receives the reference date and precomputed relative date distances for chart records.
Checking dates and time windows in the answerDates in the answer are not checked against the chart.Deterministic temporal and answer rules check date windows and lab/medication tables against the chart’s dated records; discrepancies are corrected or marked.
Out-of-the-box sampler behavior on long chartsWhen running through LocalLlmEngine without drug-reference records, dry_multiplier = 0.8 is injected over the ~12.2k-token chart prompt, which can penalize repeating legitimate chart dates and medication names.Uses clean temperature-0 structured generation without a chart-wide DRY repetition penalty.
Live main-answer token streamingStreams answer tokens live (answer_delta) during single-call generation.Streams main-answer tokens live (answer_delta) during synthesis before deterministic checks, evidence resolution and In-Depth complete.
Checking that citations support the answerThis check is available but turned off by default.Each cited record is checked, and an unsupported citation is labelled.
Clinical guideline sourcesNo guideline sources are available to the model.The Hub can cite records of WHO guidance from its knowledge base.
Drug-safety checksDrug-safety checks are applied.Drug-safety checks are applied.
Second review and supporting detailNeither is available.The checked configuration adds a second model review and an “In Depth” section, and labels its findings rather than hiding them.

Scoring and limitations

Each evaluation asks both paths the same questions and conversations about three HIV patient charts. A separate AI reviewer, working from a fixed scoring guide, scores every answer against the chart for accuracy, completeness, relevance and potential harm.

When the same answers have been scored twice, the averages have moved by up to about 0.6 points, so small differences between the paths should not be treated as meaningful. These are research findings, not product certifications, and every AI-generated answer requires review by a clinician.

Browse published evaluations Read how evaluations are run Hardware sizing and concurrency guide Read the ChartSearchAI installation guide