Heading 1
Heading 2
Heading 3
Heading 4
Heading 5
Heading 6
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
Block quote
Ordered list
- Item 1
- Item 2
- Item 3
Unordered list
- Item A
- Item B
- Item C
Bold text
Emphasis
Superscript
Subscript
Medical AI has spent years talking about “ground truth”.
The phrase suggests there is one correct answer sitting inside an image, waiting for an annotator to reveal it.
Medicine is rarely that simple.
In 2024, the updated Checklist for Artificial Intelligence in Medical Imaging (CLAIM) recommended using the term “reference standard” instead of “ground truth”. The distinction matters because biomedical labels can contain uncertainty.
The same guidance asks researchers to explain how a reference standard was created, including who annotated the data, their qualifications and subspecialty training, the instructions they received and how disagreement between readers was measured or resolved.
For medical AI teams, this raises an important annotation question.
It is no longer simply:
Who can draw the label?
It becomes:
How do you turn a clinically defined reference standard into annotations that can be applied consistently across hundreds or thousands of cases?
The responsibility split matters here.
The client’s clinical researchers and medical officers define the clinical hypothesis, reference standard, diagnostic thresholds and medical criteria.
The data partner’s responsibility is to make those specifications executable at scale.
At IngeData, that means translating client-defined clinical requirements into rigorous annotation guidelines, pairing the right qualified specialists with calibrated workflows, and building the quality controls needed to deliver precise, consistent and traceable data.
A radiologist does not automatically make a dataset consistent
Using qualified medical specialists is important.
It does not automatically guarantee that multiple people will interpret the same annotation task in exactly the same way.
Research into radiologist annotations in the CheXpert test set found that agreement varied considerably between findings. Agreement exceeded 90% for some categories, such as pneumothorax and medical devices, while labels for pneumonia and consolidation were much less stable.
For lower-agreement classes, the eventual label could depend heavily on which group of radiologists happened to annotate the images.
That distinction matters for AI teams.
Hiring a specialist answers one question:
Does this person have the knowledge needed to apply the clinical specification correctly?
It does not automatically answer another:
Will multiple specialists apply that specification in the same way?
That requires a calibrated workflow around the expertise.
For judgement-heavy annotation, the operation may need:
- explicit inclusion and exclusion criteria
- representative examples for calibration
- agreed annotation rules
- criteria for escalating uncertain cases
- measurement of inter-annotator agreement
- documentation of important decisions
- a way to incorporate client-approved clarifications into the wider guidance
Clinical expertise is therefore one input into reliable medical data.
The way that expertise is operationalised determines whether the standard can hold across the dataset.
Start with the clinical specification, then build the workflow
The question “Do we need radiologists?” can be too broad.
Medical-imaging workflows contain different types of work.
One task may involve tracing a structure whose clinical definition has already been established by the client.
Another may require applying predefined rules across different imaging conditions.
Another may concern acquisition quality, imaging planes, laterality or series structure.
And another may expose a case that falls outside the existing clinical specification entirely.
Those situations do not necessarily require the same person or the same escalation path.
A more useful model separates clinical ownership from operational execution.
Clinical specification
The client’s clinical team defines the clinical hypothesis, target structures, diagnostic thresholds, inclusion and exclusion criteria and intended reference standard.
Codification of annotation protocol
IngeData translates those specifications into operational annotation guidance: rules, representative examples, expected outputs, edge-case handling and escalation criteria.
Calibration
Qualified specialists and annotators are calibrated against the annotation protocol so differences in interpretation can be identified before production scales.
Production
The team applies the approved guidance consistently across the dataset using manual, AI-assisted or hybrid annotation workflows as appropriate.
Escalation
Cases that fall outside the protocol follow the agreed escalation route back to the client-designated clinical authority.
Quality assurance
Outputs are reviewed for protocol adherence, precision, consistency and inter-annotator agreement, with relevant decisions and guideline versions kept traceable.
This is a more precise model of expert-led medical annotation.
The objective is to use specialist expertise within a controlled system that faithfully executes the client’s clinical specification.
A technically precise contour can still fail the clinical specification
Cardiac CT makes this distinction tangible.
In one IngeData project with Lightsource Research, six cardiac radiologists worked across 300 coronary CT angiography scans.
The annotation task involved producing masks for the artery, lumen and coronary wall, alongside segmentation of calcified, non-calcified and low-attenuation plaques.
Correct Hounsfield Unit windowing was also used to differentiate artery walls, lumen and plaque tissue.
This is where medical annotation becomes more than drawing a geometrically neat shape.
A polygon can be smooth.
Its edges can be precise.
It can still represent the wrong structure or fail to meet the client-defined annotation criteria.
That is why medical-image annotation quality cannot be assessed only by asking whether the boundary was drawn accurately.
You also have to ask:
Was the client’s clinical specification translated and applied correctly?
That is what gives the annotation value as medical training or evaluation data.
Difficult cases need a defined escalation path
Some cases sit comfortably inside an established specification.
If an image clearly matches the annotation rules and representative examples used during calibration, the annotation can move through normal production.
Other cases expose ambiguity.
Examples might include:
- anatomy outside the expected presentation
- a borderline lesion
- disagreement between annotators
- image quality that makes the normal rule difficult to apply
- a finding that appears to fit more than one defined category
- a case that exposes a gap in the current instructions
The important part is what happens next.
The annotation team should not invent a new clinical rule.
The case should leave normal production and follow the agreed escalation path.
The relevant client clinical authority can then clarify the specification or determine how the case should be handled.
IngeData’s role is to capture that clarification in the annotation protocol, recalibrate the team where necessary and ensure the approved guidance is applied consistently to subsequent cases.
That is how client-defined clinical decisions become repeatable production guidance.
Disagreement can expose a weakness in the protocol
Reader disagreement is useful information.
When qualified specialists repeatedly interpret the same instruction differently, the issue may sit in the operational guidance rather than in an individual annotation.
The category may need a clearer definition.
The annotation may need more representative examples.
The escalation threshold may need refining.
Or the underlying clinical specification may require clarification from the client.
CLAIM’s 2024 guidance reflects this concern by asking researchers to describe inter- and intrarater variability and how discrepancies between annotators were mitigated or resolved.
For a data partner, the role is clear:
detect the pattern, document it, escalate it and operationalise the client-approved resolution.
That matters before thousands more cases are produced using an instruction that different people interpret differently.
“If two qualified specialists disagree, the useful question is what that disagreement tells us about the clarity of the protocol. Our role is to surface that signal, take it through the agreed clinical escalation path and make sure the clarified rule is applied consistently across the dataset.”

AI-assisted annotation changes where human expertise is applied
Medical-imaging teams increasingly have access to tools that can generate or assist with labels rather than requiring every contour to begin manually.
MONAI Label, for example, supports interactive and automated segmentation for medical images, allowing human users to work with AI-generated annotations.
Recent research on radiology image annotation also describes a broader move from fully manual methods towards semi-automated approaches, including report-derived labels and foundation image models.
This changes how human expertise is used within the production workflow.
If a system generates an initial segmentation, specialist effort can increasingly focus on:
- validating outputs against the client-defined protocol
- correcting difficult annotations covered by the protocol
- identifying recurring disagreements
- flagging new edge cases
- escalating cases that require clinical clarification
- verifying that production remains aligned with the approved standard
The important question therefore moves from:
Who manually labels every image?
towards:
How is the client-defined clinical standard translated, applied, quality-controlled and escalated at scale?
That is a more scalable use of specialist expertise and a clearer division between clinical authority and data production.
A practical model for operationalising medical annotation at scale
A medical-data workflow can be reduced to six stages.
1. Define: client clinical team
The client defines the clinical hypothesis, reference standard, diagnostic thresholds and inclusion or exclusion criteria.
2. Codify: IngeData
IngeData translates those specifications into rigorous annotation guidelines, including operational rules, representative examples, edge cases and escalation criteria.
3. Calibrate
Qualified specialists and annotators work against representative cases to confirm that the annotation protocol is being interpreted consistently.
Inter-annotator agreement can help identify areas where additional clarification or calibration is required.
4. Annotate
The approved guidance is applied through manual, AI-assisted or hybrid workflows depending on the task.
5. Escalate and clarify
Cases outside the protocol follow the agreed client clinical escalation path.
6. Verify and trace
Structured QA checks protocol adherence, precision and consistency, while preserving traceability around guideline versions, escalations and client-approved clarifications.
This is how IngeData turns client-defined clinical specifications into a repeatable data-production system.
Five questions to ask a medical-image annotation partner
For buyers, “expert-led” should be the beginning of the conversation.
1. Who actually works on the data?
Are they trained annotators, imaging technologists, doctors, radiologists or relevant subspecialists?
Does their expertise match the modality and task defined by the client?
2. How are clinical specifications translated into annotation guidance?
Can the partner show how the client’s requirements become operational rules, examples, decision criteria and a protocol that a production team can follow consistently?
3. How is consistency measured?
How are annotators calibrated?
Is inter-annotator agreement monitored?
How are recurring differences in interpretation identified?
4. What happens when the team encounters something outside the annotation protocol?
Is there a defined escalation path back to the client-designated clinical authority?
And once clarification is received, how does that decision reach the rest of the production team?
5. Can production decisions be traced?
Can you see which approved specification and annotation guidelines are applied, how exceptions were escalated and how client-approved clarifications were incorporated?
These questions reveal far more about the reliability of a medical dataset than the word “expert” alone.
The real asset is turning clinical intent into repeatable data
Medical AI will almost certainly continue to reduce the amount of manual work required to produce annotations.
But the clinical standard still has to come from the people responsible for the clinical hypothesis and diagnostic criteria.
The data partner’s job is to make that standard executable at scale.
That means codifying it clearly.
Calibrating the people applying it.
Measuring consistency.
Identifying deviations.
Escalating cases that fall outside the agreed guidance.
Applying client-approved clarifications across the wider operation.
And maintaining traceability throughout production.
At IngeData, our role is to translate client-defined clinical specifications into rigorous annotation guidelines, pairing qualified medical specialists with calibrated workflows to deliver precise, consistent, audit-ready and production-grade data.
The client retains ownership of the clinical hypothesis, reference standard and diagnostic thresholds.
We operationalise those decisions through annotation protocols, calibration, inter-annotator agreement, structured QA, escalation and traceability.
See how IngeData translates client-defined medical protocols into calibrated annotation workflows, specialist QA and production-grade datasets.
Bring us your imaging modality, clinical specification or annotation task, and we can help operationalise it into a rigorous, scalable data workflow.
Frequently Asked Questions
What is a reference standard in medical imaging AI?
A reference standard is the benchmark against which an AI system’s output or performance is assessed.
CLAIM’s 2024 guidance recommends this term rather than “ground truth”, recognising that medical labels can contain uncertainty and that the process used to establish the reference standard should be clearly documented.
Who should annotate medical images for AI?
The client’s clinical team determines the clinical criteria and level of expertise required for the task.
IngeData then matches qualified specialists and annotators to those requirements and calibrates the team against the client-approved annotation protocol.
The objective is consistent execution of the clinical specification across the dataset.
Do radiologists always agree when annotating medical images?
Agreement can vary depending on the finding and task.
That makes clear protocols, calibration, inter-annotator agreement and defined escalation pathways important.
When disagreement exposes a gap in the existing annotation protocol, IngeData can surface and document the issue for client clarification, then incorporate the approved resolution into the production guidance.
Can AI automate medical image annotation?
AI can increasingly assist with or generate annotations through interactive segmentation, automated segmentation and report-derived labelling.
The client remains responsible for the clinical reference standard and diagnostic criteria.
IngeData’s role is to operationalise those specifications through controlled annotation workflows, specialist validation, QA and escalation.
What should I ask about annotator expertise?
Ask who works on the data, whether their expertise matches the modality and task, how the client’s clinical specifications are translated into its annotation protocol, how teams are calibrated, how inter-annotator agreement is monitored, what happens outside the protocol and whether key production decisions can be traced.





