A dataset may perform well in a pilot and still become unreliable when the project scales.

At the centre of the issue is ground truth: the labels, annotations and expert decisions the model is taught to treat as the correct answer. If those decisions are inconsistent, ambiguous or poorly defined, the model learns from an unstable reference point.

Reliable training data depends on keeping that ground truth consistent as the project scales. Production puts the workflow under more pressure. Volumes increase, more people enter the process and edge cases appear more often. The challenge is maintaining the same definition of “correct” across the whole dataset.

For AI teams moving from pilot to production, this is where data debt can build: quality issues start creating rework, slowing model development and pulling technical teams back into data management.

‍

Production-ready ground truth depends on a repeatable system

Ground truth becomes production-ready when the same definition of “correct” can be applied consistently across people, batches and changing data.

That requires clear annotation rules, the right domain expertise and quality controls throughout the workflow. At production scale, the real test is repeatability: whether the workflow can maintain the same standard as the project grows and evolves.

A repeatable system is what allows training data to hold up beyond the pilot stage.

‍

1. Quality starts with the protocol

So, how do we create repeatability? Before annotation begins, someone has to define what “correct” means.

That gets harder when the data is ambiguous. For example:

• Should a borderline structure be included?
• How should an occluded object be handled?
• When does one class become another?
• What measurement rule applies when image quality is poor?

An annotation protocol turns those judgement calls into operational rules. It defines:

• inclusion and exclusion criteria
• naming conventions
• measurement rules
• escalation paths
• how new edge cases are incorporated

At IngeData, these requirements are formalised in a Functional Technical Requirement Document (FTDR), creating a shared reference for how the project should be carried out. The FTDR is created and co-construct with the client  as part of the delivery process.

The protocol should also evolve with the project. When an edge case exposes a gap, that decision should become part of the shared guidance so the whole team can apply the same rule going forward. 

‍

2. Specialist judgement changes what “correct” looks like

A clear protocol can define the rules, but some annotation decisions still depend on subject matter experts.

These experts bring technical knowledge that general annotators may not have. In high-stakes computer vision, that can mean recognising clinical anatomy, interpreting SAR geometry or distinguishing subtle industrial defects that are not obvious from the raw data alone.

For example:

• Healthcare: radiologists may need to distinguish anatomy, benign variation and pathology. In cardiac CT, that can include separating the artery, lumen, vessel wall and different plaque types.
  
  
• Earth Observation: Synthetic Aperture Radar (SAR) uses radar rather than ordinary photography, so objects can appear very differently. Annotation can require specialist geometry and image-interpretation skills.
  
  
• Industrial inspection: low-contrast corrosion can blend into complex surface textures, making the defect boundary difficult to define consistently.

At IngeData, our expertise is matched to the complexity of the task. Specialist judgement is brought in where context can change the answer, then translated into the protocol and review process.

This turns domain knowledge into a repeatable part of quality control.

3. Edge cases need a feedback loop

Edge cases are part of any complex dataset. What matters is how the workflow learns from them.

When an unusual case appears, the decision should feed back into the process:

• identify the issue
• resolve it with the right expert
• update the guidance
• brief the wider team
• review earlier work if needed

This creates a feedback loop: each resolved edge case improves the rules everyone uses next time.

In IngeData’s work with Median Technologies on lung CT data, radiologists were trained on the client’s tool, annotation criteria were defined for nodule size, density and measurement and consistent segmentation rules were applied across solid, part-solid and subsolid nodules.

Ongoing feedback loops were then used to refine quality and workflow efficiency as the team scaled to meet volume.

That turns one difficult case into a better system for the next hundred.

‍

4. Annotation QA needs to run through the workflow

Quality assurance works best while the data is still being produced, so issues can be corrected before they spread across a larger batch.

Depending on the task, that can include:

• calibration: checking that annotators apply the same rules to the same examples
• sample review: reviewing a subset of annotations during production
• peer review: having a second person check the work
• specialist checks: escalating difficult cases to the right domain expert
• protocol updates: refining the guidelines when new edge cases appear
• client or model feedback: feeding new learnings back into the workflow

The quality process is documented too. IngeData creates a DQA (Document Quality Assessment) as part of the project, giving the client visibility into how quality is assessed alongside the requirements defined in the FTDR. Both documents form part of IngeData’s standard delivery process and are accessible to the client.

For example, in medical imaging, annotation quality assurance may involve specialist review of cases where clinical interpretation can change the ground truth.

The exact mix will vary by project, but the principle stays the same: review should shape the work while it is still in progress.

QA should become part of the operating system for the dataset, helping the team keep the same standard as the work progresses. 

‍

5. Stress-test the system before you scale

Before adding more people, make sure the workflow can support them. That means being clear on:

• who owns the protocol
• which decisions need specialist review
• how reviewers stay aligned
• how feedback is shared
• how changes are documented and traced

Once those foundations are in place, additional capacity can increase throughput while keeping the same standard.

If they are not, scale can introduce more interpretation, rework and management overhead. This may pull the AI team back into fixing the dataset instead of moving the model forward.

‍

6. questions to pressure-test your training data workflow

Before you scale the dataset, check whether the workflow can keep producing the same standard.

1. Who defines what “correct” means?

Is there clear ownership of the annotation protocol?

2. Where is specialist judgement needed?

Which decisions require clinical, geospatial or industrial expertise?

3. What happens when an edge case appears?

Is it resolved, documented and added back into the guidance?

4. How is quality checked during production?

What gets reviewed, how often and by whom?

5. How does feedback improve the next batch?

Do reviewer, client or model insights make their way back into the workflow?

6. Can you trace how the rules changed?

Are protocol updates, review decisions and key changes documented?

If you can answer all six clearly, you have a much stronger foundation for scaling the data operation without letting the standard drift.

‍

Turn data quality into a repeatable system

Reliable ground truth comes from a system that can reproduce the same judgement at scale.

That means clear annotation protocols, the right specialist input, quality checks throughout production and feedback loops that keep improving the guidance over time.

IngeData helps AI teams build and run the human layer behind reliable ground truth, combining domain expertise, structured QA and scalable data operations for high-stakes computer vision.

Explore Healthcare

Discover how specialist teams turn clinical judgement into repeatable ground truth for medical imaging work, including MRI standardisation, mammography segmentation and clinical reading.

Explore Earth Observation

Take a closer look at the workflows behind consistent ground truth at scale, with examples from SAR imagery, agricultural mapping and satellite annotation.

Explore Industry 5.0

Go behind the workflow to explore how repeatable annotation systems support reliable ground truth for industrial computer vision, from corrosion detection to material recognition and machine digitisation.

Talk to the Team

Bring us your task, data type or scaling challenge and we can help shape a workflow that keeps the ground truth reliable as the project grows.

‍

Frequently asked questions

What is ground truth in computer vision?

Ground truth is the reference data a computer vision model is trained or evaluated against. It can include class labels, bounding boxes, polygons, segmentation masks, landmarks or specialist interpretations, depending on the task.

Its usefulness depends on consistency. If two annotators interpret the same object differently, the model is being given conflicting examples of what “correct” means. Teams therefore need clear annotation rules, agreed class definitions and a process for resolving ambiguous cases.

How do you measure annotation quality?

Annotation quality can be assessed in several ways depending on the task.

Common approaches include sample-based review, where a percentage of completed annotations is checked, and inter-annotator agreement, which measures how consistently different annotators interpret the same data.

For segmentation tasks, teams may also compare how closely two masks overlap. Where disagreement reflects genuine ambiguity rather than a simple error, the case may need adjudication: a specialist makes the final decision and that decision is added back into the annotation guidance.

The important question is whether the QA method is appropriate for the type of data and decision being made.

How do you keep ground truth consistent as a dataset scales?

Consistency starts with an annotation protocol that defines classes, inclusion and exclusion rules, measurement criteria and escalation paths.

Before larger-scale production, teams can also use calibration exercises, where multiple annotators work on the same examples and compare results. This helps expose differences in interpretation early.

As production continues, edge cases should be documented, reviewed and incorporated into the guidance. Keeping track of protocol versions also matters, because teams need to know which rules were in force when different parts of the dataset were produced.

This creates a repeatable system rather than relying on individual judgement alone.

Why does domain expertise matter for high-stakes annotation?

Some visual distinctions require technical knowledge that a general annotator may not have.

Specialists help resolve these judgement-heavy cases and turn their decisions into rules the wider annotation team can follow consistently. IngeData uses this kind of domain expertise across Healthcare, Earth Observation and industrial projects.

What should you look for in a training data partner?

Look beyond annotation capacity. Ask how the provider defines protocols, selects specialists, handles edge cases, performs QA/QC, incorporates feedback, documents the process and gives you visibility into it as the programme scales.

At IngeData, these elements are managed as part of the wider data workflow, combining domain expertise, structured QA and scalable data operations for high-stakes computer vision projects.