19 Jun 2026
Author
Yangjee Rai Shrestha
The model was never the problem. The team had selected a well-regarded large language model, fine-tuned it on internal data, and built a promising prototype in under six weeks. Then the accuracy reviews started. Predictions were inconsistent. Confidence scores were erratic. The engineers dug in and found what should have been obvious from the start: the training data was a mess. Labels contradicted each other. Entire categories had fewer than fifty examples. Some of the data was three years old and no longer reflected the business reality the model was supposed to serve.
Three months of development. A model that technically worked. Data that made it useless.
This is the most common failure pattern in AI projects — and it has nothing to do with choosing the wrong model. It starts with choosing the model before understanding the data. Teams evaluate architectures, compare benchmarks, and select a model based on published performance numbers. Then they feed it their own data and discover the performance evaporates. The model was tested on clean, well-labelled, abundant datasets. The organisation's data is none of those things.
The gap between a model's potential and an organisation's data reality is where most AI projects stall or fail. At EB Pearls, the Data Readiness Assessment™ is completed before model selection — not after. Across 900+ projects delivered for over 1,400 businesses, with 360+ AI-native developers, we have learned that the most valuable conversation in any AI engagement is not "which model should we use?" It is "what does our data actually look like, and is it good enough for what we want to build?"
Why Data Quality Determines AI Success
Model selection is the decision that gets the most attention. Data readiness is the decision that determines whether the model actually works.
The asymmetry is striking. Teams spend weeks evaluating model architectures, comparing parameter counts, and debating fine-tuning strategies. They spend days — sometimes hours — assessing the data that model will be trained or prompted on. The result is a precise answer to the wrong question: we know exactly which model to use, but we have no idea whether our data can support it.
This matters because models are commoditising. The performance gap between leading foundation models continues to narrow. The differentiator is no longer which model you select — it is the quality, relevance, and structure of the data you bring to it. Two organisations using the same model with different data quality will get dramatically different results. The model is the engine; the data is the fuel. No engine compensates for contaminated fuel.
The consequences of poor data readiness surface late and cost disproportionately. Research from MIT Sloan has documented how organisations consistently underestimate the effort required to prepare data for AI systems. Data cleaning, labelling, and restructuring typically consume the majority of an AI project's timeline — yet they are rarely scoped as first-class activities in project planning.
The project delivery framework at EB Pearls addresses this by making data readiness a gate that must be passed before model selection begins. If the data is not ready, the architecture conversation waits. This prevents the most expensive mistake in AI development: building the right system on the wrong foundation.
What a Data Readiness Assessment Covers
A data readiness assessment is a structured audit that evaluates whether an organisation's data can support the AI system it wants to build. It examines six dimensions, each of which can independently prevent a model from performing in production.
Volume and Coverage
The first question is deceptively simple: do you have enough data? "Enough" depends entirely on the task. A classification system with twenty well-defined categories needs a different volume than a generative system expected to handle open-ended queries. Fine-tuning a model for a narrow domain may require thousands of high-quality examples per category. Retrieval-augmented generation may need a comprehensive knowledge base covering the full range of queries users will ask.
Volume is not just total record count — it is distribution across categories. An organisation may have 100,000 labelled records, but if 90,000 belong to three categories and the remaining twenty categories share 10,000 records unevenly, the model will perform well on common cases and fail on everything else. Coverage mapping identifies which categories are underrepresented and estimates the gap between current volume and the minimum required for acceptable model performance.
Recency and Temporal Relevance
Data ages. A product catalogue from eighteen months ago may contain discontinued items, outdated pricing, and superseded descriptions. Customer support transcripts from before a major product redesign reflect problems and terminology that no longer exist. Training a model on stale data teaches it a version of reality that has already changed.
The recency audit evaluates when data was collected, whether it reflects current business conditions, and how quickly it becomes outdated. For some domains — financial data, regulatory content, market intelligence — recency is measured in days. For others — technical documentation, process descriptions — months may be acceptable. The audit establishes a recency threshold for each data source and flags anything that falls outside it.
Labelling Quality and Consistency
Labels are the ground truth that supervised models learn from. Inconsistent labels teach the model conflicting lessons. If one annotator labels a customer complaint as "billing issue" and another labels an identical complaint as "account query," the model learns that the distinction is arbitrary — because in the training data, it is.
Labelling assessment examines inter-annotator agreement: how consistently do different people label the same examples? It evaluates whether labelling guidelines exist, whether they are specific enough to resolve ambiguous cases, and whether labels have drifted over time as annotators changed or business categories evolved. Low inter-annotator agreement is a signal that the labelling taxonomy itself may be flawed — the categories may overlap, lack clear boundaries, or fail to capture meaningful distinctions.
For organisations that have not yet labelled their data, the assessment estimates the labelling effort required — the number of examples that need annotation, the expertise level of annotators, the time and cost involved, and the quality assurance process needed to maintain consistency.
Legal Usability and Compliance
Not all data that exists can be used. Privacy regulations — including the Australian Privacy Act, GDPR, and sector-specific rules — impose constraints on how personal data can be processed, stored, and used for model training. Data collected for one purpose may not be legally usable for another without explicit consent.
The legal usability audit identifies data sources that contain personal or sensitive information, evaluates whether existing consent covers AI training and inference, and flags data that requires anonymisation, pseudonymisation, or removal before it can be used. This includes third-party data obtained under licence agreements that may restrict AI use cases. Australia's Information Commissioner provides guidance on privacy obligations that directly affect how organisations can use data in AI systems.
Discovering a legal constraint after training means the model may need to be retrained from scratch on a compliant dataset — an expensive and time-consuming correction that a pre-model audit prevents entirely.
Bias and Representativeness
Data reflects the processes and populations that generated it. If historical hiring data overrepresents certain demographics, a model trained on that data will reproduce the bias. If customer data skews toward one geography, the model's performance will degrade for underrepresented regions.
Bias assessment examines whether the data represents the full range of inputs the model will encounter in production. It checks for demographic skews, geographic gaps, temporal biases (overrepresentation of certain time periods), and survivorship bias (data only from successful outcomes, missing the failures). Each bias source is documented alongside its potential impact on model behaviour.
Schema Consistency and Integration Readiness
AI systems rarely consume data from a single, clean source. Data arrives from CRM platforms, support ticketing systems, product databases, third-party APIs, and legacy systems. Each source has its own schema, naming conventions, date formats, and null-handling behaviour.
Schema assessment evaluates whether data from multiple sources can be integrated into a coherent training or inference pipeline. It identifies field-level inconsistencies — the same concept labelled differently across systems — and structural issues like missing fields, incompatible data types, and conflicting unique identifiers. Organisations using DevOps pipelines to manage data flows benefit from early schema alignment, which prevents integration failures from surfacing mid-development.
How to Run the Assessment
Start with a data inventory, not a model wishlist. Before evaluating any model, catalogue every data source relevant to the AI use case. For each source, document: what it contains, how it was collected, when it was last updated, who owns it, what format it lives in, and what legal constraints apply. This inventory becomes the foundation for every subsequent audit dimension.
Quantify volume and coverage gaps per category. Map the distribution of records across every category or label the model needs to handle. Identify categories below the minimum viable threshold. Estimate the effort required to fill gaps — whether through additional data collection, synthetic data generation, or scope reduction. At EB Pearls, this quantification happens during the Discovery Workshop™, where data gaps are surfaced before they become architecture constraints.
Assess labelling with a consistency sample. Pull a random sample of labelled data and have two independent reviewers re-label it. Measure inter-annotator agreement. If agreement falls below 80 percent on any category, the labelling guidelines need revision before training begins. This single step prevents the most common source of model inconsistency.
Run a legal and compliance review with your privacy team. Do not assume existing data can be used for AI. Engage legal and compliance stakeholders to review consent provisions, licence agreements, and regulatory requirements for each data source. Flag any source that requires additional consent, anonymisation, or removal from the training pipeline.
Document findings as a readiness scorecard. The output of the assessment is not a report that sits on a shelf — it is a scorecard that maps each data source against each audit dimension, with a clear pass, conditional, or fail status. This scorecard becomes the input to model selection and architecture decisions, ensuring the chosen approach fits the data reality, not the other way around.
The Model Was Right. The Data Was Wrong.
An enterprise AI team set out to build an intelligent document routing system. They evaluated several foundation models, selected one with strong benchmark performance on classification tasks, and began fine-tuning on their internal dataset of labelled documents.
Six weeks in, the prototype was promising. The team pushed forward. Three months in, accuracy plateaued well below the threshold needed for production deployment. The engineering team assumed the model was underpowered and began evaluating alternatives.
The problem was not the model. A retrospective audit of the training data revealed two critical issues. First, labelling inconsistency: the documents had been labelled by three different teams over two years, with no shared guidelines. Nearly half the labels were inconsistent — the same document type labelled differently depending on who annotated it and when. Second, volume insufficiency: six of the fourteen routing categories had fewer than one hundred labelled examples, far below the threshold needed for the model to learn reliable classification boundaries.
The team had spent three months optimising a model that was never going to succeed — not because of its architecture, but because the data could not teach it what it needed to learn. A data readiness assessment before model selection would have surfaced both the labelling inconsistency and the volume gaps in the first week. The team could have invested those three months in fixing the data rather than debugging a model that was doing exactly what it was trained to do — learning from flawed inputs.
When a Data Audit Is Essential and When It Can Wait
Audit before model selection if you are building an AI system that will be fine-tuned on proprietary data, trained on internal datasets, or expected to perform on domain-specific tasks where public benchmarks are not representative. Any system where data quality directly determines model performance — classification, extraction, recommendation, generation from internal knowledge — needs a readiness assessment before the architecture conversation begins.
A lighter review may suffice if you are using a foundation model off the shelf with prompt engineering only, with no fine-tuning and no proprietary training data. Even then, a review of the data feeding into retrieval-augmented generation pipelines is worth the effort — garbage in the knowledge base produces garbage in the responses.
A data audit cannot wait if you are in a regulated industry, working with personal data, or building a system where model failures have financial or reputational consequences. The development lifecycle should include a data readiness gate before any model training begins.
Where to Start
Pick one AI use case. List every data source it depends on. For each source, answer four questions: is there enough data? Is it recent enough? Are the labels consistent? Can we legally use it? If any answer is uncertain, you have found the gap that will derail your model — and you have found it before it costs you months.
When you are ready to assess your data before selecting a model, talk to our team. We surface data gaps in week one — because discovering them at month three costs ten times more.
Frequently Asked Questions
How long does a data readiness assessment typically take?
For a focused AI use case with well-defined data sources, a thorough assessment takes one to three weeks. The timeline depends on the number of data sources, the accessibility of the data, and whether legal and compliance review is straightforward or requires negotiation with third-party data providers. The investment is measured in days; the cost of skipping it is measured in months of rework when data problems surface during model training or production deployment.
What is the minimum data volume needed for AI training?
There is no universal number — it depends entirely on the task, the model architecture, and the complexity of what the model needs to learn. Fine-tuning a classification model may require hundreds to thousands of examples per category. Retrieval-augmented generation systems need a knowledge base comprehensive enough to cover the full range of expected queries. The readiness assessment quantifies the specific volume required for your use case and measures the gap between what you have and what you need.
Can we use synthetic data to fill gaps in our training dataset?
Synthetic data can supplement real data when specific categories are underrepresented, but it cannot replace real data entirely. Synthetic examples must be generated carefully to avoid amplifying existing biases or introducing artificial patterns the model learns as real. The assessment identifies where synthetic data is a viable strategy and where only genuine, domain-specific examples will produce reliable model behaviour.
How do we handle data that is partially labelled or unlabelled?
Partially labelled data is common and not necessarily a barrier. The assessment quantifies how much labelled versus unlabelled data exists and evaluates strategies to close the gap — including manual labelling with consistent guidelines, semi-supervised approaches that leverage unlabelled data, and active learning strategies that prioritise labelling the examples most valuable to model performance. The key is knowing the labelling status before selecting a model, since some approaches tolerate sparse labels while others require comprehensive annotation.
What if our data is spread across multiple systems with different formats?
This is the norm, not the exception. Most organisations store relevant data across CRM platforms, support systems, document repositories, and legacy databases — each with its own schema and conventions. The assessment maps these sources, identifies integration challenges, and estimates the engineering effort required to build a unified data pipeline. Schema inconsistencies discovered during the audit are far cheaper to resolve than schema failures discovered during model training.
How does data readiness affect AI model selection?
Data readiness should drive model selection, not the other way around. An organisation with abundant, well-labelled data can consider fine-tuning approaches that require large training sets. An organisation with limited but high-quality data may be better served by few-shot approaches, retrieval-augmented generation, or prompt engineering strategies that do not require extensive training data. Google's machine learning best practices emphasise that starting with the data — not the model — produces more reliable production systems.
As a QA Manager, Yangjee is passionate about quality, automation, and security testing. She thrives on continuous learning to deliver exceptional software.
Read more Articles by this Author