Your Data Isn't 80% of the Problem: Tracing a Myth

Every AI conversation I have with a mid-market manufacturer reaches the same sentence, usually within twenty minutes: "Well — you know 80% of this is data prep."

It is said with confidence, and it is used to settle the argument. Sometimes by a vendor, to justify selling a data warehouse before anything useful gets delivered. More often by the buyer, as a reason to postpone — because if four fifths of the work is cleaning data, and data is never clean, the project can be deferred indefinitely without anyone having to say no.

So I went looking for where the number came from. It took about an hour, and the answer changes what you should do about it.

The trace

The figure has a clear ancestor. In March 2016, Forbes reported a CrowdFlower survey of approximately 80 data scientists, which found they spent 60% of their time cleaning and organising data, and a further 19% collecting data sets.

Sixty plus nineteen is seventy-nine. Round it up, drop the fact that it is two separate activities, and you have the statistic that has been quoted continuously for a decade.

Leigh Dodds traced the same claim across the full run of surveys in 2020, and the picture that emerges is less stable than the confident repetition suggests.

Three things stand out.

The 2015 figure is not a time measurement. It is the share of people who named cleaning as one of their most time-consuming tasks — a count of respondents. It gets quoted as though it were a percentage of hours.

The 2016 figure is two numbers. Collecting data and cleaning data are different activities with different fixes. Merging them makes the problem look like one thing when it is two, and the collection half is usually a process problem, not a data problem.

Every subsequent measurement is lower, and the largest, most recent ones cluster at 38–45%.

Two things worth saying against my own argument

I am not going to present this as a debunking, because it is not one.

First, data prep is still the single largest block of work in every survey in that table. Nobody measured it and found it small. If the point of this post were "data quality is overrated," the evidence would not support it, and I would be doing the same thing I am criticising.

Second, look at who published which number. CrowdFlower sold a data enrichment service. Anaconda sells a data science platform. The vendor with the most commercial interest in a large data-prep number published the largest one, in the smallest survey. I am not alleging bad faith — surveys of your own users are legitimately hard to make representative — but it is a fact about incentives that belongs next to the figures.

And a caveat on all of it: these are surveys of data scientists, mostly outside the UK and mostly not in mid-market manufacturing. None of them measured a 200-person engineering firm trying to get job sheets into an ERP. Applying them to your business is an inference, not a reading.

Why the myth costs you money

The accurate number — data prep is roughly two fifths of the effort, and the largest single slice — supports a different decision from the myth.

At 80%, the rational move is to fix the data first. Build the warehouse, run the migration, standardise the master data, and start the AI work in a year when the foundation is solid. I have watched several of these programmes. They are expensive, they are hard to stop once started, and the AI use case that justified them is frequently forgotten by the time they finish.

At 40%, the rational move is the opposite: pick one narrow use case and test the data for that use case only. Because the other insight buried in the trace is that "is our data ready" is not a real question. Data is not ready or unready in general. It is adequate or inadequate for a specific job.

Your job sheets can be a mess by any database standard and still be perfectly adequate for extracting three fields that appear in the same box on every sheet. Your CRM can be immaculate and still be useless for forecasting because nobody records the thing that predicts the outcome. Readiness is a property of the pairing, not of the data.

What readiness means, by use case

There is no published UK benchmark for data readiness thresholds, so I am not going to invent one. What I can do is say what the question becomes for the three situations that come up most.

That third case is worth dwelling on, because it is invisible in any data audit. A data quality assessment examines the data you have. It cannot tell you about the work that produces none — and in most of the operations I look at, that is where the largest single block of unmanaged effort sits.

Four tests you can run this week

These are my own method, not an industry standard — I am labelling them that way so you can weigh them accordingly. They cost an afternoon between them and they replace a question nobody can answer with three you can.

Test three is the one I would run first if you only run one. It converts an argument into a measurement, and it produces a number that is defensible in front of a board and useful in a vendor conversation — which is exactly what a procurement process needs.

When a data programme really is the right answer

To be fair to the position I have been arguing against: sometimes it is correct.

If the same entity exists under four different identifiers in three systems and no reliable key joins them, no amount of narrow scoping fixes that, and every use case you attempt will hit the same wall. If your field-presence count comes back at 12%, you do not have a model problem, you have a capture problem, and it needs solving upstream in the process — which is almost always the real fix anyway.

The difference is that you would be doing the data work because a specific, measured deficiency blocked a specific use case. Not because a survey of eighty people in 2016 said 80%.

Where to start

Pick one process. Run the four tests on it. If it passes, you have a candidate and a realistic accuracy ceiling to hold a supplier to. If it fails, you know precisely which of the four failed and therefore what to fix — which is a far better position than "our data isn't ready."

If you want help choosing which process to point this at, that is what the hidden waste audit does, or book a call and we can walk through one together.

Related reading

• Job Sheet Parsing: Turning Paper and PDFs into ERP Data
• What to Ask an AI Vendor Before You Sign
• The Real Cost of Poor CRM Data in Field Sales
• Ghost Workflows: The Hidden Manual Tasks Nobody Owns
• Why 70% of AI Projects Fail — And What the 30% Do Differently

Frequently Asked Questions

Do data scientists really spend 80% of their time on data preparation?

The figure traces to a CrowdFlower survey of approximately 80 data scientists reported in March 2016, and it is two numbers added together: 60% for cleaning and organising data plus 19% for collecting data sets. Later and larger surveys measured considerably less — Anaconda's State of Data Science reported 45% in 2020, 39% in 2021 and 38% in 2022. Data preparation remains the single largest block of work in all of these surveys, but the 80% figure overstates it and merges two distinct activities.

What does "data readiness" mean for an AI project?

Readiness is not a general property of your data — it is a property of the pairing between a specific dataset and a specific use case. Job sheets can be untidy by database standards and still be perfectly adequate for extracting three fields that appear in the same place on every sheet. A well-maintained CRM can be useless for forecasting if nobody records the field that predicts the outcome. The useful question is never "is our data ready" but "is this data adequate for this job".

How do I test whether our data is good enough for AI?

Four cheap checks. Pull fifty real records at random and have a person try to extract the target field by eye — whatever they cannot do, a model will not do either. Count what percentage of records have the target field populated at all. Have two people independently key the same twenty documents and compare, because their agreement rate is your realistic accuracy ceiling and no model resolves what two humans cannot. Finally, ask who enters each field and what happens if it is wrong, since fields with no consequence attached are unreliable however full the column looks.

Should we fix our data before starting an AI project?

Usually not as a prerequisite programme. A large upfront data programme is justified when a specific measured deficiency blocks your use case — for example the same entity existing under different identifiers across systems with no reliable join key, or a target field populated in only a small minority of records. Otherwise, scope one narrow use case and test the data for that use case alone, because data is never finished and "our data isn't ready" can defer a project indefinitely.

Where does the "80% of AI is data prep" statistic come from?

From a CrowdFlower data science survey published in March 2016 with a sample of about 80 respondents, which found 60% of time spent cleaning and organising data and 19% collecting data sets. The two were summed to produce roughly 80%. CrowdFlower, later renamed Figure Eight, sold a data enrichment service. Its 2015 predecessor figure of 66.7% is frequently quoted as a time measurement but was actually the share of respondents naming cleaning as a most time-consuming task.

---

Sources. CrowdFlower 2016 sample size and the 60% / 19% split via Forbes, 23 March 2016. CrowdFlower 2015 and 2017, Kaggle 2018 and Figure Eight 2019 figures via Leigh Dodds, 31 January 2020. Anaconda time-allocation figures via State of Data Science 2020 and BigDATAwire's coverage of the 2020 survey, 6 July 2020; 2021 and 2022 figures from the corresponding editions. Anaconda has published no time-allocation breakdown since 2022. All URLs fetched and verified 29 July 2026.