Chapter 05

Data Quality: Dimensions, Cleaning & Profiling

Garbage in, garbage out — the success of any data-driven decision rests on the quality of its data. This chapter defines data quality as fitness for use, names the four objective dimensions (accuracy, completeness, consistency, timeliness), and walks the cleaning pipeline: profiling, standardization, imputation, outlier and duplicate detection.

Reading: ~30 min Interactive: 1 widgets Source: Polimi TIS 2025/26 — Data Quality, Prof. Cappiello (deck 06)

01 · Motivation

Why data quality (GIGO)

Data-driven management collects data, analyses it, and decides on the insights. Its success rests on two things: the quality of the data collected and the methods used to analyse it. Feed in garbage and you get garbage out — GIGO — and even a small error can snowball as it flows downstream.

The horror stories make it concrete: NASA’s Mars Climate Orbiter (1999) was destroyed because thrust was left in English units instead of newtons — a $125-million unit-conversion bug. Miscoded hospital records produced headlines about “17,000 pregnant men”; workers demolished the wrong house trusting a map; a decimal-point slip in 1870 overstated spinach’s iron content tenfold and drove a generation to eat more of it. Real-world data is routinely incomplete, inconsistent and error-ridden, which is why data preparation, cleaning and transformation are the ~90% of effort in a real data-mining application.

key

What data quality means

The traditional definition is fitness for use — the ability of a data collection to meet user requirements. From an information-system view: the real-world user view and the derived view must show no contradictions. Managing it is a four-activity loop — define the dimensions, assess them, analyse the issues, improve — formalised as the TDQM cycle (Wang 1998): Define → Measure → Analyze → Improve, repeated.

02 · Exam-hot

The four objective dimensions

Data-quality dimensions — asked in 2018-02-20

“List and describe the main dimensions of Data Quality” is a Part I question (2018-02-20). Graders want the four objective dimensions with a crisp definition and an example each — and ideally the point that they trade off against each other in assessment cost. The safe answer is the professor’s own four, below; the trap is listing vague adjectives (“good”, “clean”) instead of the defined dimensions.

Accuracy

The extent to which data are correct, reliable and certified — the stored value matches the real-world one (e.g. Leonardo recorded, not Lonardo).

Completeness

The degree to which a collection includes the data describing the corresponding real-world objects — no missing tuples, attributes or values that should be there.

Consistency

The satisfaction of the semantic rules defined over a set of data items — e.g. province must agree with postcode; two records of one person must not contradict.

Timeliness

The extent to which data are sufficiently up-to-date for the task at hand.

key

They are not equally easy to measure

The deck orders the dimensions by both business impact and assessment complexity, increasing together: completeness < consistency < timeliness < accuracy. Completeness you can often count; accuracy is the hardest — deciding whether a present, well-formed value is the right one usually needs an external source of truth.

The problems these dimensions catch are the everyday ones: missing values, duplicates, inconsistencies, outliers, noise and out-of-date data — near-duplicate spellings (Kathy Robert / Cathy Robbert), clashing date formats (08.08.1969 / 08/07/1975), synonym values (Flu / Influenza), placeholder costs (9999), and town-name variants (St. Pete Beach / Saint Pete Beach).

Here they all are in one table. Naming the four is the easy half; the graded half is saying which dimension a given cell offends, and two of these are routinely mis-filed:

Hands-on

The four dimensions, on a dirty table

One small patient table carrying the chapter's own defects — near-duplicate spellings, clashing date formats, synonym values, a placeholder cost and a town-name variant. Pick a dimension and only the cells it condemns light up.

Every seeded defect is highlighted. Narrow to one dimension to see how differently the four of them partition the same table.

#	Name	City	Diagnosis	Admitted	Cost	Last updated
1	Leonardo Bianchi	Saint Pete Beach	Influenza	08/07/1975	1200	2026-07-10
2	Lonardo Bianchi	St. Pete Beach	Flu	08.08.1969	9999	2019-01-04
3	Kathy Robert	Milano	Influenza	missing	800	2026-07-12
4	Cathy Robbert	Milano	missing	12/03/2020	800	2015-06-30

#2 · Name	Accuracy	"Lonardo" is a typo for Leonardo — a well-formed value that does not match the real world. Only an external source of truth reveals it.
#2 · Cost	Accuracy	9999 is a placeholder wearing a number's clothes. The field is populated, so completeness looks fine — which is exactly what makes it dangerous.
#4 · Name	Accuracy	"Cathy Robbert" and "Kathy Robert" are almost certainly one person recorded twice. Duplicates make the stored picture disagree with reality.
#3 · Admitted	Completeness	An admission date that should exist is simply absent.
#4 · Diagnosis	Completeness	A missing value in a field every admitted patient must have.
#2 · City	Consistency	"St. Pete Beach" and "Saint Pete Beach" are the same town under two spellings. Each value is individually correct — they just cannot both be the canonical one.
#2 · Diagnosis	Consistency	"Flu" and "Influenza" are synonyms for one condition. Nothing here is inaccurate; the vocabulary is simply not reconciled.
#2 · Admitted	Consistency	Dotted dd.mm.yyyy against the slashed dd/mm/yyyy used everywhere else — a format rule broken, not a wrong date.
#2 · Last updated	Timeliness	Last touched in 2019. The values may all be correct and still be too stale to act on.
#4 · Last updated	Timeliness	Last touched in 2015 — the oldest record in the table.

Cells flagged

Assessment cost

—

Try thisCompare two cells in row 2. Flu is a consistency failure — it is a perfectly correct word, just not the same vocabulary as Influenza. The cost 9999 is an accuracy failure, and note what it is not: completeness looks satisfied, because the field is populated. That is precisely why placeholders survive cleaning.

TakeawayName the four and give one example each — vague adjectives score nothing. Completeness asks whether the value is there, consistency whether the values agree, timeliness whether it is current, accuracy whether it is true. Only the last needs the real world to answer, which is why it is both the most valuable and the hardest to measure.

2018-02-20-q12018Q01Data quality dimensionsmedium5 pts

List and describe the main dimensions of Data Quality.

03 · Method

Cleaning, profiling & imputation

Improvement comes in two flavours: data-based approaches fix the values (find and correct errors regardless of context), while process-based actions chase the root cause so the error stops recurring. The data-based workhorse is data cleaning — “identifying and eliminating inconsistencies, discrepancies and errors” (Naumann 2000) — run as a pipeline:

Pipeline The data-cleaning pipeline (Naumann & Sattler 2006)

1 · Data collection / selection

Gather the records to clean.

2 · Data profiling

Analyse content and structure — per-attribute type, domain, distribution, null occurrence, uniqueness, format; functional dependencies and primary-key candidates; cross-relation overlaps (redundancies, foreign keys); counts of missing/wrong values and duplicates. Tools like Ydata profiling automate the report (variables, missing %, correlations).

3 · Standardization / normalization

Convert to a common format — data-type conversion, dates (03/01/15 → 01-MAR-2015), currency ($ → EUR), tokenising ("Smith, Paul" → "Smith", "Paul"), discretization, and domain-specific transforms ("St." → "Street").

4 · Error correction

Localise and correct inconsistencies and incomplete data, and localise outliers. Correcting a record may require new data acquisition.

5 · Duplicate detection

Find multiple representations of the same object (see below).

6 · Quality assessment

Re-measure the dimensions to confirm the cleaning worked.

Two of these steps carry the real subtlety:

Imputing missing values

Prefer unbiased estimators that fill gaps without shifting the dataset’s characteristics (mean, variance) — median/mean; exploit functional dependencies (#Bedrooms → Income); or use statistics such as linear regression (or neural nets for non-linear dependencies). Watch for bias introduced by treating nulls carelessly.

Outlier detection

An outlier is a suspicious observation deviating too much from the rest. Detect it by distribution, geometry or over a time series — then interpret: is it a data error, or a real (rare) event? The two calls demand different responses.

tip

Duplicate detection — the high-level process

The same shape as record linkage in Chapter 3. Input files A and B → preprocessing → search-space reduction (blocking) to a candidate set $C \subseteq A \times B$ → comparison & decision producing match / possible-match / non-match, with a quality-assessment feedback loop into every stage. The two governing concerns are, again, choosing a good similarity measure and minimising the number of comparisons.

key

Where this connects

When cleaning spans multiple sources, it is integration: schema alignment → entity reconciliation → data fusion (Chapter 3). And in a data warehouse, all of this happens in the ETL / staging area before the clean data lands — the bridge to the warehousing half of the course.

04 · Context

Big data & data quality

High-quality data is the precondition for trustworthy analysis — and big data did not dissolve the problem by throwing quantity at it. The veracity V is exactly this: “1 in 3 business leaders don’t trust the information they use to decide,” and poor data quality is estimated to cost the US economy around $3.1 trillion a year.

Cai & Zhu (2015) frame four big-data challenges, one per V:

Variety — diverse sources and complex structures (structured, semi-structured, IoT); traditional ETL struggles.
Volume — profiling and assessment become hard to run in reasonable time.
Velocity — timeliness windows are tiny; stale data is invalid data.
Veracity — DQ standards exist for traditional sources but not for big data.

Assessing quality on unstructured big data needs dedicated pipelines (e.g. the Taleb et al. 2018 model: sampling & profiling → per-media extraction → dimension selection & metric mapping → evaluation → quality report). The recurring big-data DQ issues: not-integrated, incomplete or incorrect data; cleaning needed frequently; inconsistent, unreliable sources; and the need for data provenance and lineage — which returns as a frontier topic later in the course.

Load-bearing ideas

GIGO: decisions are only as good as the data; small errors snowball, and preparation is ~90% of the work.
Fitness for use: no contradiction between the real-world and derived views; managed via the TDQM loop (Define → Measure → Analyze → Improve).
The four objective dimensions: accuracy (correctness), completeness (nothing missing), consistency (semantic rules hold), timeliness (up-to-date). Ordered by cost/impact, completeness < consistency < timeliness < accuracy.
Cleaning pipeline: profiling → standardization → error correction → duplicate detection → assessment; imputation prefers unbiased estimators / FDs / regression; outliers need detection and interpretation.
Big data: quantity ≠ quality; the four V challenges (Cai & Zhu); provenance matters.
Exam radar: be able to name and define the four objective dimensions with an example each and place them by assessment difficulty.