Modern Data Landscape: Governance, Lakes & Ethics
The course's closing survey of where data management is heading: DAMA data governance and metadata, data virtualization (the logical, Denodo-style approach), data lakes / lakehouse / data mesh and FAIR, and the frontier problems — uncertainty, provenance, crowdsourcing, and the ethics of data-driven decisions.
A survey chapter — new in 2025/26, not yet in the exams
Everything here is taught in the 2025/26 course (guest lectures on DAMA and Denodo, and the trends lectures) but none of it appears in the 16 past exams in our bank (the newest is 2022). Learn it for understanding and for the possibility of a fresh Part I question — but the exercise marks live in the integration and warehousing chapters. Ethics is the one topic the 2025/26 slides visibly emphasise, so it gets the fullest treatment below.
01 · Governance
Data governance & metadata
Data governance is “the exercise of authority and control (planning, monitoring, enforcement) over the management of information assets.” In the DAMA framework it is the central hub: the other knowledge areas define what to do; governance defines how, and who does it.
DAMA (the vendor-, technology- and method-independent Data Management Association) publishes the DMBoK (Body of Knowledge) — a framework of standards, practices and a common vocabulary, drawn as the “DAMA Wheel”, with a professional certification (CDMP). Being data-driven — deciding on objective facts, backed by a data strategy — is the payoff; the cautionary pairs are Netflix/Amazon (who leaned into data) versus Blockbuster/Kodak (who did not).
A governance framework fixes an organizational model, and the three archetypes trade centralization against autonomy:
Centralized
One governance body oversees every subject area — a consistent view and uniform metadata, but it lacks deep domain knowledge and risks being self-referential.
Decentralized
Each business unit runs the same operating model — strong participation and vertical knowledge, but it risks data silos and divergent rules per unit.
Federated
A central function delegates to the units under shared guidelines and standards — efficient in a mature organization, but it needs a solid regulatory framework and strict monitoring.
Roles run from the Chief Data Officer and Data Owner through the Data Steward, Data Architect, Data Custodian and CISO down to the Data Consumer.
Metadata — the Carbonara analogy
Metadata is “data about data” — like a recipe or a food label, it tells you the meaning of a dataset, helps you use it, and certifies its origin. Three staples: a business glossary (agreed definitions — “reducing ambiguity so everyone means the same Carbonara”), a data dictionary (technical structure), and a data catalog (searchable inventory). Lineage / provenance — “there is no Carbonara without the hen” — records where each datum came from and how it was derived.
02 · Virtualization
Data virtualization & logical data management
Technology is a means, not an end — the goal is always business value. Data virtualization is the logical approach to that value: build a unified semantic model over the sources without copying the data. Only the definition (the logical/intensional component) is extracted; the data (the physical/extensional component) stays where it lives and is fetched only when required — a direct descendant of the virtual integration of Chapter 1, at enterprise scale.
Augmented Concept View — Enterprise → Extended → Augmented
A concept is represented by all the data that characterises it, wherever it resides. Denodo frames three concentric scopes: the Enterprise (data the organisation owns), the Extended Enterprise (+ external data providers), and the Augmented Enterprise (+ data shared out to partners, suppliers, outsourcers) — an ecosystem bounded by business scope, not legal scope. The semantic model grows Lego-like as sources connect: connect → represent → use, with a data marketplace letting consumers explore from the logical down to the physical.
The everyday analogy is PayPal: your money is not pre-transferred into it, but can be moved when needed; PayPal keeps a semantic model of who can send and receive, so adding a new payment method — or changing your bank — is invisible to merchants, and security is handled centrally. Denodo does the same for data sources. The take-homes: embrace a distributed data landscape, manage it through a logical, centralized semantic model decoupled from location and physical schema, and use AI to automate the tedious parts.
03 · Architecture
Data lakes, lakehouse & data mesh
Traditional architectures hit limits — rigid ETL, bottlenecks, cost — against explosively heterogeneous data. The responses form a family:
Data lake
A flexible store for raw, heterogeneous data (schema-on-read), organised in zones (landing → work → gold, plus a sensitive area). Without discipline it becomes a data swamp — so metadata and a data catalog are essential (a lake has no native catalog).
Data lakehouse
Combines the lake’s flexible storage with the warehouse’s analytics; a governance layer (cataloguing, curation, metadata, access policies) is fundamental, and it integrates a catalog natively.
Data mesh
A decentralized, socio-technical approach — analytical data owned by the domains, not a central pipeline.
Two patterns recur. The medallion architecture improves quality in layers — bronze (raw) → silver (cleansed, conformed, integrated) → gold (curated, business-level, ready to consume). And data-lake federation lets organisations share data across nodes (an operational / service / policy stack per node, and a federation / cooperation layer above), doing integration pay-as-you-go — temporary, metadata-based links instead of a permanent mediated schema.
Four architectures side by side, including the warehouse of Chapters 6–8. The row worth reading twice is the last one: each architecture’s characteristic way of going wrong separates them far better than its feature list does.
The data mesh (Dehghani) rests on four principles, and treats data as a first-class product:
Domain ownership
Each business domain owns its data end-to-end (creation, quality, pipelines, serving, access) — ownership shifts from a central ETL team to the domains.
Data as a product
Domains publish data products (the architectural quantum) with SLAs, metadata, APIs and quality tests — the DATSIS properties: Discoverable, Addressable, Trustworthy, Self-describing, Interoperable, Secure.
Self-serve platform
A platform provides the tooling so domains build products without specialist infrastructure knowledge.
Federated computational governance
Global standards (naming, security, interoperability, lineage) are enforced automatically across the autonomous domains.
FAIR data
Cutting across all of these, research data should be FAIR: Findable (unique persistent id + rich metadata), Accessible (standard protocols), Interoperable (common vocabulary/format), and Reusable (clear licence + documentation) — the goal being maximal, machine-friendly reuse.
04 · Frontier
Uncertainty & provenance
Real data is not the certain, closed-world set a database assumes: sensor readings are unreliable, a person may be undecided. Uncertain databases (probabilistic or fuzzy) model a set of possible worlds compatible with the uncertainty, and answer queries over them — uncertainty enters integration through unreliable sources, approximate (auto-matched) mappings, imprecise mediated schemas and keyword queries.
Data provenance (a.k.a. lineage or pedigree) records where a datum came from and how it was produced — who created it, when, and how (a stored value, a computation, a sensor). It has two equivalent views — annotations on tuples/values, or a graph with tuples as vertices and derivations as hyper-edges — and it is used for explanations, scoring sources / data quality, and tracing source influence. Where computers struggle (judging image content, extracting web data), crowdsourcing (e.g. Amazon Mechanical Turk) turns hard integration tasks — wrapping, checking schema mappings, record matching — into human microtasks.
05 · 2025/26 emphasis
Ethics in data management
Big data’s five V’s now include value — and with data touching every part of life, the ethics of using it has become a first-class concern. A recommendation engine can quietly discriminate (surfacing good jobs only to some groups) as a side effect of biased data or algorithms.
The objectivity illusion
“It’s an algorithm, so it must be objective” is false. Algorithms learn from data that may contain bias, and many (deep neural nets) are opaque — they cannot be explained, which is why explainability is a research area of its own. Technology has to be accompanied by ethical and legal competence, not assumed neutral.
The responsibility falls on the data scientist: choosing which datasets genuinely answer a question, understanding their contents, and picking a knowledge-extraction technique that yields a fair result — a chain of choices that can inject bias at any link. The course frames four ethical dimensions the database and AI communities must treat as fundamental:
Data protection
Safeguarding personal data — codified legally in the EU GDPR (fully enforceable since May 2018), which unifies data-protection rights and obligations across the Union.
Fairness
The lack of bias. It has many formal interpretations (e.g. equality vs equity), made urgent by the harm of training systems on biased data.
Transparency
Being able to see and explain how a data-driven decision was reached — the counter to opaque models.
Diversity
Ensuring the data and its outcomes represent the full population, not a skewed slice.
Where ethics meets the rest of the course
Ethics is not a bolt-on: it connects straight back to data quality (biased or incomplete data gives unfair results) and to provenance (you cannot audit fairness without knowing where data came from). It is the through-line the 2025/26 course closes on.
Load-bearing ideas
- Governance = authority and control over data assets (the how/who of DAMA); organizational models trade centralization vs autonomy — centralized, decentralized (silo risk), federated. Metadata (glossary / dictionary / catalog) and lineage make data usable and auditable.
- Data virtualization = logical integration at scale: a unified semantic model, data left in place and fetched on demand (Enterprise → Extended → Augmented; connect–represent–use).
- Architectures: data lake (raw, zoned, needs a catalog or becomes a swamp), lakehouse (lake
- warehouse analytics + governance), data mesh (domain-owned data products, DATSIS, self-serve, federated governance); medallion (bronze/silver/gold) and FAIR cut across.
- Frontiers: uncertain databases model possible worlds; provenance records origin/derivation (annotations or graph); crowdsourcing handles human-easy, computer-hard tasks.
- Ethics: algorithms are not automatically objective; responsibility sits with the data scientist; the four dimensions are data protection (GDPR), fairness (lack of bias), transparency, diversity.
- Exam radar: no past-exam evidence for this chapter — but if a fresh Part I theory question appears, the likely targets are governance organizational models, data virtualization vs materialization, DW vs lake vs mesh, or ethics/fairness.