post data-engineering · 2026-08-03 · 5 min read

Metadata is product infrastructure, not documentation

#data-engineering#metadata#data-governance#architecture#analytics

Every data platform eventually acquires a knowledge problem.

At first, the answer to “what is this table?” is in a README. Then it is in a dashboard description, a pull request, a Slack thread, and one person’s memory. Eventually somebody needs a list of active assets, their owners, their schemas, and the date a definition changed. The spreadsheet that was meant to help has become another thing nobody trusts.

This is the point where metadata stops being documentation and becomes infrastructure.

The idea is simple: treat metadata as data with a schema, history, owners, and a query interface. The mechanics are not particularly exotic. The discipline is the difficult part.

A catalogue should answer real questions

Before designing tables or buying a tool, write down the questions the catalogue must answer without manual investigation:

What data assets are available, and which ones are currently supported?
Who owns each asset and who should be contacted when it breaks?
What does a column mean, and when did its definition last change?
Which downstream reports depend on this source?
Which version of a schema or transformation was active on a particular date?

If the system cannot answer these by query, it is still mostly a wiki. Wikis are useful, but they are not a reliable join target for reporting or incident response.

Start with a stable identity

One recurring mistake is using a deployment, a storage location, or a version number as the identity of an asset. Those things change. A useful catalogue separates the enduring thing from its changing representations.

For example, customer_activity_daily is the stable data asset. The schema published on 15 July is one version. A move from one storage location to another is another operational change, but it should not make every historical report believe an entirely new asset appeared.

create table dim_data_asset (
  asset_id              bigint primary key,
  stable_key            varchar not null unique,
  display_name          varchar not null,
  asset_type            varchar not null, -- table, stream, api, dashboard
  domain                varchar not null,
  owner_team            varchar not null,
  description           varchar not null,
  lifecycle_status      varchar not null, -- draft, active, deprecated, retired
  created_at            timestamp not null
);

create table dim_data_asset_version (
  asset_version_id      bigint primary key,
  asset_id              bigint not null references dim_data_asset(asset_id),
  version_label         varchar not null,
  effective_from        timestamp not null,
  effective_to          timestamp,
  schema_ref            varchar,
  source_ref            varchar,
  change_summary        varchar not null,
  published_at          timestamp not null,
  unique (asset_id, version_label)
);

The stable record answers “what is this?”. The version record answers “what did it look like at that time?”

That distinction looks small in a schema diagram and saves a surprising amount of pain later.

History is not optional

Current metadata is easy to maintain. Historical metadata is where catalogues either become valuable or quietly fail.

Suppose a column changes from gross_amount to net_amount. Updating a description in place makes the catalogue accurate today and misleading for every report built before the change. The same applies to ownership, classifications, schemas, and quality expectations.

The useful invariant is:

For one asset, version effective-time ranges must never overlap.

When a definition changes, close the previous version and create a new one. Do not overwrite history. You will need it at exactly the moment when a number in an old report looks strange and the original context has disappeared.

select
  a.stable_key,
  a.display_name,
  v.version_label,
  v.change_summary,
  v.schema_ref
from dim_data_asset a
join dim_data_asset_version v
  on v.asset_id = a.asset_id
where a.stable_key = 'customer_activity_daily'
  and timestamp '2026-07-20 12:00:00' >= v.effective_from
  and (
    v.effective_to is null
    or timestamp '2026-07-20 12:00:00' < v.effective_to
  );

This query is deliberately unexciting. If a historical question requires a detective story, the metadata model is doing too little.

Make ownership and purpose required

Technical metadata is usually the easy part. Schema information can be extracted. Locations can be discovered. Lineage can often be inferred.

The metadata that makes a catalogue useful to people is more stubborn: ownership, purpose, support expectations, and lifecycle. Those fields need a clear policy.

I would make these mandatory before an asset is marked active:

Field	Why it matters
`owner_team`	Gives incidents, questions, and change requests somewhere to go.
`description`	Explains the intended meaning rather than only the physical shape.
`domain`	Lets people browse by business area without parsing names.
`lifecycle_status`	Prevents deprecated assets appearing as recommended choices.
`change_summary`	Provides the human reason behind a version change.
`schema_ref`	Connects the catalogue to the contract consumers need to follow.

An empty owner is not harmless missing information. It is a deferred support problem with no destination.

Keep metadata close to the thing it describes

A catalogue gets stale when updating it is a separate chore after a code or schema change. The better model is that important metadata moves through the same delivery path as the asset itself.

For a transformation or data product, a small manifest can sit alongside the code:

stable_key: customer_activity_daily
display_name: Customer activity, daily
asset_type: table
domain: customer-analytics
owner_team: analytics-platform
description: Daily aggregate of customer interactions for reporting.
schema_ref: contracts/customer-activity-v3.yaml
change_summary: Added the interaction channel dimension.

The release process validates the manifest and publishes the new version metadata with the change. That does not eliminate all manual work, but it ensures the operational fields do not depend on somebody remembering an extra update after deployment.

There is still a place for a UI: richer descriptions, stewardship workflows, tags, and ownership handovers are often easier there. The point is to avoid asking people to enter the same operational facts twice.

One consumer view beats ten raw tables

Metadata systems tend to accumulate normalised tables, event logs, source snapshots, and ingestion status. Those are useful for the platform itself. They are not a friendly interface for analysts, engineers, or data stewards.

Create a consumer-facing view for the common case:

create view catalog_active_assets as
select
  a.stable_key,
  a.display_name,
  a.asset_type,
  a.domain,
  a.owner_team,
  a.description,
  v.version_label,
  v.effective_from,
  v.schema_ref,
  v.source_ref,
  v.change_summary
from dim_data_asset a
join dim_data_asset_version v
  on v.asset_id = a.asset_id
where a.lifecycle_status = 'active'
  and v.published_at <= current_timestamp
  and (v.effective_to is null or v.effective_to > current_timestamp);

This view becomes the stable surface for discovery, reporting, and eventually a UI. Consumers should not need to understand effective dating or internal identifiers to find a supported dataset.

For historical questions, expose a documented query pattern or a table function rather than silently using a “current” view. Current state and historical state are different questions. Mixing them is the source of very plausible wrong answers.

Lineage needs just enough structure

Lineage can become an enormous project if the ambition is to infer every field-level dependency automatically. Start with the relationships people actually need.

create table bridge_asset_dependency (
  upstream_asset_version_id    bigint not null,
  downstream_asset_version_id  bigint not null,
  dependency_type              varchar not null, -- reads_from, publishes_to, powers
  recorded_at                  timestamp not null,
  primary key (upstream_asset_version_id, downstream_asset_version_id)
);

This unlocks practical questions:

Which active dashboards depend on this source?
What should be checked before a breaking schema change?
Which downstream assets should be marked at risk after a failed pipeline?

Do not wait for perfect automatic lineage before publishing any lineage at all. A small, reliable graph that covers critical assets is more useful than an ambitious graph that is always two quarters away.

The awkward cases are the design test

The happy path is easy. Decide what should happen in the awkward cases before they arrive.

An asset changes physical location. Create a new version or update a source reference according to your history policy; retain the stable identity.

A dataset is replaced by a successor. Mark the old asset deprecated, link it to the replacement, and preserve both records. Deleting the old entry turns old reports into mysteries.

Ownership changes. Record when it changed. Current ownership is enough for routing a question; historical ownership matters when reconstructing a past incident.

A schema change is backward compatible. Still publish a version. Compatibility is a useful attribute, not a reason to erase the fact that a contract changed.

Automated extraction disagrees with a manual description. Prefer a clear source-of-truth rule: generated facts for physical metadata, reviewed fields for business meaning. An unresolved disagreement should be visible, not silently picked by whichever job ran last.

What good looks like

A useful catalogue makes the path from a question to an answer short:

asset -> version -> owner, schema, purpose, and dependencies

That helps different people for different reasons. An analyst can find a trusted source. An engineer can estimate the impact of a change. A platform team can see what has no owner or no description. A reviewer can understand what a dataset meant when an older report was produced.

The building blocks are ordinary: dimensions, version history, a few bridge tables, validation in the delivery flow, and a consumer-facing view. The shift is conceptual. Metadata is not the explanatory text added after the product is finished. It is part of how the product stays understandable as it changes.