project · 2025-2026
Self-service data-quality platform
Designed a self-service platform for defining, running, monitoring, and investigating data-quality checks across distributed data workloads.
#data-engineering#data-quality#great-expectations#databricks#fastapi#streamlit#postgresql#pyspark#slack-alerts
The problem
Data teams needed quality checks, but each team was rebuilding the same validation, alerting, and reporting mechanics in isolation.
Approach
- Designed a shared validation SDK alongside a self-service configuration interface.
- Separated the execution layer from a control plane that records results, ownership, thresholds, and history.
- Made checks reusable, observable, and accessible to both engineers and non-engineering users.
How it works
The platform separates validation execution from shared configuration and reporting. Teams keep control of their data while sharing contracts, validation history, and alerting conventions.
POST validation results
install SDK
inside the control plane
- ·REST API receives result records from every team's data plane
- ·Fact / dim relational schema: history is insert-only, configs are upsertable
- ·No-code UI for non-engineers; same backend as the developer SDK
- ·Cross-team dashboards, SLA tracking, trend analysis all roll up here
Outcome
The platform provided a common path for teams to adopt reliable data-quality practices while retaining ownership of their data and rules.