Secure Data Mesh with Distributed Data Asset Ownership on AWS

This case study documents a data mesh architecture on AWS that solves the fundamental tension in enterprise data platforms: how to scale data access across many business domains without creating a centralised bottleneck or ungoverned sprawl from decentralised teams producing siloed, inconsistent data assets.

The architecture implements the Data Mesh paradigm (federated data ownership with centralised governance) using AWS-native services. Each business domain owns, curates, and publishes its own data products to a central AWS DataZone catalog. Consumers discover, subscribe, and access data products through DataZone with fine-grained Lake Formation permissions, without any data movement. Every access is audited via CloudTrail, encrypted via KMS, and governed by IAM Identity Center federated roles.

Enterprise data platforms evolve through a predictable failure sequence. They start centralised (a single data warehouse managed by a central team), then become bottlenecks (every domain waits for the central team to build pipelines), then fragment (domains build their own shadow data systems), then produce inconsistency (three domains produce three different definitions of "active customer").

The architecture operates across three tiers in a multi-account AWS Organisation structure: domain producer accounts (one per business domain), a central governance account (AWS DataZone + IAM Identity Center), and consumer accounts (analytics, ML, BI teams). Data never leaves its producer account; access is granted and audited through the governance layer.

Tier 1: Domain Producer Accounts

• Each business domain (e.g. Finance, Supply Chain, Customer, Product) operates in its own AWS account, providing IAM boundary isolation, independent cost attribution, and blast radius containment
• Domain data is stored in S3 with a consistent naming convention: raw layer (as-ingested from source systems), curated layer (cleansed, typed, deduped), and product layer (business-logic-applied, consumer-ready)
• AWS Glue Crawlers run on schedule against the S3 product layer, automatically discovering schemas and updating the domain Glue Data Catalog; new tables and schema changes are reflected in the catalog without manual intervention
• AWS Lake Formation applies fine-grained access control on each table: column-level permissions (masking PII columns for non-privileged consumers), row-level filtering (restricting regional consumers to their own geography), and tag-based access control (classifying tables as PII, CONFIDENTIAL, or PUBLIC and granting access by tag)

Tier 2: Central Governance Account

• AWS DataZone is the central data catalog. Domain data stewards publish curated data products to DataZone from their Glue Data Catalog, adding business metadata: plain-English descriptions, data quality SLAs, refresh frequency, data owner contact, and lineage documentation
• Consumers browse the DataZone business catalog (not a technical catalog of table names and column types, but a product catalog with business context) and submit subscription requests for the data products they need
• Governance admins configure subscription approval rules: auto-approval for PUBLIC-tagged assets, human approval for CONFIDENTIAL assets, and deny-with-explanation for PII assets unless the consumer account has a DPA (Data Processing Agreement) approval flag
• Upon subscription approval, DataZone issues cross-account Lake Formation access grants to the consumer account, granting the minimum column and row-level permissions required by the consumer without requiring any data movement or ETL pipeline
• IAM Identity Center (formerly AWS SSO) federates enterprise identity (Active Directory, Okta, Azure AD) into AWS; data stewards, consumers, and governance admins authenticate with their corporate credentials and receive role-appropriate access to DataZone and Lake Formation
• AWS KMS customer-managed keys (CMK) are domain-owned: each producer account holds its own KMS key for S3 and Glue encryption. Cross-account key grants allow authorised consumer accounts to decrypt data during query execution; key policy changes immediately revoke consumer access without modifying Lake Formation grants

Tier 3: Consumer Accounts

• Amazon Athena: serverless SQL queries against the governed data products directly from S3 in the producer account, billed per TB scanned; no infrastructure to provision, no idle cluster costs. Athena results are cached in the consumer account S3 for repeat queries
• Amazon Redshift: for consumers requiring structured data warehousing and complex SQL analytics, Redshift Spectrum queries the producer S3 data via Lake Formation-governed external tables, combining governed domain data with Redshift-native tables in a single query
• Amazon EMR: Spark and Hadoop workloads in the consumer account access governed S3 datasets using Lake Formation-aware EMR clusters for large-scale data processing with the same column/row-level access controls that govern Athena queries
• Amazon SageMaker: ML engineers discover data products in DataZone, subscribe, and access governed datasets directly from SageMaker Studio notebooks, enabling feature engineering and model training on governed enterprise data without bypassing the access control layer
• Amazon QuickSight: BI dashboards connect to Athena or Redshift query results, providing business users with self-service analytics on governed data without direct S3 or Lake Formation access
• Amazon Bedrock: GenAI applications use DataZone-governed datasets as knowledge bases or context for RAG (Retrieval-Augmented Generation) patterns; LLM responses are grounded in governed, trusted enterprise data with full access audit trails

The Data Mesh paradigm (Zhamak Dehghani, 2019) defines four principles. This AWS architecture applies each principle through specific AWS services and organisational patterns.

Fine-Grained Access Control with Lake Formation

• Column-level security: PII columns (name, email, SSN, date of birth) are masked or excluded from consumer queries unless the consumer account has explicit column-level Lake Formation grants; a data analyst can query transaction amounts without seeing customer names
• Row-level filtering: regional compliance requirements (e.g. GDPR data residency) are enforced by Lake Formation row filters; a European consumer account only receives rows where the data_region = "EU" column matches, without requiring separate regional copies of the dataset
• Tag-based access control (TBAC): tables are tagged with security classifications (PII, CONFIDENTIAL, FINANCIAL, PUBLIC). IAM Identity Center roles are mapped to Lake Formation tag permissions; a GDPR Data Officer role automatically receives access to all PII-tagged tables across all domains without per-table grant management
• Attribute-based access control: consumer access can be further restricted by user attributes (department, seniority, geographic location) inherited from IAM Identity Center federation; a junior analyst automatically receives a more restrictive Lake Formation policy than a senior data scientist in the same team

Encryption and Key Management

• Domain-owned KMS keys: each producer account owns the KMS CMK encrypting its S3 data and Glue catalog metadata. Key policies grant decrypt permissions to specific consumer IAM roles; revoking access is as simple as removing a key grant, immediately blocking all consumer queries without modifying Lake Formation permissions
• Glue catalog encryption: both the Glue Data Catalog metadata (table schemas, partition information) and the catalog connection passwords are encrypted at rest using KMS CMKs; metadata disclosure is as controlled as data access
• S3 bucket policy enforcement: all producer S3 buckets enforce aws:SecureTransport and deny any request that does not originate from approved VPC endpoints or IAM identities; no unauthenticated access to the data lake layer is possible even from within the AWS account

Audit and Observability

• CloudTrail data events: enabled on all producer S3 buckets, so every GetObject (Athena query, Glue job read, SageMaker dataset access) is logged with the consumer IAM identity, timestamp, and object path. Provides a forensic-quality access audit trail for GDPR data subject access requests and regulatory examinations
• Lake Formation access logs: all Lake Formation permission grants and access decisions are logged, providing a record of when data access was granted, by whom, and which tables were included
• CloudWatch dashboards: Athena query metrics (queries by table, bytes scanned by consumer account, query failure rates), Glue Crawler run status, and DataZone subscription request volumes are aggregated in a governance operations dashboard visible to the central data governance team
• AWS Config rules: custom Config rules verify that all producer S3 buckets have Lake Formation enabled, all Glue tables have mandatory metadata tags, and all consumer accounts have CloudTrail data events active; governance drift is detected and alerted automatically

One of the most significant architectural benefits of the data mesh approach is that it creates a governed, discoverable foundation for generative AI applications. Amazon Bedrock and SageMaker can consume governed data products from DataZone, grounding LLM responses in authoritative enterprise data.

RAG on Enterprise Data Products

• A DataZone consumer account for the AI platform team subscribes to relevant data products (e.g. product catalog, customer interaction history, financial reporting data)
• Data products are ingested into Amazon Bedrock Knowledge Bases, creating vector embeddings stored in a managed vector store (OpenSearch Serverless) that the Bedrock retrieval system can search semantically
• Business users query the knowledge base via a natural-language interface ("what were our top 10 suppliers by spend last quarter, and how has their delivery performance trended?"). Bedrock retrieves relevant chunks from the governed data embeddings and constructs a grounded response
• Every Bedrock Knowledge Base access is audited via CloudTrail; the AI application cannot access data that the platform team account does not have Lake Formation permissions to query, maintaining the data mesh governance boundary

Amazon Q for Business on the Data Mesh

Amazon Q for Business can be configured with DataZone-governed data sources as connectors; business analysts ask questions in natural language and receive answers drawn from the governed data mesh without writing SQL or understanding the underlying table structure.

The data mesh architecture is designed for cost efficiency through zero data duplication and serverless query execution. The following estimates are indicative for a mid-size enterprise with 10 producer domains, 100 data products, and 200 active data consumers.