Independent architecture study
From files to governed dashboards
One-Click Serverless Analytics Stack on AWS
A business has useful data in export files but no dedicated platform team. This independent architecture study follows AWS guidance to deploy a governed path from uploaded files to dashboards, using a configurable CloudFormation template.
An independent architecture study based on public AWS guidance. A design exercise, not a client engagement.
- Workload
- Business reporting from uploaded flat files
- Constraint
- Limited engineering capacity and controlled access to data
- Approach
- A template deploys storage, governance, cataloguing, SQL queries and dashboards
- One templateStack deploymentPublished reference figure. The guidance deploys and configures the analytics stack through a single CloudFormation template.
- DailyCatalogue refreshPublished reference figure. The Glue crawler runs once a day to update table metadata when new data is available.
What makes this difficult
Too much wiring
Storage, permissions, table definitions and dashboards must agree before a business can ask useful questions of its files.
Access needs boundaries
A dashboard needs permission to read business data, but that does not justify opening the underlying bucket to everyone.
Freshness has a limit
Uploaded files can change before the scheduled crawler refreshes their metadata, so reporting expectations must account for that delay.
Follow a business report
An uploaded file becomes a governed table that a dashboard can query when someone opens a visual.
- 01
Put the files in one place
An administrator creates the data bucket and uploads flat files, giving the reporting tools a shared location from which to discover and read business data.
Illustrative example Illustrative example: a sales export is available in the data bucket.
- 02
Set access before reporting
Lake Formation registers the bucket and governs catalogue permissions, granting access to the role used by QuickSight so reporting follows an explicit access boundary.
Illustrative example Illustrative example: the reporting role receives access to the governed data.
- 03
Discover what the files contain
The scheduled crawler reads the bucket and updates database and table metadata, allowing query tools to interpret the files without manually maintained table definitions.
Illustrative example Illustrative example: the catalogue describes the columns in a sales export.
- 04
Fetch the data behind a chart
QuickSight uses its access role and issues SQL to Athena when a visual loads; Athena resolves table metadata and reads the underlying files.
Illustrative example Illustrative example: a chart requests sales grouped by product.
- 05
Turn the table into a report
The administrator opens the generated QuickSight analysis and builds visuals, turning queryable files into a report that can support everyday business questions.
Illustrative example Illustrative example: a sales chart makes product patterns easier to inspect.
- Request
- Data
- Later, queued
- Control
The hard part
A table description is not the data
The crawler records what the files look like in a catalogue; it does not move their contents into a warehouse. When a visual loads, Athena first resolves that description, then reads the files in S3. Separating discovery from querying avoids a permanent query cluster, but a daily catalogue refresh is not a promise of immediate schema freshness.
Why this design
Query files on demand
- Chosen
- Use Athena to query files held in S3.
- Because
- This suits exploratory reporting without provisioning an always-running warehouse.
- Cost
- Query response times and scanned data depend on file organisation and query shape.
- Reconsider when
- Consider a warehouse when predictable interactive performance becomes the overriding requirement.
Govern access centrally
- Chosen
- Register the bucket with Lake Formation and grant access to the QuickSight role.
- Because
- The reporting path needs explicit permissions for both data and table metadata.
- Cost
- The team must understand Lake Formation permissions alongside IAM access.
- Reconsider when
- Revisit the permission model when additional tools or teams need distinct access boundaries.
Refresh metadata daily
- Chosen
- Run the Glue crawler on the daily schedule described in the guidance.
- Because
- Scheduled discovery keeps catalogue maintenance manageable for an initial reporting project.
- Cost
- New structures may wait until the next crawl before becoming queryable.
- Reconsider when
- Use a different catalogue update strategy when reporting requires more immediate schema changes.
Cost and operating assumptions
Queries
Athena charges depend on data scanned, so file organisation and the queries issued by visuals affect spend.
Dashboard access
QuickSight author and reader usage contributes to the bill under the selected pricing options.
Catalogue maintenance
Glue crawler execution and Data Catalog storage contribute costs, with the guidance scheduling a daily crawl.
File storage
S3 storage and requests depend on retained files and access patterns, while same-Region deployment reduces transfer exposure.
Assumptions behind the numbers
No monthly total is published for this guidance; pricing depends on usage, region and options, including file volume, query scans and dashboard access.
How it stays reliable
Review activity
CloudWatch and CloudTrail provide logs for operational review and incident investigation.
Constrain data access
Lake Formation governs the database and table, with access granted to the IAM role used by QuickSight.
Handle deployment failures
The guidance uses CloudFormation rollback when a resource fails to provision.
Use managed endpoints
The serverless services use AWS-managed endpoints and DNS, reducing infrastructure the team must operate.
When this is a poor fit. A poor fit when reporting requires immediate schema discovery or tightly predictable query latency across a demanding, continuously active workload.
Full technical detailThe numbered AWS flow, every service and the Well-Architected notes.
How the AWS guidance describes it
- Amazon S3. An administrator creates a data bucket and uploads flat files for visualisation in QuickSight.
- AWS Lake Formation and AWS IAM. Lake Formation registers the S3 bucket and allows access to the IAM role used by QuickSight.
- AWS Glue crawler. A crawler reads the bucket daily and updates metadata when new data is available.
- AWS Glue Data Catalog. The crawler updates database and table metadata, with permissions managed by Lake Formation.
- Amazon Athena. Athena queries the table using SQL issued by QuickSight when visuals load.
- Amazon QuickSight. A subscription is created if needed, followed by a dataset and analysis using the IAM role to query Athena and access S3 data.
- Amazon QuickSight. The user opens the analysis and creates visuals to explore the uploaded data.
Well-Architected
- Operational Excellence
- CloudWatch and CloudTrail record activity for operational review and incident response.
- Security
- Lake Formation secures catalogue resources and grants access to the QuickSight role, with resources private by default.
- Reliability
- Managed endpoints and DNS support service access, while CloudFormation rolls back failed provisioning.
- Performance Efficiency
- QuickSight, Athena and S3 provide purpose-built services for querying and extending the reporting tables.
- Cost Optimisation
- Same-Region deployment, daily crawling and queries triggered by visuals limit unnecessary usage.
- Sustainability
- On-demand queries, automatically scaling services and scheduled crawling align resource use with reporting activity.