Skip to content

Independent architecture study

Website events with a history

Near Real-Time Server-Side Website Analytics on AWS

A business wants to inspect current website activity and retain the same events for later analysis. This independent architecture study follows the AWS API Gateway variant, collecting browser events through server-side tagging before feeding separate historical and operational views.

An independent architecture study based on public AWS guidance. A design exercise, not a client engagement.

Workload
Website interaction events collected through Google Tag Manager
Constraint
Serve near real-time analysis and historical reporting from the same event stream
Approach
An ECS collector sends events through private API Gateway integration to Kinesis

What makes this difficult

  • Collection needs ownership

    Browser tags still generate events, but the business needs a controlled collection point before routing those events to different destinations.

  • Different views, different needs

    Operational analysis needs recent events quickly, while historical reporting needs retained files that remain available for later business questions.

  • Copies have consequences

    Events sent to external collectors create additional data paths, so transfer costs and sensitive fields need attention at each destination.

Follow a website interaction

A browser event reaches the business collector, then branches into retained files and a near real-time search view.

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  1. 01

    Capture the website interaction

    Triggers a siteeventWebsite visitorRoutes theeventRoute 53 + ALBCollectsserver-sideAmazon ECS

    Configured browser pixels fire as visitors use the website, and Route 53 with the load balancer routes those events to the tagging service on ECS.

    Illustrative example Illustrative example: a product-page interaction reaches the server-side collector.

  2. 02

    Enter the stream privately

    Collectsserver-sideAmazon ECSAccepts privatePOSTsAPI GatewayRetains eventstreamKinesis Streams

    The collector posts events through a VPC interface endpoint to private API Gateway, whose direct Kinesis integration places them on the stream without custom producer code.

    Illustrative example Illustrative example: the collected interaction enters the shared event stream.

  3. 03

    Keep history and update the live view

    Retains eventstreamKinesis StreamsDelivers eventfilesData FirehoseKeeps eventhistoryAmazon S3Loads recenteventsAWS LambdaExplores liveactivityOpenSearch

    Firehose delivers stream data to S3 while a separate Lambda consumer loads OpenSearch, allowing historical retention and near real-time analysis to progress independently.

    Illustrative example Illustrative example: the interaction appears in retained files and the operational search view.

  4. 04

    Describe the retained events

    Keeps eventhistoryAmazon S3Cataloguesevent filesAWS GlueQueries eventhistoryAmazon Athena

    Glue crawls the stored event files and maintains catalogue metadata, giving Athena table definitions it can use when someone asks a historical question.

    Illustrative example Illustrative example: retained page events are described as queryable tables.

  5. 05

    Explore behaviour over time

    Keeps eventhistoryAmazon S3Cataloguesevent filesAWS GlueQueries eventhistoryAmazon AthenaCharts sitebehaviourAmazon QuickSight

    QuickSight sends queries to Athena, which uses the catalogue and S3 files to populate dashboards without relying on the live search store for historical reporting.

    Illustrative example Illustrative example: a dashboard compares engagement across website pages.

  • Request
  • Data
  • Later, queued
  • Control

The hard part

The same event serves different clocks

The stream feeds independent consumers: Firehose writes files for later analysis, while Lambda loads OpenSearch for a near real-time view. Neither destination waits for the other to finish. An optional Firehose transformation can remove sensitive fields before S3 storage, but that transformation does not automatically sanitise the separate Lambda consumer or external collector paths.

Shares collectedeventsKinesis StreamsDelivers storedhistoryData FirehoseRetains eventfilesAmazon S3Loads the searchindexAWS LambdaShows recentactivityOpenSearch
Illustrative: a product-page event reaches historical storage and a live index through separate consumers.

Why this design

  1. Own the collection point

    Chosen
    Run Google Tag Manager server-side on ECS behind a load balancer.
    Because
    A central collector gives the business control over routing events to AWS and external analytics tools.
    Cost
    The team must operate tagging and preview services as well as the browser configuration.
    Reconsider when
    Keep a simpler collection design when existing reporting tools already meet the business requirements.

  2. Use the direct integration variant

    Chosen
    Send private API Gateway requests directly to Kinesis Data Streams.
    Because
    This variant avoids custom producer code between the ingestion API and the stream.
    Cost
    The integration must fit the event format and ingestion requirements of the tagging service.
    Reconsider when
    Assess the guidance producer-service variant when ingestion requires producer-specific behaviour.

  3. Separate historical and live consumers

    Chosen
    Use Firehose for S3 history and a Lambda consumer for OpenSearch.
    Because
    Retained reporting data and near real-time operational search serve different access patterns.
    Cost
    Separate consumers increase storage, monitoring and data-handling responsibilities.
    Reconsider when
    Remove the live index when historical dashboards satisfy the reporting need.

Cost and operating assumptions

  • Collection and ingestion

    ECS tagging and preview compute, the load balancer and API Gateway contribute to the collection cost.

  • Streaming and retained data

    Kinesis capacity, Firehose delivery volume and S3 retention grow with event traffic and stored history.

  • Analysis destinations

    Lambda processing, OpenSearch capacity, Athena scans and QuickSight access support the live and historical views.

  • Data transfer

    The guidance flags transfer within the Region and outbound traffic to Google Analytics and each third-party collector as costs to estimate early.

Assumptions behind the numbers

No monthly total is published for this guidance; pricing depends on usage, region and options, including event size, retention, search capacity and external destinations.

How it stays reliable

  • Observe the pipeline

    Services publish CloudWatch metrics for dashboards and alarms, with SNS supporting incident notification and escalation.

  • Protect transport and storage

    The guidance uses HTTPS, KMS encryption for S3 and a VPC interface endpoint for the private ingestion API.

  • Replace failed collectors

    ECS replaces non-responding containers, while the architecture places components across Availability Zones in the same Region.

  • Keep recovery material

    OpenSearch automated snapshots and Firehose backup to S3 provide recovery options for the analysis paths.

When this is a poor fit. A poor fit for a small website that only needs occasional traffic reports and cannot justify operating both a streaming pipeline and a live search destination.

Full technical detailThe numbered AWS flow, every service and the Well-Architected notes.

How the AWS guidance describes it

  1. Google Tag Manager UI. The control plane configures pixel tags for business-relevant interactions using Server Side Tagging mode.
  2. Browser pixel. The configured pixel code fires events as visitors interact with the website.
  3. Amazon Route 53 and Application Load Balancer. DNS and load balancing direct browser events to the backend service.
  4. Amazon ECS. The Google Server Side Tag Manager service runs on ECS and receives events from the load balancer.
  5. Amazon ECS preview service. A separate preview service tests tag configuration and connects to the client and tagging service through a load balancer and a Route 53 private hosted zone record.
  6. Amazon API Gateway. The tagging service sends HTTPS POST requests to a private API Gateway endpoint through a VPC interface endpoint.
  7. Google Analytics and third-party collectors. The tagging service also sends events to existing external collectors without requiring front-end code changes for those destinations.
  8. Amazon Kinesis Data Streams. API Gateway delivers requests to Kinesis through a direct integration.
  9. Amazon Data Firehose and Amazon S3. Firehose receives Kinesis data and stores it in S3 for long-term retention and historical or ad-hoc analysis.
  10. AWS Lambda. An optional Firehose Lambda transformation enriches events or removes sensitive data before S3 storage.
  11. AWS Glue and Amazon Athena. The crawler and Data Catalog capture metadata so Athena can query the retained event data.
  12. Amazon QuickSight. QuickSight uses Athena queries to build interactive dashboards.
  13. AWS Lambda. A Lambda function is configured as a secondary consumer of the same Kinesis stream.
  14. Amazon OpenSearch Service. The Kinesis-integrated Lambda consumer loads OpenSearch for near real-time analysis and observability.
  15. Amazon S3, Amazon Personalize and AWS Clean Rooms. Stored website events provide a foundation for customer profiles, personalisation, privacy-enhanced collaboration and AI or machine learning use cases.

Well-Architected

Operational Excellence
CloudWatch metrics, dashboards and alarms combine with SNS for incident notification and escalation.
Security
KMS protects stored S3 data, HTTPS protects transport and the collector accesses the private API through a VPC endpoint.
Reliability
ECS replaces unresponsive containers, while OpenSearch snapshots and Firehose S3 backup support recovery.
Performance Efficiency
Components share a Region and span Availability Zones, with automated deployment supporting regional placement.
Cost Optimisation
Estimate outbound transfer early alongside the costs of supporting near real-time analysis across Availability Zones.
Sustainability
Elastic ECS scaling and decoupled managed services align allocated resources with the workload.