Skip to main content

Request Flow and Console Runbook

This runbook explains how website requests enter AWS and how to follow downstream processing to API, Horizon, Scheduler, and Ops services.

Stage Prefixes​

Use these stage prefixes while searching in AWS console:

  • HubDev (resource names usually start with hubdev-)
  • HubStg (resource names usually start with hubstg-)
  • HubProd (resource names usually start with hubprod-)

End-to-End Path​

flowchart LR
  Browser[Browser or Client] --> Cloudflare[Cloudflare DNS]
  Cloudflare --> CloudFront[CloudFront]
  CloudFront --> WAF[WAF]
  WAF --> ALB[ALB HTTPS 443]
  ALB --> TG[Target Group HTTP 80]
  TG --> API[ECS API Service]

  API --> RDSProxy[RDS Proxy]
  RDSProxy --> Aurora[Aurora MySQL]
  API --> Redis[ElastiCache Redis]
  API --> EventBridge[EventBridge]
  EventBridge --> SQS[SQS Queues]
  Redis --> Horizon[ECS Horizon Service]
  SQS --> Horizon
  Scheduler[ECS Scheduler Service] --> Redis
  Ops[ECS Ops Service] --> RDSProxy

Critical Clarification​

  • Public inbound website traffic goes only to API tasks through CloudFront and ALB.
  • Horizon, Scheduler, and Ops are private internal services.
  • Private services are not attached to public ALB target groups.

Trace One Website Request​

  1. Validate Cloudflare DNS record for the app hostname.
  2. Validate CloudFront origin points to the hub ALB.
  3. Review WAF sampled requests for allow/block outcomes.
  4. Confirm ALB listener and target group health.
  5. Open API service task logs and match timestamp/path.

Follow Fan-Out To Internal Services​

API service:

  • Confirm synchronous DB/cache path in API logs.
  • Confirm async events/messages were emitted.

Horizon service:

  • Check /ecs/hub/horizon logs.
  • Check SQS depth, in-flight count, and oldest message age.

Scheduler service:

  • Check /ecs/hub/scheduler logs.
  • Correlate expected run windows with command execution.

Ops service:

  • Check /ecs/hub/ops logs.
  • Use ECS Exec for emergency and maintenance commands.

Troubleshooting Matrix​

  1. Browser 4xx/5xx:
  • CloudFront metrics and WAF sampled requests
  • ALB target group health
  • API task logs
  1. API succeeded but async work missing:
  • SQS backlog and message age
  • Horizon desired/running counts and logs
  1. Scheduled jobs not running:
  • Scheduler desired/running counts
  • Scheduler logs for command errors
  1. Emergency artisan command needed:
  • Confirm Ops task is running
  • Exec into Ops container

Useful Resource Name Patterns​

  • Clusters: hubdev-cluster, hubstg-cluster, hubprod-cluster
  • Services: ApiService, HorizonService, SchedulerService, OpsService
  • DB proxy: hubdev-db-proxy, hubstg-db-proxy, hubprod-db-proxy
  • Runner host tag pattern: hubdev-bitbucket-runner, hubstg-bitbucket-runner, hubprod-bitbucket-runner