Monday, 30 June 2025

Architecture Design - Lambda, ECS, EKS

 

✅ 1. AWS Lambda – Event-Driven Serverless App

🟢 Use AWS Lambda When:

  • You have event-driven workloads (e.g. file upload triggers, API Gateway, Kinesis).

  • Functions are stateless and execution time < 15 min.

  • You want to minimize DevOps/infrastructure management.

  • You don’t need advanced networking like service mesh or complex security groups.

Example use cases:

  • Image processing after S3 upload.

  • Lightweight backend microservices.

  • Scheduled jobs via EventBridge.

  • IoT data ingestion.

🎯 Use Case: Real-time Image Processing for a News Website (e.g., Dow Jones or NYT)

🛠️ Architecture:

  • A user uploads an image to S3.

  • An S3 event trigger invokes an AWS Lambda.

  • Lambda resizes the image, applies watermarks, and stores it in another S3 bucket.

  • CloudWatch Logs and X-Ray provide monitoring and tracing.

🔍 Why Lambda:

  • Instant reaction to events.

  • No infrastructure to manage.

  • Cost-efficient for low-latency, sporadic workloads.

  • Scaling is automatic based on file uploads.

✅ 2. Amazon ECS (with Fargate) – Containerized Batch Processing

🔵 Use ECS (preferably with Fargate) When:

  • You want to run containers without managing servers.

  • Simpler than EKS and you don't need full Kubernetes features.

  • Great for batch jobs, cron jobs, or small-to-mid scale container workloads.

Example use cases:

  • RESTful APIs in containers.

  • Scheduled ETL jobs.

  • Background processing microservices.

🎯 Use Case: Financial Data Ingestion and Transformation Job 

🛠️ Architecture:

  • A scheduled job runs every hour (via EventBridge or Step Functions).

  • ECS Task (Python app in a Docker container) pulls data from a vendor API.

  • It performs data cleaning, transformation, and stores it in RDS/S3.

  • Runs on Fargate, no EC2 management.

🔍 Why ECS:

  • Use Docker for packaging complex libraries (e.g., pandas, NumPy).

  • No need to manage Kubernetes (EKS overkill).

  • Better for stateful, long-running batch jobs.

  • IAM roles per task ensure secure data access.

✅ 3. Amazon EKS – Enterprise Microservices Platform

Use EKS When:

  • You need fine-grained control over orchestration and want to use the Kubernetes ecosystem.

  • Your team is experienced with Kubernetes.

  • You need:

    • Custom ingress controllers

    • Service mesh (Istio, AWS App Mesh)

    • RBAC, network policies

    • Multi-tenant isolation

    • Advanced CI/CD deployment strategies

🎯 Use Case: Multi-Tenant SaaS Platform for Trading 

🛠️ Architecture:

  • Frontend + backend microservices deployed as Kubernetes Deployments.

  • Istio handles service mesh with traffic routing, observability (Prometheus, Grafana).

  • Tenants are separated by Namespaces.

  • ArgoCD manages GitOps-based CI/CD.

  • App uses Kafka for pub/sub messaging and EFS for shared storage.

🔍 Why EKS:

  • Complex microservices require custom Ingress, TLS termination, mTLS, canary deploys.

  • Separate teams manage different namespaces securely using RBAC.

  • Network segmentation with Calico policies.

  • Built-in autoscaling, self-healing, and logging (Fluentd + OpenTelemetry).

🔄 Combined Architecture Real-World Example

🏢 Company: FinTech Startup

  • Frontend deployed on Lambda + API Gateway (low-latency, low-traffic).

  • Authentication & ETL services on ECS (with Docker).

  • Main trading engine, risk analysis, audit logs, and multi-tenant APIs run on EKS (with full control and compliance policies).

📌 Summary

Real-World Use CaseService UsedWhy
Real-time file/image processingLambdaEvent-driven, fast scaling, minimal infra
Hourly ETL job with custom dependenciesECS (Fargate)Dockerized app, periodic workload
Multi-service trading platformEKSAdvanced CI/CD, service mesh, multi-tenancy

❌ Features Not Fully Supported in ECS (compared to EKS)

FeatureECS SupportEKS Support
Custom Ingress Controllers❌ Not supported✅ Full support (e.g., NGINX, ALB Ingress, Traefik)
Service Mesh (Istio, AWS App Mesh)⚠️ Very limited / unofficial✅ Fully supported
RBAC (Role-Based Access Control)❌ No Kubernetes-style RBAC✅ Full Kubernetes RBAC
Network Policies❌ Not available✅ Calico or Cilium (fine-grained control)
Multi-tenant Isolation⚠️ Only via IAM and networking tricks✅ Kubernetes Namespaces + Policies
Advanced CI/CD (e.g., canary, blue-green, GitOps)⚠️ Requires custom scripts/pipelines✅ Native via ArgoCD, Flagger, etc.

🔍 Detailed Explanation

🔸 1. Custom Ingress Controllers

  • ECS: Only supports ALB/NLB directly with limited customization.

  • EKS: You can plug in any ingress controller: NGINX, Contour, HAProxy, Traefik, etc.

🔸 2. Service Mesh (App Mesh / Istio)

  • ECS: App Mesh can be integrated, but it’s complex, not native, and rarely used in practice.

  • EKS: Istio, App Mesh, or Linkerd can be natively deployed via Helm or CRDs. They support sidecar injection, traffic shifting, mTLS, and telemetry.

🔸 3. RBAC

  • ECS: Permissions are managed via IAM roles (coarse-grained).

  • EKS: Uses Kubernetes RBAC at the cluster, namespace, or resource level.

🔸 4. Network Policies

  • ECS: Lacks native support for pod-level or container-to-container firewall rules.

  • EKS: You can define K8s NetworkPolicies using Calico, Cilium, or AWS native options.

🔸 5. Multi-Tenant Isolation

  • ECS: Isolation is done using separate clusters, VPCs, or task definitions with IAM — not ideal for soft multi-tenancy.

  • EKS: Can isolate tenants via namespaces, network policies, and resource quotas — all native to Kubernetes.

🔸 6. Advanced CI/CD

  • ECS: Basic blue/green possible via CodeDeploy, but no native GitOps or canary support.

  • EKS: Supports:

    • GitOps (ArgoCD, Flux)

    • Canary Deployments (Flagger)

    • Progressive delivery (with Istio/App Mesh)

    • Helm, Kustomize, and native CRD-based pipelines


Posted by at No comments:
Subscribe to: Posts (Atom)