VPC CNI and Istio Overview

✅ VPC CNI Plugin (aws-node DaemonSet)

• Purpose: Pod networking — assigns ENIs and IP addresses from your VPC subnets to pods.
• Scope: Controls how pods communicate at the network layer (Layer 3/4).
• Key Features:
  • Each pod gets a VPC IP — visible inside the VPC.
  • Native integration with AWS networking/security groups.
  • Enables Kubernetes services, DNS, etc.

✅ Istio (Service Mesh)

• Purpose: Handles application-level traffic control (Layer 7).
• Scope: Service-to-service communication inside your cluster.
• Key Features:
  • mTLS encryption between services.
  • Traffic routing, retries, circuit breaking.
  • Observability: metrics, tracing, logging.
  • Policy enforcement.

---

🧠 Analogy

Layer	Role	Tool	Analogy
L3/L4	IP, Ports	VPC CNI	Road + Traffic lanes
L7	HTTP, gRPC logic	Istio	Traffic lights + Checkpoints

🧠 What is the VPC CNI Plugin?

The Amazon VPC CNI plugin (aws-node DaemonSet) allows EKS pods to receive native VPC IPs, letting them directly communicate within the VPC.

🔍 Key Features of the VPC CNI Plugin

Feature	Description
Pod IPs from VPC	Assigned from ENI’s secondary IP pool in the VPC subnet.
Native VPC Routing	No overlay network. Uses native AWS networking.
Security Groups for Pods	Different SGs per pod (via SG for Pods).
PrivateLink Compatible	Pods communicate over VPC PrivateLink.
CloudWatch Metrics	Export CNI metrics.

🧩 How It Works

1. ENI Allocation: EC2 nodes get multiple ENIs with multiple IPs.
2. aws-node DaemonSet: Allocates IPs by managing ENIs and assigning IPs to pods.
3. Pods Get VPC IPs: No NAT, direct VPC communication.

📏 Default IP Allocation Strategy

Each EC2 type has ENI and IP limits. For m5.large:

• 3 ENIs (1 primary + 2 secondary)
• 10 IPv4 addresses per ENI
• Max pods: (3 * 10) - 1 = 29

Use:

curl https://raw.githubusercontent.com/awslabs/amazon-eks-ami/master/files/eni-max-pods.txt

⚙️ Custom Networking Mode (Secondary CIDR)

• Assign pod IPs from different subnets.
• Helps with IP exhaustion or network segmentation.
• Needs branch ENIs in custom subnets.

📉 Observability & Metrics

Enable metrics with Helm:

helm install aws-vpc-cni \
  --namespace kube-system \
  --set enableNetworkPolicy=true \
  --set env.ENABLE_PROMETHEUS=true \
  aws/aws-vpc-cni
    

Metrics exposed:

• vpc_cni_ip_assigned
• vpc_cni_ip_in_use
• vpc_cni_eni_allocations_failed_total

⚠️ Common Issues

Problem	Cause	Fix
Pods stuck in Pending	No IPs on ENIs	Scale node group or increase ENIs/IPs
ENI allocation fails	Missing IAM permissions	Check policy for ec2:AssignPrivateIpAddresses
Pod IPs not released	CNI bug/misconfig	Update CNI
IP Exhaustion	No free IPs in subnet	Use secondary CIDRs or split traffic

📘 Configuration Parameters

Variable	Description	Example
WARM_IP_TARGET	Free IPs to keep per node	3
MAX_ENI	Max ENIs to allocate	4
ENABLE_POD_ENI	Enable SGs for pods	true
AWS_VPC_K8S_CNI_LOGLEVEL	Debug log level	DEBUG

Check with:

kubectl -n kube-system describe daemonset aws-node

🔐 Security Groups for Pods (Advanced)

• Needs prefix delegation or custom networking
• Assign SGs per pod
• Compatible with Calico network policies

✅ When to Use VPC CNI

Use Case	Use VPC CNI?
Need VPC-native networking	✅ Yes
Pods need access to RDS, S3, ELB	✅ Yes
Using service mesh (e.g., Istio)	✅ Use alongside
Large clusters with IP pressure	⚠️ Use custom networking
High-density, internal clusters	❌ Use Calico/Cilium

📐 EC2 ENI/IP Limits and Pod IP Assignment

1. Step 1: Check ENI & IP limits per instance type (e.g., m5.large)
2. Step 2: Max Pods = (ENIs * IPs per ENI) - 1 = 29
3. Step 3: Pod IPs are assigned from subnet CIDR (e.g. 10.0.0.0/24)
4. Step 4: Example allocation:

   ENI Name	IP Addresses
   eth0 (primary)	10.0.1.10 to 10.0.1.19
   eni1	10.0.1.20 to 10.0.1.29
   eni2	10.0.1.30 to 10.0.1.39

5. Step 5: aws-node assigns IPs as secondary IPs on ENIs, directly usable by pods

📌 Important Notes

• Pods per node limited by ENI/IP limits
• Need more? Use larger instances or custom networking mode
• Ensure VPC subnet has enough free IPs

Summary Table (m5.large)

Resource	Count
ENIs per node	3
IPs per ENI	10
Total IPs	30
Pods allowed	29