ClodFormation, terraform and CDK example

The complete CloudFormation Template (CFT) for creating VPC Interface Endpoints, Route 53 Private Hosted Zone, and CNAME records for Kafka brokers within an AWS environment. Each endpoint is given a friendly DNS name (like broker1.bgt.pulse.ABC) that points to its corresponding endpoint's DNS name within the VPC.

CloudFormation Template

AWSTemplateFormatVersion: '2010-09-09'
Description: CloudFormation template to create VPC endpoints and Route 53 resources for Kafka brokers in ABC environment.
 
Parameters:
  VpcId:
    Type: AWS::EC2::VPC::Id
    Default: "vpc-0example000000000"  # Replace with your default VPC ID
    Description: The ID of the VPC where the endpoints and Route 53 hosted zone will be created.
 
  SubnetIds:
    Type: List<AWS::EC2::Subnet::Id>
    Default:
      - "subnet-0example000000000"  # Replace with your default subnet IDs
      - "subnet-0example000000000"
      - "subnet-0example000000000"
    Description: List of subnet IDs where the endpoints should be created. Must be in 3 Availability Zones.
 
  Broker1ServiceName:
    Type: String
    Default: "com.amazonaws.vpc.us-east-1.vpce-svc-aaaaaaaaaaaa"
    Description: Service name for Broker1.
 
  Broker2ServiceName:
    Type: String
    Default: "com.amazonaws.vpc.us-east-1.vpce-svc-bbbbbb"
    Description: Service name for Broker2.
 
  Broker3ServiceName:
    Type: String
    Default: "com.amazonaws.vpc.us-east-1.vpce-svc-ccccccc"
    Description: Service name for Broker3.
 
  Dc1ServiceName:
    Type: String
    Default: "com.amazonaws.vpc.us-east-1.vpce-svc-ddddddd"
    Description: Service name for DC1.
 
Resources:
  # VPC Endpoints
  Broker1VPCEndpoint:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      VpcEndpointType: Interface
      VpcId: !Ref VpcId
      ServiceName: !Ref Broker1ServiceName
      SubnetIds: !Ref SubnetIds
      PrivateDnsEnabled: true
      TagSpecifications:
        - ResourceType: vpc-endpoint
          Tags:
            - Key: Name
              Value: broker1.bgt.pulse.ABC
 
  Broker2VPCEndpoint:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      VpcEndpointType: Interface
      VpcId: !Ref VpcId
      ServiceName: !Ref Broker2ServiceName
      SubnetIds: !Ref SubnetIds
      PrivateDnsEnabled: true
      TagSpecifications:
        - ResourceType: vpc-endpoint
          Tags:
            - Key: Name
              Value: broker2.bgt.pulse.ABC
 
  Broker3VPCEndpoint:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      VpcEndpointType: Interface
      VpcId: !Ref VpcId
      ServiceName: !Ref Broker3ServiceName
      SubnetIds: !Ref SubnetIds
      PrivateDnsEnabled: true
      TagSpecifications:
        - ResourceType: vpc-endpoint
          Tags:
            - Key: Name
              Value: broker3.bgt.pulse.ABC
 
  Dc1VPCEndpoint:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      VpcEndpointType: Interface
      VpcId: !Ref VpcId
      ServiceName: !Ref Dc1ServiceName
      SubnetIds: !Ref SubnetIds
      PrivateDnsEnabled: true
      TagSpecifications:
        - ResourceType: vpc-endpoint
          Tags:
            - Key: Name
              Value: dc1.bgt.pulse.ABC
 
  # Route 53 Private Hosted Zone
  PrivateHostedZone:
    Type: AWS::Route53::HostedZone
    Properties:
      Name: "bgt.pulse.ABC"  # Replace with your desired domain name
      VPCs:
        - VPCId: !Ref VpcId
          VPCRegion: "us-east-1"  # Change to your VPC region
      HostedZoneConfig:
        Comment: "Private hosted zone for Kafka brokers"
 
  # Route 53 DNS Records for VPC Endpoints
  Broker1DNSRecord:
    Type: AWS::Route53::RecordSet
    Properties:
      HostedZoneId: !Ref PrivateHostedZone
      Name: "broker1.bgt.pulse.ABC"
      Type: CNAME
      TTL: "300"
      ResourceRecords:
        - !GetAtt Broker1VPCEndpoint.DnsEntries[0].DnsName
 
  Broker2DNSRecord:
    Type: AWS::Route53::RecordSet
    Properties:
      HostedZoneId: !Ref PrivateHostedZone
      Name: "broker2.bgt.pulse.ABC"
      Type: CNAME
      TTL: "300"
      ResourceRecords:
        - !GetAtt Broker2VPCEndpoint.DnsEntries[0].DnsName
 
  Broker3DNSRecord:
    Type: AWS::Route53::RecordSet
    Properties:
      HostedZoneId: !Ref PrivateHostedZone
      Name: "broker3.bgt.pulse.ABC"
      Type: CNAME
      TTL: "300"
      ResourceRecords:
        - !GetAtt Broker3VPCEndpoint.DnsEntries[0].DnsName
 
  Dc1DNSRecord:
    Type: AWS::Route53::RecordSet
    Properties:
      HostedZoneId: !Ref PrivateHostedZone
      Name: "dc1.bgt.pulse.ABC"
      Type: CNAME
      TTL: "300"
      ResourceRecords:
        - !GetAtt Dc1VPCEndpoint.DnsEntries[0].DnsName
 
Outputs:
  Broker1EndpointDNS:
    Description: The DNS name for broker1 endpoint
    Value: !GetAtt Broker1VPCEndpoint.DnsEntries[0].DnsName
 
  Broker2EndpointDNS:
    Description: The DNS name for broker2 endpoint
    Value: !GetAtt Broker2VPCEndpoint.DnsEntries[0].DnsName
 
  Broker3EndpointDNS:
    Description: The DNS name for broker3 endpoint
    Value: !GetAtt Broker3VPCEndpoint.DnsEntries[0].DnsName
 
  Dc1EndpointDNS:
    Description: The DNS name for dc1 endpoint
    Value: !GetAtt Dc1VPCEndpoint.DnsEntries[0].DnsName

 

Explanation of Key Components:

1. VPC Endpoints (Broker1VPCEndpoint, Broker2VPCEndpoint, etc.): These resources create the VPC Interface Endpoints for each Kafka broker and dc1 service.
2. Route 53 Private Hosted Zone (PrivateHostedZone): This hosted zone allows you to manage the DNS within your VPC for the specified domain (e.g., bgt.pulse.ABC).
3. CNAME Records (Broker1DNSRecord, Broker2DNSRecord, etc.): Each record maps a user-friendly name (e.g., broker1.bgt.pulse.ABC) to the actual DNS of the VPC endpoint.
4. Outputs: Provides the DNS names of each endpoint as output values after stack deployment.

Usage Notes

• Update the default values (like VpcId, SubnetIds, and Service Names) as per your environment.
• This template assumes you are deploying it in the us-east-1 region. Adjust VPCRegion if needed.

Deploying this CloudFormation stack will create VPC Interface Endpoints, a Route 53 private hosted zone, and DNS records within your AWS environment, enabling you to use friendly DNS names for each broker in your private VPC network.

 

Terraform configuration for creating VPC Interface Endpoints, Route 53 Private Hosted Zone, and CNAME records for Kafka brokers within an AWS environment.

 

provider "aws" {

  region = "us-east-1"  # Adjust as needed

}

 

variable "vpc_id" {

  type    = string

  default = "vpc-0example000000000"  # Replace with your VPC ID

}

 

variable "subnet_ids" {

  type    = list(string)

  default = ["subnet-0example000000000", "subnet-0example000000000", "subnet-0example000000000"]  # Replace with your subnet IDs

}

 

variable "domain_name" {

  type    = string

  default = "bgt.pulse.ABC"  # Replace with your desired domain name

}

 

variable "broker1_service_name" {

  type    = string

  default = "com.amazonaws.vpc.us-east-1.vpce-svc-aaaaaaaaaaaa"  # Replace with your Broker1 service name

}

 

variable "broker2_service_name" {

  type    = string

  default = "com.amazonaws.vpc.us-east-1.vpce-svc-bbbbbb"  # Replace with your Broker2 service name

}

 

variable "broker3_service_name" {

  type    = string

  default = "com.amazonaws.vpc.us-east-1.vpce-svc-ccccccc"  # Replace with your Broker3 service name

}

 

variable "dc1_service_name" {

  type    = string

  default = "com.amazonaws.vpc.us-east-1.vpce-svc-ddddddd"  # Replace with your DC1 service name

}

 

# VPC Endpoints

resource "aws_vpc_endpoint" "broker1" {

  vpc_id             = var.vpc_id

  service_name       = var.broker1_service_name

  vpc_endpoint_type  = "Interface"

  subnet_ids         = var.subnet_ids

  private_dns_enabled = true

 

  tags = {

    Name = "broker1.${var.domain_name}"

  }

}

 

resource "aws_vpc_endpoint" "broker2" {

  vpc_id             = var.vpc_id

  service_name       = var.broker2_service_name

  vpc_endpoint_type  = "Interface"

  subnet_ids         = var.subnet_ids

  private_dns_enabled = true

 

  tags = {

    Name = "broker2.${var.domain_name}"

  }

}

 

resource "aws_vpc_endpoint" "broker3" {

  vpc_id             = var.vpc_id

  service_name       = var.broker3_service_name

  vpc_endpoint_type  = "Interface"

  subnet_ids         = var.subnet_ids

  private_dns_enabled = true

 

  tags = {

    Name = "broker3.${var.domain_name}"

  }

}

 

resource "aws_vpc_endpoint" "dc1" {

  vpc_id             = var.vpc_id

  service_name       = var.dc1_service_name

  vpc_endpoint_type  = "Interface"

  subnet_ids         = var.subnet_ids

  private_dns_enabled = true

 

  tags = {

    Name = "dc1.${var.domain_name}"

  }

}

 

# Route 53 Private Hosted Zone

resource "aws_route53_zone" "private_zone" {

  name = var.domain_name

  vpc {

    vpc_id = var.vpc_id

  }

}

 

# Route 53 CNAME Records for each VPC Endpoint

resource "aws_route53_record" "broker1_cname" {

  zone_id = aws_route53_zone.private_zone.zone_id

  name    = "broker1.${var.domain_name}"

  type    = "CNAME"

  ttl     = 300

  records = [aws_vpc_endpoint.broker1.dns_entry.0.dns_name]

}

 

resource "aws_route53_record" "broker2_cname" {

  zone_id = aws_route53_zone.private_zone.zone_id

  name    = "broker2.${var.domain_name}"

  type    = "CNAME"

  ttl     = 300

  records = [aws_vpc_endpoint.broker2.dns_entry.0.dns_name]

}

 

resource "aws_route53_record" "broker3_cname" {

  zone_id = aws_route53_zone.private_zone.zone_id

  name    = "broker3.${var.domain_name}"

  type    = "CNAME"

  ttl     = 300

  records = [aws_vpc_endpoint.broker3.dns_entry.0.dns_name]

}

 

resource "aws_route53_record" "dc1_cname" {

  zone_id = aws_route53_zone.private_zone.zone_id

  name    = "dc1.${var.domain_name}"

  type    = "CNAME"

  ttl     = 300

  records = [aws_vpc_endpoint.dc1.dns_entry.0.dns_name]

}

 

output "broker1_endpoint_dns" {

  description = "The DNS name for broker1 endpoint"

  value       = aws_vpc_endpoint.broker1.dns_entry.0.dns_name

}

 

output "broker2_endpoint_dns" {

  description = "The DNS name for broker2 endpoint"

  value       = aws_vpc_endpoint.broker2.dns_entry.0.dns_name

}

 

output "broker3_endpoint_dns" {

  description = "The DNS name for broker3 endpoint"

  value       = aws_vpc_endpoint.broker3.dns_entry.0.dns_name

}

 

output "dc1_endpoint_dns" {

  description = "The DNS name for dc1 endpoint"

  value       = aws_vpc_endpoint.dc1.dns_entry.0.dns_name

}

 

Explanation

1. VPC Endpoints (aws_vpc_endpoint resources): Creates VPC Interface Endpoints for each Kafka broker and dc1, using the specified service_name, vpc_id, and subnet_ids.
2. Route 53 Private Hosted Zone (aws_route53_zone): Creates a private hosted zone for the domain specified in domain_name, making it accessible only within the specified VPC.
3. CNAME Records (aws_route53_record resources): Creates a CNAME record in the private hosted zone for each VPC endpoint (e.g., broker1.bgt.pulse.ABC), pointing it to the DNS name of the respective endpoint.
4. Outputs: Provides the DNS names of each endpoint, allowing you to access them easily after deployment.

Usage Notes

• Adjust vpc_id, subnet_ids, and service names to match your environment.
• This configuration assumes the us-east-1 region; adjust the provider region if needed.

This Terraform configuration enables you to manage your AWS resources with a similar setup as the provided CloudFormation template, allowing access to Kafka brokers via user-friendly DNS names in a VPC.

 

AWS CDK (Cloud Development Kit) stack written in Python that creates the same resources as the previous Terraform and CloudFormation templates: VPC Interface Endpoints, Route 53 Private Hosted Zone, and CNAME records for Kafka brokers.

 

AWS CDK (Python) Code

To get started, make sure you have AWS CDK installed, and initialize a Python CDK project by running:

 

cdk init app --language python

 

Then, add the following dependencies to requirements.txt:

aws-cdk-lib

constructs

 

Then, run pip install -r requirements.txt to install dependencies.

Next, create the following CDK stack in lib/my_kafka_stack.py:

from aws_cdk import (

    Stack,

    aws_ec2 as ec2,

    aws_route53 as route53,

    aws_route53_targets as targets,

)

from constructs import Construct

 

class MyKafkaStack(Stack):

 

    def __init__(self, scope: Construct, id: str, **kwargs) -> None:

        super().__init__(scope, id, **kwargs)

 

        # Parameters for VPC, Subnets, and Service Names

        vpc_id = "vpc-0example000000000"  # Replace with your VPC ID

        subnet_ids = ["subnet-0example000000000", "subnet-0example000000000", "subnet-0example000000000"]  # Replace with your subnet IDs

        domain_name = "bgt.pulse.ABC"  # Replace with your domain name

 

        # Kafka Broker Service Names

        broker1_service_name = "com.amazonaws.vpc.us-east-1.vpce-svc-aaaaaaaaaaaa"  # Replace with actual service name

        broker2_service_name = "com.amazonaws.vpc.us-east-1.vpce-svc-bbbbbb"  # Replace with actual service name

        broker3_service_name = "com.amazonaws.vpc.us-east-1.vpce-svc-ccccccc"  # Replace with actual service name

        dc1_service_name = "com.amazonaws.vpc.us-east-1.vpce-svc-ddddddd"  # Replace with actual service name

 

        # Fetch existing VPC

        vpc = ec2.Vpc.from_lookup(self, "ExistingVPC", vpc_id=vpc_id)

 

        # VPC Endpoint for Broker 1

        broker1_endpoint = ec2.InterfaceVpcEndpoint(

            self, "Broker1Endpoint",

            vpc=vpc,

            service=ec2.InterfaceVpcEndpointService(broker1_service_name, 443),

            subnets=ec2.SubnetSelection(subnets=[ec2.Subnet.from_subnet_id(self, f"Subnet{i+1}", subnet_id) for i, subnet_id in enumerate(subnet_ids)]),

            private_dns_enabled=True

        )

 

        # VPC Endpoint for Broker 2

        broker2_endpoint = ec2.InterfaceVpcEndpoint(

            self, "Broker2Endpoint",

            vpc=vpc,

            service=ec2.InterfaceVpcEndpointService(broker2_service_name, 443),

            subnets=ec2.SubnetSelection(subnets=[ec2.Subnet.from_subnet_id(self, f"Subnet{i+1}", subnet_id) for i, subnet_id in enumerate(subnet_ids)]),

            private_dns_enabled=True

        )

 

        # VPC Endpoint for Broker 3

        broker3_endpoint = ec2.InterfaceVpcEndpoint(

            self, "Broker3Endpoint",

            vpc=vpc,

            service=ec2.InterfaceVpcEndpointService(broker3_service_name, 443),

            subnets=ec2.SubnetSelection(subnets=[ec2.Subnet.from_subnet_id(self, f"Subnet{i+1}", subnet_id) for i, subnet_id in enumerate(subnet_ids)]),

            private_dns_enabled=True

        )

 

        # VPC Endpoint for DC1

        dc1_endpoint = ec2.InterfaceVpcEndpoint(

            self, "DC1Endpoint",

            vpc=vpc,

            service=ec2.InterfaceVpcEndpointService(dc1_service_name, 443),

            subnets=ec2.SubnetSelection(subnets=[ec2.Subnet.from_subnet_id(self, f"Subnet{i+1}", subnet_id) for i, subnet_id in enumerate(subnet_ids)]),

            private_dns_enabled=True

        )

 

        # Route 53 Private Hosted Zone

        private_hosted_zone = route53.PrivateHostedZone(

            self, "PrivateHostedZone",

            zone_name=domain_name,

            vpc=vpc

        )

 

        # CNAME Records for each broker

        route53.CnameRecord(

            self, "Broker1Cname",

            zone=private_hosted_zone,

            record_name=f"broker1.{domain_name}",

            domain_name=broker1_endpoint.vpc_endpoint_dns_entries[0]

        )

 

        route53.CnameRecord(

            self, "Broker2Cname",

            zone=private_hosted_zone,

            record_name=f"broker2.{domain_name}",

            domain_name=broker2_endpoint.vpc_endpoint_dns_entries[0]

        )

 

        route53.CnameRecord(

            self, "Broker3Cname",

            zone=private_hosted_zone,

            record_name=f"broker3.{domain_name}",

            domain_name=broker3_endpoint.vpc_endpoint_dns_entries[0]

        )

 

        route53.CnameRecord(

            self, "DC1Cname",

            zone=private_hosted_zone,

            record_name=f"dc1.{domain_name}",

            domain_name=dc1_endpoint.vpc_endpoint_dns_entries[0]

        )

 

Steps to Deploy

1. Save the code in lib/my_kafka_stack.py.
2. In app.py, add the following to import and instantiate the stack:

 

from aws_cdk import App

from my_kafka_stack import MyKafkaStack

 

app = App()

MyKafkaStack(app, "MyKafkaStack")

app.synth()

 

3.      Deploy the stack:
cdk deploy