Bastion Host Module - Artos Documentation

Overview

The Bastion Host Module provides secure, SSH-key-free access to Amazon EKS clusters deployed in private subnets. By leveraging AWS Systems Manager (SSM) Session Manager, this module eliminates the need for traditional bastion hosts with SSH access, providing a more secure and auditable solution for cluster administration.

Key Features

SSH-Free Access: Connect to your EKS cluster without managing SSH keys or opening inbound ports
SSM Session Manager: Use AWS Systems Manager for secure, logged, and auditable access
Private Subnet Deployment: Bastion host runs in private subnets with no direct internet access
Pre-configured Tools: Automatically installs kubectl, AWS CLI, and other essential tools
EKS Integration: Full permissions for cluster management using eksctl and kubectl
Encrypted Storage: EBS volumes encrypted at rest for enhanced security

Use Cases

EKS Cluster Administration: Manage Kubernetes workloads and configurations
Troubleshooting: Debug applications running in private EKS clusters
Database Management: Access RDS instances in private subnets
Secure Operations: Perform operational tasks without exposing infrastructure to the internet

Architecture Components

Core Resources

1. EC2 Instance (Bastion Host)

The bastion host is a lightweight EC2 instance (default: t3.micro) running Amazon Linux 2023. It serves as a secure jump host for accessing resources in private subnets. Key Characteristics:

Deployed in private subnet (no public IP)
Encrypted EBS root volume (GP3)
Pre-installed with kubectl, AWS CLI, and git
Configured for SSM Session Manager access
Automatic tool installation via user data script

Default Configuration:

instance_type = "t3.micro"
volume_size   = 20  # GB
volume_type   = "gp3"
encrypted     = true

2. Security Group

The security group controls network traffic to and from the bastion host, following the principle of least privilege. Egress Rules (Outbound Traffic):

Port	Protocol	Destination	Purpose
443	TCP	VPC CIDR	HTTPS to EKS API endpoint
443	TCP	0.0.0.0/0	AWS services (IAM, STS, CloudFormation)
443	TCP	S3 Prefix List	Package repositories via VPC endpoint
443	TCP	ALB Security Group	Internal application access
5432	TCP	VPC CIDR	PostgreSQL/RDS database access
53	UDP	VPC CIDR	DNS resolution
123	UDP	169.254.169.123/32	NTP time synchronization

Security Features:

No inbound rules (access via SSM only)
Restricted egress to specific AWS services
VPC endpoint integration for S3 and other services
Optional GitHub access for development environments

Production Security: The default configuration includes broad S3 access for development convenience. For production deployments, restrict S3 access to specific buckets containing required binaries and tools.

3. IAM Role and Policies

The IAM role provides the bastion host with necessary permissions while maintaining least-privilege access. Attached Managed Policies:

AmazonSSMManagedInstanceCore - Enables SSM Session Manager access

Custom IAM Policy Permissions: EKS Access:

Cluster read operations (DescribeCluster, ListClusters)
Kubernetes API access (AccessKubernetesApi)
Node group and addon management
eksctl operations support

IAM Management (for eksctl):

Create/delete IAM roles and policies
Manage OIDC providers for service accounts
Attach/detach role policies
Tag management

CloudFormation:

Stack creation and management (required by eksctl)
Stack resource inspection
Template operations

Secrets Manager:

Read secrets from namespaced paths
Support for application configuration retrieval

S3 Access:

Download kubectl binary from specified S3 bucket
Optional: Development access to all S3 buckets

KMS (Optional):

Decrypt S3 objects and Secrets Manager values
Conditional based on KMS key ARN configuration

IAM Permissions: The bastion role includes broad IAM permissions required for eksctl to create service accounts and OIDC providers. Consider creating a separate, more restricted role for day-to-day operations if eksctl functionality is not needed.

4. SSM Session Manager Integration

AWS Systems Manager Session Manager provides secure, auditable access to the bastion host without requiring SSH keys or open inbound ports. Benefits:

No SSH Keys: Eliminates key management overhead and security risks
Audit Logging: All session activity logged to CloudWatch Logs
IAM-Based Access: Uses AWS IAM for authentication and authorization
No Inbound Ports: No need to open port 22 or manage security groups for SSH
Session Recording: Optional session recording for compliance requirements

Connection Method:

# Connect to bastion host
aws ssm start-session --target <instance-id> --region <aws-region>

# Port forwarding (e.g., for RDS access)
aws ssm start-session --target <instance-id> \
    --document-name AWS-StartPortForwardingSession \
    --parameters "portNumber=5432,localPortNumber=5432"

The script scripts/connect-bastion.sh can also be used to connect to the bastion server. This is the preferred method.

5. User Data Script

The user data script automatically configures the bastion host on first boot. Installation Tasks:

System updates via yum/dnf
SSM Agent verification and installation
kubectl binary installation from S3 bucket
AWS CLI v2 installation
Git installation (development environments)
Helper script creation for EKS configuration

Helper Script:

# Generated at /home/ec2-user/update-kubeconfig.sh
#!/bin/bash
aws eks update-kubeconfig --region <region> --name <cluster-name>
kubectl get nodes

Configuration Variables

Required Variables

# Network configuration
vpc_id      = "vpc-xxxxxxxx"      # VPC where bastion will be deployed
vpc_cidr    = "10.0.0.0/16"       # CIDR block of the VPC
subnet_id   = "subnet-xxxxxxxx"    # Private subnet ID for bastion

# EKS configuration
cluster_name = "artos-production"  # Name of your EKS cluster
aws_region   = "us-east-1"        # AWS region

# Resource naming
name_prefix = "artos"              # Prefix for all resource names

Optional Variables

# Instance configuration
instance_type = "t3.micro"         # EC2 instance type
ami_id        = null               # Custom AMI (default: latest AL2023)
volume_size   = 20                 # Root volume size in GB

# EKS access restriction
cluster_arn = "arn:aws:eks:..."    # Restrict EKS API access to specific cluster

# kubectl installation
kubectl_s3_bucket = "my-tools-bucket"  # S3 bucket with kubectl binary
kubectl_s3_key    = "kubectl"          # S3 object key for kubectl

# Encryption
kms_key_arn = "arn:aws:kms:..."    # KMS key for S3/Secrets Manager decryption

# Application access
alb_security_group_id = "sg-xxx"   # ALB security group for app access

# Development options (not recommended for production)
github_allowed_cidrs       = []    # GitHub CIDR ranges for repo access
github_repo_url            = null  # GitHub repository to clone
github_token_ssm_param_name = null # SSM parameter with GitHub PAT

# Resource tagging
tags = {
  Environment = "production"
  Application = "artos"
  ManagedBy   = "terraform"
}

Module Usage

Basic Deployment

module "bastion" {
  source = "./modules/bastion"

  # Network configuration
  vpc_id    = module.vpc.vpc_id
  vpc_cidr  = module.vpc.vpc_cidr_block
  subnet_id = module.vpc.private_subnets[0]

  # EKS configuration
  cluster_name = module.eks.cluster_name
  cluster_arn  = module.eks.cluster_arn
  aws_region   = var.aws_region

  # Naming
  name_prefix = var.name_prefix

  # Tags
  tags = local.common_tags
}

Production Deployment with kubectl from S3

module "bastion" {
  source = "./modules/bastion"

  # Network configuration
  vpc_id    = module.vpc.vpc_id
  vpc_cidr  = module.vpc.vpc_cidr_block
  subnet_id = module.vpc.private_subnets[0]

  # EKS configuration
  cluster_name = module.eks.cluster_name
  cluster_arn  = module.eks.cluster_arn
  aws_region   = var.aws_region

  # kubectl binary from S3 (required for private subnet)
  kubectl_s3_bucket = "artos-tools-us-east-1"
  kubectl_s3_key    = "binaries/kubectl-v1.28.0"

  # KMS encryption for S3 access
  kms_key_arn = module.kms.key_arn

  # ALB access for application debugging
  alb_security_group_id = module.alb.security_group_id

  # Instance configuration
  instance_type = "t3.small"
  volume_size   = 30

  # Naming
  name_prefix = var.name_prefix

  # Tags
  tags = merge(local.common_tags, {
    Component = "bastion"
    Backup    = "true"
  })
}

Development Environment

module "bastion_dev" {
  source = "./modules/bastion"

  # Network configuration
  vpc_id    = module.vpc.vpc_id
  vpc_cidr  = module.vpc.vpc_cidr_block
  subnet_id = module.vpc.private_subnets[0]

  # EKS configuration
  cluster_name = module.eks.cluster_name
  aws_region   = var.aws_region

  # kubectl binary
  kubectl_s3_bucket = "artos-dev-tools"
  kubectl_s3_key    = "kubectl"

  # GitHub repository access (dev-only)
  github_repo_url            = "https://github.com/yourorg/artos-config.git"
  github_token_ssm_param_name = "/dev/github/pat"
  github_allowed_cidrs       = [
    "140.82.112.0/20",  # GitHub IP ranges
    "143.55.64.0/20"
  ]

  # Naming
  name_prefix = "artos-dev"

  # Tags
  tags = {
    Environment = "development"
    Purpose     = "eks-access"
  }
}

Module Outputs

The module provides the following outputs for integration with other resources:

output "bastion_instance_id" {
  description = "Instance ID of the bastion host"
  value       = module.bastion.bastion_instance_id
}

output "bastion_security_group_id" {
  description = "Security group ID of the bastion host"
  value       = module.bastion.bastion_security_group_id
}

output "bastion_role_arn" {
  description = "IAM role ARN of the bastion host"
  value       = module.bastion.bastion_role_arn
}

output "connection_command" {
  description = "SSM Session Manager command to connect to bastion"
  value       = module.bastion.connection_command
}

Connecting to the Bastion Host

Prerequisites

AWS CLI: Install and configure AWS CLI v2
Session Manager Plugin: Install the Session Manager plugin

# Install Session Manager plugin (macOS)
brew install --cask session-manager-plugin

# Install Session Manager plugin (Linux)
curl "https://s3.amazonaws.com/session-manager-downloads/plugin/latest/ubuntu_64bit/session-manager-plugin.deb" -o "session-manager-plugin.deb"
sudo dpkg -i session-manager-plugin.deb

# Verify installation
session-manager-plugin --version

IAM Permissions: Ensure your IAM user/role has ssm:StartSession permission

Connect via SSM

# Basic connection
aws ssm start-session \
    --target i-1234567890abcdef0 \
    --region us-east-1

# Once connected, configure kubectl
./update-kubeconfig.sh

# Verify cluster access
kubectl get nodes
kubectl get pods -A

Port Forwarding to RDS

# Forward local port to RDS through bastion
aws ssm start-session \
    --target i-1234567890abcdef0 \
    --document-name AWS-StartPortForwardingSessionToRemoteHost \
    --parameters '{
      "host":["artos-db.cluster-xxx.us-east-1.rds.amazonaws.com"],
      "portNumber":["5432"],
      "localPortNumber":["5432"]
    }'

# Connect to database from local machine
psql -h localhost -p 5432 -U artos_admin -d artos

Security Best Practices

1. Network Isolation

Private Subnet Deployment: Always deploy bastion hosts in private subnets
VPC Endpoints: Use VPC endpoints for AWS services to avoid NAT gateway traffic
Security Group Restrictions: Limit egress to only required destinations

2. IAM Access Control

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ssm:StartSession"
      ],
      "Resource": [
        "arn:aws:ec2:*:*:instance/*"
      ],
      "Condition": {
        "StringEquals": {
          "ssm:resourceTag/Environment": "production",
          "ssm:resourceTag/Type": "Bastion"
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "ssm:TerminateSession",
        "ssm:ResumeSession"
      ],
      "Resource": [
        "arn:aws:ssm:*:*:session/${aws:username}-*"
      ]
    }
  ]
}

3. Session Logging and Monitoring

Enable CloudWatch Logs for session activity:

{
  "schemaVersion": "1.0",
  "description": "Session Manager Logging Configuration",
  "sessionType": "Standard_Stream",
  "inputs": {
    "s3BucketName": "artos-session-logs",
    "s3KeyPrefix": "bastion-sessions/",
    "s3EncryptionEnabled": true,
    "cloudWatchLogGroupName": "/aws/ssm/bastion-sessions",
    "cloudWatchEncryptionEnabled": true,
    "cloudWatchStreamingEnabled": true
  }
}

4. Automated Compliance Checks

Monitor bastion host compliance:

# Check for unauthorized inbound rules
aws ec2 describe-security-groups \
    --group-ids <bastion-sg-id> \
    --query 'SecurityGroups[0].IpPermissions' \
    | jq 'length == 0'  # Should return true

# Verify SSM Agent status
aws ssm describe-instance-information \
    --filters "Key=InstanceIds,Values=<instance-id>" \
    --query 'InstanceInformationList[0].PingStatus'

# Check for required tags
aws ec2 describe-instances \
    --instance-ids <instance-id> \
    --query 'Reservations[0].Instances[0].Tags'

Troubleshooting

Common Issues

1. Unable to Connect via SSM

Symptoms: TargetNotConnected error when starting session Solutions:

# Verify SSM Agent is running
aws ssm describe-instance-information \
    --filters "Key=InstanceIds,Values=<instance-id>"

# Check IAM instance profile
aws ec2 describe-instances \
    --instance-ids <instance-id> \
    --query 'Reservations[0].Instances[0].IamInstanceProfile'

# Verify security group allows HTTPS to Systems Manager endpoint
aws ec2 describe-security-groups \
    --group-ids <bastion-sg-id> \
    --query 'SecurityGroups[0].IpPermissionsEgress'

# Check VPC endpoints
aws ec2 describe-vpc-endpoints \
    --filters "Name=vpc-id,Values=<vpc-id>" \
    --query 'VpcEndpoints[?ServiceName==`com.amazonaws.<region>.ssm`]'

2. kubectl Not Installed

Symptoms: kubectl: command not found Solutions:

# Manual installation via S3
aws s3 cp s3://artos-tools-bucket/kubectl /tmp/kubectl
sudo chmod +x /tmp/kubectl
sudo mv /tmp/kubectl /usr/local/bin/kubectl

# Verify installation
kubectl version --client

# Update kubeconfig
aws eks update-kubeconfig --region us-east-1 --name artos-production

3. EKS API Access Denied

Symptoms: error: You must be logged in to the server (Unauthorized) Solutions:

# Verify IAM role has EKS access
aws eks describe-cluster --name artos-production --region us-east-1

# Check aws-auth ConfigMap in EKS cluster
kubectl get configmap aws-auth -n kube-system -o yaml

# Add bastion role to aws-auth ConfigMap
kubectl edit configmap aws-auth -n kube-system

# Add the following under mapRoles:
# - rolearn: arn:aws:iam::ACCOUNT:role/artos-bastion-role
#   username: bastion-user
#   groups:
#     - system:masters

4. Cannot Access RDS

Symptoms: Connection timeout to database Solutions:

# Verify security group allows bastion to RDS
aws ec2 describe-security-groups \
    --group-ids <rds-sg-id> \
    --query 'SecurityGroups[0].IpPermissions'

# Test connection from bastion
telnet artos-db.cluster-xxx.us-east-1.rds.amazonaws.com 5432

# Check DNS resolution
nslookup artos-db.cluster-xxx.us-east-1.rds.amazonaws.com

# Verify RDS is in same VPC
aws rds describe-db-instances \
    --db-instance-identifier artos-production \
    --query 'DBInstances[0].DBSubnetGroup.VpcId'

Operational Tasks

Installing Additional Tools

# Connect to bastion
aws ssm start-session --target <instance-id> --region us-east-1

# Install eksctl
curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
sudo mv /tmp/eksctl /usr/local/bin
eksctl version

# Install Helm
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
helm version

# Install k9s for cluster monitoring
curl -sS https://webinstall.dev/k9s | bash
k9s version

Creating EKS Service Accounts

# Create IAM service account for workload
eksctl create iamserviceaccount \
    --cluster=artos-production \
    --namespace=artos-system \
    --name=artos-worker-sa \
    --role-name=ArtosWorkerRole \
    --attach-policy-arn=arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess \
    --approve

# Verify service account creation
kubectl get serviceaccount artos-worker-sa -n artos-system -o yaml

EKS Module - Amazon Elastic Kubernetes Service configuration
VPC Module - Virtual Private Cloud and networking setup
RDS Module - Relational Database Service configuration
IAM Module - Identity and Access Management roles and policies
Monitoring Module - CloudWatch monitoring and alerting

Module Maintenance: This module is actively maintained and tested with Terraform 1.0+ and AWS Provider 5.x. For questions or issues, contact the infrastructure team or refer to the internal Terraform documentation.

​Overview

​Key Features

​Use Cases

​Architecture Components

​Core Resources

​1. EC2 Instance (Bastion Host)

​2. Security Group

​3. IAM Role and Policies

​4. SSM Session Manager Integration

​5. User Data Script

​Configuration Variables

​Required Variables

​Optional Variables

​Module Usage

​Basic Deployment

​Production Deployment with kubectl from S3

​Development Environment

​Module Outputs

​Connecting to the Bastion Host

​Prerequisites

​Connect via SSM

​Port Forwarding to RDS

​Security Best Practices

​1. Network Isolation

​2. IAM Access Control

​3. Session Logging and Monitoring

​4. Automated Compliance Checks

​Troubleshooting

​Common Issues

​1. Unable to Connect via SSM

​2. kubectl Not Installed

​3. EKS API Access Denied

​4. Cannot Access RDS

​Operational Tasks

​Installing Additional Tools

​Creating EKS Service Accounts

​Related Modules

Overview

Key Features

Use Cases

Architecture Components

Core Resources

1. EC2 Instance (Bastion Host)

2. Security Group

3. IAM Role and Policies

4. SSM Session Manager Integration

5. User Data Script

Configuration Variables

Required Variables

Optional Variables

Module Usage

Basic Deployment

Production Deployment with kubectl from S3

Development Environment

Module Outputs

Connecting to the Bastion Host

Prerequisites

Connect via SSM

Port Forwarding to RDS

Security Best Practices

1. Network Isolation

2. IAM Access Control

3. Session Logging and Monitoring

4. Automated Compliance Checks

Troubleshooting

Common Issues

1. Unable to Connect via SSM

2. kubectl Not Installed

3. EKS API Access Denied

4. Cannot Access RDS

Operational Tasks

Installing Additional Tools

Creating EKS Service Accounts

Related Modules