C3 - Testing

7 minute read

How metarex can build an API to cost your running cloud

| C3 | testing |

The development testbed will consist of 2 different high availability Kubernetes cluster types, each running the same kubernetes pods on different node types. Identical infrastructure (otel, metarex, prometheus, grafana) will be used to extract, store and present the results.

  • Groups A cluster - Amazon EKS
    Using the native Amazon Kubernetes infrastructure to host the test environment - in theory, this should be the most controllable running on AWS
  • Group R cluster - Rancher K3s
    Designed to run with the same definitions as Group A - in theory this should be portable across platfrom so that we can measure a baseline infrastructure cost and isolate the executable code costs more accurately.
  • TBD - GKE - Google’s Kubernetes Engine for native costs
  • TBD - AKS - Microsoft Azure Kubernetes Service for native costs
  • TBD - ACK - Alibaba Cloud for Kubernetes native costs

I asked Claude (Anthropic Sonnet 4.0) to estimate the costs of the proposed comparison architecture

Quick Comparison Table

FeatureGroup A (EKS)Group R (k3s)
ManagementFully managed control planeSelf-managed
Monthly Cost~$248~$195
Setup Time~15 minutes~20 minutes
Control Plane HAAWS managed (3 AZs)Self-managed etcd cluster
Node ManagementManaged node groupsManual EC2 instances
NetworkingVPC CNIFlannel CNI
Load BalancingAWS LoadBalancer ServiceCaddy NodePort + NLB
StorageEBS CSI DriverLocal-path provisioner
MonitoringCloudWatch integrationPrometheus/Grafana
UpgradesManagedManual
ScalabilityAuto-scaling groupsManual scaling
ComplexityMediumLow

Architecture Comparison

Group A (EKS) Architecture

Internet → AWS ALB → EKS Managed Nodes → Pods
            ↓
        VPC CNI → ENI per Pod
            ↓
        EBS CSI → Persistent Volumes
            ↓
        CloudWatch → Monitoring

Group R (k3s) Architecture

Internet → NLB → NodePort (30080) → Caddy Pods → Apps
           ↓
       Flannel → VXLAN Overlay
           ↓
       Local Storage → Persistent Volumes
           ↓
       Prometheus → Monitoring

Detailed Feature Comparison

1. Control Plane Management

Group A (EKS):

  • AWS manages etcd, API server, scheduler, controller manager
  • Automatic backups and disaster recovery
  • Multi-AZ deployment by default
  • Managed security patches and updates
  • 99.95% SLA

Group R (k3s):

  • Self-managed embedded etcd cluster
  • Manual backup strategy required
  • HA through multiple controller nodes
  • Manual security updates needed
  • Depends on EC2 SLA (99.99%)

2. Networking

Group A (EKS):

1
2
3
4
5
6

# VPC CNI provides:
- Native AWS networking
- Pod-to-pod communication via VPC
- Security groups per pod
- Multiple ENIs per node
- Direct integration with AWS services

Group R (k3s):

1
2
3
4
5
6

# Flannel provides:
- VXLAN overlay network
- Simpler networking model
- Less AWS integration
- Lower resource overhead
- Standard Kubernetes networking

3. Storage Options

Group A (EKS):

  • EBS CSI driver for persistent volumes
  • EFS CSI driver for shared storage
  • FSx integration available
  • Automatic volume provisioning
  • Cross-AZ volume attachment

Group R (k3s):

  • Local-path provisioner (default)
  • Manual EBS integration possible
  • Simpler storage model
  • Node-local storage binding
  • Manual backup strategies

4. Load Balancing

Group A (EKS):

1
2
3
4
5
6
7
8

# AWS Load Balancer Controller
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
spec:
  type: LoadBalancer  # Creates AWS NLB automatically

Group R (k3s):

1
2
3
4
5
6
7

# Manual NLB + NodePort
apiVersion: v1
kind: Service
spec:
  type: NodePort
  ports:
  - nodePort: 30080  # Fixed port on all nodes

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117

c3/
├── 📁 group-a-eks/                     # EKS cluster deployment
│   ├── 📁 terraform/                   # Infrastructure as code
│   │   ├── main.tf                     # Main Terraform configuration
│   │   ├── variables.tf                # Input variables
│   │   ├── outputs.tf                  # Output values
│   │   ├── vpc.tf                      # VPC configuration
│   │   ├── eks.tf                      # EKS cluster configuration
│   │   └── terraform.tfvars.example    # Example variables
│   ├── 📁 k8s-manifests/              # Kubernetes manifests
│   │   ├── 📁 caddy/                  # Reverse proxy
│   │   ├── 📁 khello/                 # Hello application
│   │   ├── 📁 mariadb/                # Database
│   │   ├── 📁 pocketbase/             # Authentication service
│   │   ├── 📁 prometheus/             # Monitoring
│   │   └── 📁 grafana/                # Dashboards
│   ├── 📁 scripts/                    # Management scripts
│   │   ├── setup-eks.sh               # Cluster setup
│   │   ├── deploy-apps.sh             # Application deployment
│   │   ├── test-deployment.sh         # Testing
│   │   ├── install-pod.sh             # Pod installation
│   │   ├── backup-cluster.sh          # Backup operations
│   │   ├── performance-tuning.sh      # Optimization
│   │   └── troubleshoot-cluster.sh    # Diagnostics
│   ├── configure-environment.sh       # Environment configuration
│   └── README.md                      # EKS documentation
│
├── 📁 group-r-k3s/                     # k3s cluster deployment
│   ├── 📁 terraform/                   # Infrastructure as code
│   │   ├── main.tf                     # Main Terraform configuration
│   │   ├── variables.tf                # Input variables
│   │   ├── outputs.tf                  # Output values
│   │   ├── vpc.tf                      # VPC configuration
│   │   ├── instances.tf                # EC2 instances
│   │   ├── security-groups.tf          # Security groups
│   │   ├── 📁 user-data/              # Instance initialization
│   │   └── terraform.tfvars.example    # Example variables
│   ├── 📁 k8s-manifests/              # Kubernetes manifests
│   │   └── [same structure as EKS]     # Adapted for k3s
│   ├── 📁 scripts/                    # Management scripts
│   │   ├── setup-k3s.sh               # Cluster setup
│   │   ├── ssh-to-node.sh             # Node access
│   │   └── [same as EKS where applicable]
│   ├── configure-environment.sh       # Environment configuration
│   └── README.md                      # k3s documentation
│
├── 📁 gitops/                          # GitOps configuration
│   ├── argocd-install.sh              # ArgoCD installation
│   ├── app-of-apps.yaml               # Application management
│   └── 📁 applications/               # ArgoCD applications
│       ├── khello-app.yaml
│       ├── pocketbase.yaml
│       └── monitoring-stack.yaml
│
├── 📁 environments/                    # Multi-environment configs
│   ├── 📁 base/                       # Base configurations
│   │   └── kustomization.yaml
│   ├── 📁 development/                # Development environment
│   │   ├── kustomization.yaml
│   │   └── khello-dev.yaml
│   ├── 📁 staging/                    # Staging environment
│   │   ├── kustomization.yaml
│   │   └── khello-staging.yaml
│   └── 📁 production/                 # Production environment
│       ├── kustomization.yaml
│       ├── khello-production.yaml
│       └── monitoring-production.yaml
│
├── 📁 monitoring/                      # Advanced monitoring
│   ├── alertmanager-config.yaml       # Alert management
│   ├── prometheus-rules-advanced.yaml # Monitoring rules
│   └── grafana-dashboard-comparison.json
│
├── 📁 security-policies/              # Security configurations
│   ├── pod-security-policy.yaml       # Pod security
│   ├── network-policies.yaml          # Network security
│   └── policy-enforcement.yaml        # OPA Gatekeeper
│
├── 📁 compliance/                      # Compliance and audit
│   ├── audit-report.sh                # Audit automation
│   └── policy-enforcement.yaml        # Compliance policies
│
├── 📁 scripts/                        # Shared scripts
│   ├── backup-cluster.sh              # Universal backup
│   ├── restore-cluster.sh             # Universal restore
│   ├── security-scan.sh               # Security scanning
│   ├── performance-tuning.sh          # Performance optimization
│   ├── cost-optimizer.sh              # Cost optimization
│   ├── troubleshoot-cluster.sh        # Troubleshooting
│   └── schedule-backups.sh            # Backup scheduling
│
├── 📁 test-suite/                     # Comprehensive testing
│   ├── run-all-tests.sh              # Test automation
│   ├── load-test.yml                 # Load testing config
│   ├── curl-format.txt               # HTTP testing
│   └── test-performance.sh           # Performance testing
│
├── 📁 .github/workflows/             # CI/CD pipelines
│   ├── cluster-deployment.yml        # Main deployment pipeline
│   ├── gitops-deployment.yml         # GitOps pipeline
│   └── security-scan.yml             # Security automation
│
├── 📁 docs/                          # Documentation
│   ├── comparison-guide.md           # EKS vs k3s comparison
│   ├── deployment-guide.md           # Deployment instructions
│   ├── troubleshooting-guide.md      # Troubleshooting help
│   ├── security-hardening.md         # Security best practices
│   ├── cost-optimization.md          # Cost management
│   └── operational-runbooks.md       # Operation procedures
│
├── prerequisites-check.sh            # Prerequisites validation
├── deploy-comparison-environment.sh  # Main deployment script
├── manage-environments.sh            # Environment management
├── README.md                         # Main project documentation
├── CONTRIBUTING.md                   # Contribution guidelines
├── LICENSE                           # Project license
└── .gitignore                        # Git ignore rules