Amazon EKS provides a robust, managed Kubernetes experience, but a critical piece of the puzzle remains your responsibility: disaster recovery and data protection. A running cluster is not a backup. Configuration errors, accidental deletions, or even region-wide outages can happen. Without a solid backup strategy, you risk significant downtime and data loss.
This blog provides an introductory walkthrough for implementing a backup and restore solution for your EKS clusters using two powerful open-source tools: Velero for orchestrating backups and Terraform for automating the entire infrastructure setup.
We will cover:
- Why backing up EKS is non-negotiable.
- An introduction to Velero.
- A step-by-step guide to deploying Velero on EKS using Terraform, including creating the S3 bucket and securing access with EKS Pod Identity.
- How to create on-demand and scheduled backups.
- How to validate backups once they’re created.
- How to restore a backup to an existing cluster or a completely new one.
Why You Need to Backup EKS Clusters
Before we dive into the “how,” let’s establish the “why.” You need a backup strategy for your EKS cluster for several critical reasons:
- Disaster Recovery: If your cluster suffers a catastrophic failure or an entire AWS region goes down, a backup is your only path to recovery.
- Data Protection for Stateful Apps: If you run databases, message queues, or other stateful services on your EKS cluster, you must back up their Persistent Volumes (PVs). Velero integrates with AWS to create EBS snapshots of your PVs, ensuring your application data is safe.
- Cluster Migrations & Upgrades: Backups are the safest way to migrate applications from one EKS cluster to another or to create a clone of your production environment for testing a major Kubernetes upgrade.
- Protection Against Human Error: The most common cause of outages is human error. A kubectl delete command issued against the wrong namespace can be devastating. Velero allows you to quickly roll back to a known good state.
What is Velero?
Velero is an open-source tool designed to safely back up and restore Kubernetes cluster resources and persistent volumes. It works by:
- Backing up Cluster Objects: It queries the Kubernetes API server and saves the YAML definitions of your resources (Deployments, Services, ConfigMaps, etc.) into a compressed tarball, which it then stores in an object storage bucket, like Amazon S3.
- Snapshotting Persistent Volumes: For Persistent Volumes backed by a supported provider, Velero calls the provider’s API to create a snapshot. On AWS, this means creating an EBS snapshot of the volume attached to your pod.
This two-pronged approach ensures you can fully reconstruct both your application’s state and its underlying data.
Deploying Velero with Terraform
Now for the core of our setup. We will use Terraform to provision all the necessary AWS resources and then deploy Velero using the official Helm chart.
Prerequisites:
- Terraform CLI installed.
- kubectl installed and configured to access your EKS cluster.
- Helm CLI installed (for understanding, though Terraform will manage the release).
- An existing EKS cluster.
- The EKS Pod Identity Agent add-on installed and active on your cluster. This is the modern, recommended way to handle IAM permissions for EKS workloads.
NOTE: For a brief overview of EKS Pod Identity including deployment directions refer to the blog post Leveraging EKS Pod Identity on AWS.
Securing Access with EKS Pod Identity
Velero can and should be configured using EKS Pod Identity. As the successor to the older IAM Roles for Service Accounts (IRSA), it is the current AWS best practice for granting pods permissions to AWS resources.
With EKS Pod Identity, we create a dedicated IAM role that Velero’s pod can assume. This is far more secure than creating and managing static IAM user access keys. The EKS Pod Identity agent on your cluster nodes automatically handles vending temporary credentials to the pod, adhering to the principle of least privilege.
Our Terraform code will create:
- An S3 bucket to store the backup files.
- An IAM policy with the exact permissions Velero needs.
- An IAM role with the correct trust policy for EKS Pod Identity.
- An EKS Pod Identity Association to link the role to the Velero service account.
Create a file named velero.tf and add the following code.
# Configure the AWS Provider
provider "aws" {
region = "us-east-1" # Change to your EKS cluster's region
}
# Configure the Helm Provider
provider "helm" {
kubernetes = {
config_path = "~/.kube/config" # Or your specific kubeconfig path
}
}
# Assumes you have an existing EKS cluster with the Pod Identity Agent add-on enabled.
data "aws_eks_cluster" "cluster" {
name = "my-eks-cluster" # <-- IMPORTANT: Change to your cluster name
}
# A unique name for our backup bucket
resource "random_pet" "bucket_name" {
length = 2
}
# 1. S3 bucket for Velero backups
resource "aws_s3_bucket" "velero_backups" {
bucket = "${data.aws_eks_cluster.cluster.name}-velero-backups-${random_pet.bucket_name.id}"
tags = {
Name = "Velero Backups for ${data.aws_eks_cluster.cluster.name}"
}
}
# Block all public access to the backup bucket
resource "aws_s3_bucket_public_access_block" "velero_backups" {
bucket = aws_s3_bucket.velero_backups.id
block_public_acls = true
block_public_policy = true
ignore_public_acls = true
restrict_public_buckets = true
}
# 2. The IAM Policy: Permissions for Velero
resource "aws_iam_policy" "velero_policy" {
name = "${data.aws_eks_cluster.cluster.name}-VeleroPolicy"
description = "IAM policy for Velero to access S3 and EC2 snapshots"
# The policy document grants permissions for S3 operations on the backup bucket
# and EC2 operations for creating and managing EBS snapshots.
policy = jsonencode({
"Version" : "2012-10-17",
"Statement" : [
{
"Effect" : "Allow",
"Action" : [
"ec2:DescribeVolumes",
"ec2:DescribeSnapshots",
"ec2:CreateTags",
"ec2:CreateVolume",
"ec2:CreateSnapshot",
"ec2:DeleteSnapshot"
],
"Resource" : "*"
},
{
"Effect" : "Allow",
"Action" : [
"s3:GetObject",
"s3:DeleteObject",
"s3:PutObject",
"s3:AbortMultipartUpload",
"s3:ListMultipartUploadParts"
],
"Resource" : [
"${aws_s3_bucket.velero_backups.arn}/*"
]
},
{
"Effect" : "Allow",
"Action" : [
"s3:ListBucket"
],
"Resource" : [
aws_s3_bucket.velero_backups.arn
]
}
]
})
}
# 3. The IAM Role for Velero (for EKS Pod Identity)
resource "aws_iam_role" "velero_role" {
name = "${data.aws_eks_cluster.cluster.name}-VeleroRole"
# The fix is in this trust policy
assume_role_policy = jsonencode({
"Version" : "2012-10-17",
"Statement" : [
{
"Effect" : "Allow",
"Principal" : {
"Service" : "pods.eks.amazonaws.com"
},
"Action" : [
"sts:AssumeRole",
"sts:TagSession"
]
}
]
})
}
# Attach the policy to the role
resource "aws_iam_role_policy_attachment" "velero_attach" {
role = aws_iam_role.velero_role.name
policy_arn = aws_iam_policy.velero_policy.arn
}
# 4. Create the Pod Identity Association
# This resource links the IAM role to the Velero service account.
resource "aws_eks_pod_identity_association" "velero" {
cluster_name = data.aws_eks_cluster.cluster.name
namespace = "velero"
service_account = "velero" # This is the default service account name created by the Helm chart
role_arn = aws_iam_role.velero_role.arn
depends_on = [
aws_iam_role_policy_attachment.velero_attach
]
}
# 5. The Deployment: Velero Helm Chart
resource "helm_release" "velero" {
name = "velero"
repository = "https://vmware-tanzu.github.io/helm-charts"
chart = "velero"
namespace = "velero"
create_namespace = true
version = "10.0.4" # Use a specific, recent version
values = [
yamlencode({
# For EKS Pod Identity, we don't need annotations.
# The aws_eks_pod_identity_association resource handles linking the role.
serviceAccount = {
server = {
create = true,
name = "velero"
}
}
# Tell Velero we are not providing a static secret
credentials = {
useSecret = false
}
# Configuration for the backup storage location
configuration = {
backupStorageLocation = [
{
name = "default",
provider = "aws"
bucket = aws_s3_bucket.velero_backups.bucket,
config = {
region = "us-east-1" # <-- Change to your EKS cluster's region
}
}
],
# volumeSnapshotLocation must be an array
volumeSnapshotLocation = [
{
name = "default",
provider = "aws",
config = {
region = "us-east-1" # <-- Change to your EKS cluster's region
}
}
]
}
# Install the Velero AWS plugin and enable the node-agent for volume snapshots
initContainers = [
{
name = "velero-plugin-for-aws",
image = "velero/velero-plugin-for-aws:v1.12.0",
volumeMounts = [
{
mountPath = "/target",
name = "plugins"
}
]
}
]
# Enable snapshots by default
snapshotsEnabled = true
})
]
# Velero depends on the pod identity association being created
depends_on = [
aws_eks_pod_identity_association.velero
]
}
Now, run Terraform to apply this configuration:
Initialize Terraform
terraform init
Apply the configuration
terraform apply --auto-approve
After a few minutes, Terraform will provision the S3 bucket, IAM resources, associate the identity, and deploy Velero to your cluster. You can verify the installation by checking for pods in the velero namespace:
kubectl get pods -n velero NAME READY STATUS RESTARTS AGE velero-785d4596b6-abcde 1/1 Running 0 2m
How Backups Are Created
Now that Velero is installed, you need to install the Velero CLI to interact with it. Follow the official instructions to install it. You can create backups that are scoped to a namespace or that cover the entire cluster.
Backing Up a Specific Namespace
This is useful for backing up a single application or a team’s resources.
Create a backup of only the ‘my-app’ namespace
velero backup create my-app-backup --include-namespaces my-app
Check the status of the backup
velero backup describe my-app-backup
Backing Up the Entire Cluster
For a full disaster recovery backup, you can back up all resources. It’s a best practice to exclude namespaces that contain transient or cluster-specific operational data that you wouldn’t want to restore, like kube-system or Velero’s own namespace.
Create a backup of the entire cluster, excluding some system namespaces
velero backup create full-cluster-backup --exclude-namespaces kube-system,velero
Check the status
velero backup describe full-cluster-backup --details
Creating a Scheduled Backup
For production, you’ll want scheduled backups. This command creates a backup of the production-db and monitoring namespaces every day at 3 AM.
velero schedule create daily-prod-backup --schedule="0 3 * * *" --include-namespaces production-db,monitoring --ttl 336h0m0s # Set a Time-To-Live (TTL) of 14 days (336 hours)
Similarly, you can schedule a daily full-cluster backup for disaster recovery purposes.
Schedule a full cluster backup to run every night at 1 AM
velero schedule create daily-full-cluster-backup --schedule="0 1 * * *" --exclude-namespaces kube-system,velero --ttl 720h0m0s # Set a Time-To-Live (TTL) of 30 days
The backup process will start. Velero will save the object definitions to S3 and trigger EBS snapshots for any PVs in the selected scope.
Managing Schedules with Infrastructure as Code
While using the Velero CLI is great for ad-hoc tasks, in a production environment it is better to manage your backup schedules declaratively, just like any other piece of your infrastructure. This aligns with GitOps principles, where your Git repository is the single source of truth.
When Velero is installed, it adds Custom Resource Definitions (CRDs) to your cluster, including one for a Schedule. This allows you to define a backup schedule in a YAML or JSON manifest and apply it with kubectl or, even better, manage it with Terraform.
Why use IaC for Schedules?
- Declarative State: Your backup strategy is defined in code. If someone manually deletes a schedule from the cluster, your automation (e.g., a terraform apply) will detect the drift and recreate it, enforcing the desired state.
- Version Control & Auditing: Every change to your backup schedule (frequency, scope, TTL) is captured in your Git history. This is crucial for auditing and understanding how your backup strategy has evolved.
- Consistency: It ensures your backup schedules are managed with the same rigor and process (e.g., pull requests) as your application deployments.
Here is how you can define the same daily full-cluster backup schedule using the kubernetes_manifest resource in Terraform. Add this to your velero.tf file:
resource "kubernetes_manifest" "daily_full_cluster_schedule" {
# This depends on the Helm release to ensure the Velero CRDs exist before we try to create a schedule.
depends_on = [helm_release.velero]
manifest = {
"apiVersion" = "velero.io/v1"
"kind" = "Schedule"
"metadata" = {
"name" = "daily-full-cluster-from-tf"
"namespace" = "velero"
}
"spec" = {
"schedule" = "0 1 * * *" # Daily at 1 AM
"template" = {
"includedNamespaces" = ["*"]
"excludedNamespaces" = ["kube-system", "velero"]
"storageLocation" = "default"
"ttl" = "720h0m0s" # 30 days
}
}
}
}
How to Validate Your Velero Backups
A backup strategy is incomplete without validation. You must regularly verify that your backups are viable and can be successfully restored. Here is a step-by-step process for inspecting your backups.
Step 1: Check the Backup Status
The first and simplest check is to see if Velero reports the backup as Completed without any errors.
List all backups and their status
velero backup get # Example Output: NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR full-cluster-backup Completed 0 0 2025-06-11T20:15:00Z 29d default <none> failing-backup Failed 1 3 2025-06-11T19:00:00Z 29d default <none>
Look for a STATUS of Completed and ERRORS equal to 0. A Failed status requires immediate investigation.
Step 2: Describe the Backup in Detail
This is the most important command for manual inspection. It shows you exactly what Velero captured. Use the --details flag to see a complete list of every resource included in the backup file.
# Describe a specific backup and list all backed-up resources velero backup describe full-cluster-backup --details
In the output, you will see sections for Persistent Volumes, Resources, and a count of items backed up. Skim this list to ensure the critical resources you expected to be backed up (like Deployments, StatefulSets, PVCs, ConfigMaps, etc.) are present.
Step 3: Check the Backup Logs
If a backup has errors or warnings, or if you just want to be extra thorough, check the logs for that specific backup process. This will show you the step-by-step actions Velero took and any issues it encountered.
# Get the logs for a specific backup velero backup logs full-cluster-backup
Look for any lines with level=error or level=warning. An error might indicate that Velero couldn’t access a resource or that a volume snapshot failed.
While these inspection steps are crucial, the only way to be 100% certain a backup is valid is to perform a restore. The following sections will discuss in detail how to restore your backups to validate them and recover from disaster.
Restore Overview
Creating backups is only half the process; the true value lies in being able to restore them effectively. Velero provides a flexible and powerful set of restore capabilities to handle a variety of scenarios, from recovering a single deleted resource to migrating an entire cluster. The following sections provide detailed, hands-on examples for these common use cases:
- Restoring to Your Existing Cluster: This covers the most frequent use case—recovering from accidental deletions or configuration errors within the same EKS cluster. We’ll explore granular restores of specific resources, full namespaces, and even the entire cluster.
- Restoring to a New Cluster: This is your guide for disaster recovery and cluster migrations. We’ll walk through the process of taking a backup from one cluster and restoring it to a completely new one, including the necessary setup.
- Advanced Restores: This section details how to use Velero’s remapping features to restore resources into different namespaces, a key technique for cloning production environments for testing and development.
Restoring to Your Existing Cluster
This is your primary disaster recovery scenario. Imagine a developer accidentally deleted resources. Velero provides granular restore capabilities.
Scenario 1: Restore an Entire Cluster
This is the “big red button” for a major incident where multiple namespaces were affected.
# Create a restore from the full-cluster-backup velero restore create --from-backup full-cluster-backup # Check the status of the restore velero restore describe <RESTORE_NAME> # The name is generated automatically, find it with `velero restore get`
Scenario 2: Restore an Entire Namespace
This is the most common use case, recovering from an accidental kubectl delete namespace my-app.
# Let's simulate the disaster kubectl delete namespace my-app # Create a restore from the specific backup for that app velero restore create --from-backup my-app-backup --include-namespaces my-app # Wait for the restore to complete and verify velero restore get kubectl get pods -n my-app # Your pods should be coming back online!
Scenario 3: Restore a Specific Resource Type in a Namespace
This is useful for a more surgical fix, like restoring only the Deployments and ConfigMaps in the my-app namespace without touching other resources.
# Let's say we only deleted a specific deployment kubectl delete deployment my-app-deployment -n my-app # Restore ONLY deployments and configmaps within the my-app namespace from the backup velero restore create --from-backup my-app-backup --include-namespaces my-app --include-resources deployments,configmaps
Restoring to a New Cluster (Migration)
This process is perfect for migrating to a new EKS cluster or recovering in a different AWS region.
Prepare the New Cluster
On the new EKS cluster, you must first enable the EKS Pod Identity Agent add-on. Then, run the exact same Terraform setup (velero.tf) that you ran on the original cluster. This is critical. It ensures the new Velero instance:
- Points to the same S3 backup bucket.
- Has an IAM role and Pod Identity Association with the same permissions to read from that bucket and create volumes from EBS snapshots.
Check for Existing Backups
Once Velero is running on the new cluster, it will automatically sync with the S3 bucket. You should be able to see the backups created by the old cluster.
# On the NEW cluster velero backup get # NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR # my-app-backup Completed 0 0 2025-06-11T18:30:00Z 29d default <none> # full-cluster-backup Completed 0 0 2025-06-11T18:45:00Z 29d default <none>
Restore to the New Cluster
The restore command is identical. You’re simply running it on the new cluster.
# On the NEW cluster, restore the desired backup velero restore create --from-backup my-app-backup # Verify the resources are created in the new cluster kubectl get all -n my-app
Your application, along with its persistent data restored from EBS snapshots, will now be running on the new cluster.
Advanced Restores: Remapping and Modifying Resources
Velero’s capabilities extend beyond simple one-to-one restores. You can also manipulate resources as they are restored, which is incredibly useful for tasks like cloning an environment for testing or adapting to changes in a new cluster.
Restoring to a Different Namespace
A common requirement is to restore a production backup into a new namespace to create a staging or development environment. This allows you to test changes against a realistic dataset without affecting production. The --namespace-mappings flag makes this easy.
Restore resources from the ‘prod-ns’ namespace into the ‘staging-ns’ namespace
velero restore create prod-to-staging-clone --from-backup daily-prod-backup --namespace-mappings prod-ns:staging-ns
This command tells Velero to find all resources in the backup that were in the prod-ns namespace and create them in the staging-ns namespace instead.
Modifying Resources on Restore
For more complex transformations, such as changing a resource’s storage class, removing labels, or other in-depth modifications, Velero uses restore hooks. These are pods that run custom logic either before a resource is restored in order to modify it or after it is restored in order to perform post-restore actions.
Implementing restore hooks is an advanced topic that involves creating custom container images with your logic. For detailed examples and instructions, refer to the official Velero Restore Hooks documentation.
Conclusion
Protecting your Kubernetes workloads is not an optional extra; it is a fundamental operational responsibility. By combining the power of Velero with the declarative automation of Terraform, you can create a robust, repeatable, and secure backup and recovery system for your EKS clusters.
Following the steps in this guide, you have built a solution that leverages modern AWS best practices like EKS Pod Identity and provides the flexibility to handle everything from minor rollbacks to full-scale cluster migrations. With scheduled backups running, you gain the peace of mind that comes from knowing your application state and data are protected, allowing you to focus on building and innovating.
