Skip to content
About Me
----

AWS Automation with Python Boto3 SDK: Enterprise Cloud Engineering

Architect Key Takeaways

Production guide to AWS cloud automation using Python Boto3, covering client vs resource architecture, pagination, exponential backoff retries, cross-account STS assume role, FinOps cost optimization, and automated security remediation.

Python is the universal language of Site Reliability Engineering, Cloud Platform Engineering, and DevSecOps. The AWS SDK for Python (Boto3) enables engineers to programmatically provision, audit, govern, and auto-remediate thousands of cloud resources across multi-account enterprise AWS Organizations.

AWS Boto3 Cloud Automation Architecture


Understanding how Boto3 models the AWS ecosystem is critical for writing high-performance, maintainable automation code.

Primitive Abstraction Level AWS Service Coverage Return Type Best For
boto3.Session Configuration container All services Session instance Managing credentials, multi-region routing, cross-account STS roles.
boto3.client Low-level 1:1 service API 100% (Every AWS API endpoint) Python dict (parsed JSON) Production automation, paginators, botocore configs, custom headers, high performance.
boto3.resource High-level Object-Oriented ~30% (Legacy/Partial) Resource OOP objects Quick scripts, simple CRUD operations where OOP syntax (instance.stop()) is convenient.

[!IMPORTANT] Production Standard: AWS no longer updates boto3.resource with new services or advanced API parameters. Always default to boto3.client in production SRE tools and serverless Lambda functions for maximum API coverage, speed, and paginator support.


2. Production Resilience: Pagination, Timeouts, and Backoff

Section titled β€œ2. Production Resilience: Pagination, Timeouts, and Backoff”

AWS APIs enforce strict rate limits (API Throttling with RequestLimitExceeded / ThrottlingException). Real-world scripts must implement resilient network configuration and pagination.

2.1 Botocore Adaptive Retries and Backoff Configuration

Section titled β€œ2.1 Botocore Adaptive Retries and Backoff Configuration”
import boto3
from botocore.config import Config
# Standard enterprise-resilient botocore configuration
resilient_config = Config(
region_name="us-east-1",
retries={
"max_attempts": 10,
"mode": "adaptive", # Adaptive rate-limiting automatically throttles requests based on client response
},
connect_timeout=5,
read_timeout=15,
max_pool_connections=50, # Optimal for multi-threaded workers
)
ec2_client = boto3.client("ec2", config=resilient_config)

Never write manual while 'NextToken' in response loops when scanning thousands of resources. Use Boto3’s built-in get_paginator:

import boto3
def audit_all_running_instances(client) -> list[str]:
"""Retrieves all running EC2 instance IDs across pages safely."""
paginator = client.get_paginator("describe_instances")
instance_ids = []
# Filter serverside to minimize egress payload
page_iterator = paginator.paginate(
Filters=[{"Name": "instance-state-name", "Values": ["running"]}],
PaginationConfig={"PageSize": 100}
)
for page in page_iterator:
for reservation in page.get("Reservations", []):
for instance in reservation.get("Instances", []):
instance_ids.append(instance["InstanceId"])
return instance_ids

In modern organizations, automation runs from a central β€œTooling” or β€œSecurity” account and assumes temporary roles in spoke accounts (e.g., prod, staging, sandbox).

import boto3
from botocore.exceptions import ClientError
def get_spoke_account_session(account_id: str, role_name: str = "OrganizationDevSecOpsExecutionRole") -> boto3.Session:
"""Assumes an IAM Role in a spoke account and returns an authenticated boto3.Session."""
sts_client = boto3.client("sts")
role_arn = f"arn:aws:iam::{account_id}:role/{role_name}"
try:
response = sts_client.assume_role(
RoleArn=role_arn,
RoleSessionName=f"DevSecOpsAudit-{account_id}",
DurationSeconds=3600 # 1 hour temporary token
)
credentials = response["Credentials"]
return boto3.Session(
aws_access_key_id=credentials["AccessKeyId"],
aws_secret_access_key=credentials["SecretAccessKey"],
aws_session_token=credentials["SessionToken"],
)
except ClientError as e:
raise RuntimeError(f"Failed to assume role in {account_id}: {e}")

4.1 FinOps: Finding and Cleaning Orphan Unattached EBS Volumes

Section titled β€œ4.1 FinOps: Finding and Cleaning Orphan Unattached EBS Volumes”

Unattached EBS volumes accrue substantial storage costs while sitting idle after EC2 terminations.

import boto3
from datetime import datetime, timezone, timedelta
def cleanup_orphan_ebs_volumes(ec2_client, retention_days: int = 7, dry_run: bool = True):
"""Identifies and purges unattached 'available' EBS volumes older than retention threshold."""
paginator = ec2_client.get_paginator("describe_volumes")
iterator = paginator.paginate(Filters=[{"Name": "status", "Values": ["available"]}])
threshold_date = datetime.now(timezone.utc) - timedelta(days=retention_days)
orphan_vols = []
for page in iterator:
for volume in page["Volumes"]:
create_time = volume["CreateTime"]
vol_id = volume["VolumeId"]
vol_size_gb = volume["Size"]
if create_time < threshold_date:
orphan_vols.append((vol_id, vol_size_gb))
print(f"[DISCOVERED] Orphan Volume: {vol_id} ({vol_size_gb} GB, Created: {create_time.date()})")
if not dry_run:
ec2_client.delete_volume(VolumeId=vol_id)
print(f" └─ Successfully deleted {vol_id}")
total_wasted_gb = sum(s for _, s in orphan_vols)
print(f"Summary: Found {len(orphan_vols)} unattached volumes wasting {total_wasted_gb} GB.")
return orphan_vols

4.2 SecOps: Event-Driven Remediation of Unsafe Security Groups

Section titled β€œ4.2 SecOps: Event-Driven Remediation of Unsafe Security Groups”

When a developer inadvertently opens port 22 (SSH) or 3389 (RDP) to the entire world (0.0.0.0/0), AWS EventBridge invokes this Lambda function to instantly revoke the permission within seconds.

import boto3
import json
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
DANGEROUS_PORTS = {22, 3389}
def lambda_handler(event, context):
"""EventBridge triggered function on AuthorizeSecurityGroupIngress events."""
ec2 = boto3.client("ec2")
detail = event.get("detail", {})
group_id = detail.get("requestParameters", {}).get("groupId")
ip_permissions = detail.get("requestParameters", {}).get("ipPermissions", {}).get("items", [])
if not group_id or not ip_permissions:
return {"status": "No action taken"}
for perm in ip_permissions:
from_port = perm.get("fromPort")
to_port = perm.get("toPort")
ip_ranges = [r.get("cidrIp") for r in perm.get("ipRanges", {}).get("items", [])]
if "0.0.0.0/0" in ip_ranges and any(p in range(from_port, to_port + 1) for p in DANGEROUS_PORTS):
logger.warning(f"CRITICAL: World-accessible management port detected on {group_id}!")
# Auto-remediate: Instantly revoke the ingress rule
ec2.revoke_security_group_ingress(
GroupId=group_id,
IpPermissions=[perm]
)
logger.info(f"Remediated: Revoked dangerous ingress on {group_id} for port range {from_port}-{to_port}.")
return {"status": "Remediation scan completed"}

4.3 IAM Governance: Automatic Inactive Access Key Deactivation

Section titled β€œ4.3 IAM Governance: Automatic Inactive Access Key Deactivation”

Enforcing CIS AWS Foundations Benchmark 1.4 requires rotating or disabling access keys older than 90 days:

import boto3
from datetime import datetime, timezone, timedelta
def audit_and_deactivate_stale_iam_keys(iam_client, max_age_days: int = 90):
"""Scans all IAM users and disables keys that exceed the maximum age."""
users_paginator = iam_client.get_paginator("list_users")
cutoff_date = datetime.now(timezone.utc) - timedelta(days=max_age_days)
for page in users_paginator.paginate():
for user in page["Users"]:
user_name = user["UserName"]
keys = iam_client.list_access_keys(UserName=user_name)["AccessKeyMetadata"]
for key in keys:
key_id = key["AccessKeyId"]
status = key["Status"]
created = key["CreateDate"]
if status == "Active" and created < cutoff_date:
age_days = (datetime.now(timezone.utc) - created).days
print(f"[STALE KEY] User: {user_name} | Key: {key_id} | Age: {age_days} days")
# Deactivate stale key
iam_client.update_access_key(
UserName=user_name,
AccessKeyId=key_id,
Status="Inactive"
)
print(f" └─ Status set to INACTIVE.")

  1. Use botocore.config.Config(retries={'mode': 'adaptive'}) to prevent catastrophic cascades and API rate-limiting failure in high-concurrency environments.
  2. Never hardcode AWS keys in source files. Rely on IAM Roles for EC2/ECS/EKS pods (IRSA) or short-lived STS tokens.
  3. Always use server-side filters and paginators (client.get_paginator) to reduce memory consumption and API latency when querying large infrastructures.
  4. Build unit tests with moto: Test Python Boto3 automation scripts locally by mocking AWS services using the moto test library before executing against live accounts.