Skip to content
About Me
----

Data Encryption & Multi-Account Governance: KMS & Organizations

Architect Key Takeaways

Master enterprise cloud governance: AWS Key Management Service (KMS) envelope encryption, multi-account Landing Zones, and Service Control Policies (SCPs).

Data Encryption & Multi-Account Governance: KMS & Organizations

Section titled β€œData Encryption & Multi-Account Governance: KMS & Organizations”

Operating securely in the cloud requires moving beyond single-account architectures toward Multi-Account Landing Zones governed by preventative guardrails and hardware-backed cryptographic keys. In enterprise cloud engineering, security is enforced deterministically through mathematical cryptography (AWS KMS) and organizational perimeter policies (AWS Organizations and SCPs).


AWS Key Management Service (AWS KMS) uses FIPS 140-2 Level 3 (and FIPS 140-3 Level 3) Hardware Security Modules (HSMs) to generate, store, and manage cryptographic keys.

Encrypting large payloads (such as a 500 GB Aurora database snapshot, EBS volumes, or 5 TB S3 objects) directly inside an HSM over network TLS calls introduces prohibitive latency and severe throughput bottlenecks. Envelope encryption solves this by decoupling key hierarchy from bulk data processing:

[ Customer Managed Key (CMK / KMS Master Key in HSM) ]
β”‚
β–Ό (GenerateDataKey API call)
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Plaintext Data Key (DEK) 256b β”‚ ──> Encrypts 500GB Data locally in host RAM (AES-GCM-256)
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
β”‚ (Plaintext DEK is explicitly overwritten / zeroed in memory!)
β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Encrypted Data Key (Ciphertext)β”‚ ──> Stored safely in object metadata / header alongside data
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
  1. Root Master Key (KMS CMK): Never leaves the physical HSM in plaintext under any circumstances.
  2. GenerateDataKey API: KMS returns two components: a plaintext Data Encryption Key (DEK) and an encrypted copy of the same DEK (ciphertext).
  3. Local Encryption: The host application or AWS service encrypts the data using the plaintext DEK, zeroes out the plaintext DEK in memory immediately, and attaches the encrypted DEK to the ciphertext block.
  4. Decryption Cycle: To read the data, the client sends only the encrypted DEK to kms:Decrypt. KMS unwraps the DEK using the HSM master key, returns the ephemeral DEK over TLS, and the client decrypts the payload locally.

Production AWS CLI: Generating and Testing Envelope Encryption

Section titled β€œProduction AWS CLI: Generating and Testing Envelope Encryption”
Terminal window
# 1. Generate an envelope Data Key (AES_256)
aws kms generate-data-key \
--key-id alias/prod-data-protection-key \
--key-spec AES_256 \
--encryption-context Environment=Production,Application=Billing
# Output returns Plaintext (Base64) and CiphertextBlob (Base64)
# 2. Re-decrypt the data key during recovery inspection
aws kms decrypt \
--ciphertext-blob fileb://encrypted-dek.bin \
--encryption-context Environment=Production,Application=Billing \
--output text \
--query Plaintext | base64 --decode

[!IMPORTANT] Always enforce Encryption Context in KMS API calls. Encryption context acts as authenticated additional data (AAD). If the context provided during kms:Decrypt does not match the exact key-value pairs supplied during encryption, decryption fails with an invalid ciphertext errorβ€”preventing ciphertext tampering and replay attacks.


Feature AWS Owned Keys AWS Managed Keys (aws/s3, aws/ebs) Customer Managed Keys (CMK)
Visibility in KMS Console Invisible Visible (aws/* alias) Fully visible and configurable
Key Policy Customization None Read-only Full control over resource-based key policy
Annual Automatic Rotation Managed by AWS Every 1 year (Automatic) Configurable (1 to 7 years, or manual)
Cross-Account Usage No No (Account-locked) Yes (Enables multi-account access)
Cost Free Free $1.00 / month per key + API request fees

Production Terraform: Multi-Region CMK with Enforced Rotation & Least-Privilege Key Policy

Section titled β€œProduction Terraform: Multi-Region CMK with Enforced Rotation & Least-Privilege Key Policy”
resource "aws_kms_key" "production" {
description = "Production Enterprise CMK with automatic annual rotation"
deletion_window_in_days = 30
enable_key_rotation = true
rotation_period_in_days = 365
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Sid = "EnableRootAdministration"
Effect = "Allow"
Principal = {
AWS = "arn:aws:iam::${var.account_id}:root"
}
Action = "kms:*"
Resource = "*"
},
{
Sid = "AllowAppServiceKeyUsage"
Effect = "Allow"
Principal = {
AWS = var.workload_role_arn
}
Action = [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:DescribeKey"
]
Resource = "*"
Condition = {
StringEquals = {
"kms:EncryptionContext:Environment" = "Production"
}
}
}
]
})
tags = {
Environment = "Production"
ManagedBy = "Terraform"
Compliance = "PCI-DSS-Level-1"
}
}
resource "aws_kms_alias" "production" {
name = "alias/production-workload-cmk"
target_key_id = aws_kms_key.production.key_id
}

Enterprise cloud environments isolate workloads across separate AWS accounts grouped into Organizational Units (OUs). Relying on a single AWS account with separate VPCs creates shared blast radiuses, noisy-neighbor API rate limiting, and administrative credential vulnerabilities.

AWS Organizations Root (Payer Account - ZERO Workloads Allowed!)
β”œβ”€β”€ πŸ“ Core / Foundation OU
β”‚ β”œβ”€β”€ πŸ”’ Log Archive Account (Central S3 Bucket with MFA Delete & S3 Object Lock)
β”‚ └── πŸ›‘οΈ Security Tooling Account (GuardDuty Delegated Admin, Security Hub Master, IAM Access Analyzer)
β”œβ”€β”€ πŸ“ Network & Shared Infrastructure OU
β”‚ β”œβ”€β”€ 🌐 Network Hub Account (Transit Gateway, Direct Connect, Inspection VPCs)
β”‚ └── πŸ“¦ Shared Services Account (Golden AMI pipelines, ECR image registries, centralized CI/CD)
└── πŸ“ Business Workloads OU
β”œβ”€β”€ πŸ“ Development Sub-OU (Dev accounts - permissive IAM, automatic nightly instance shutdown)
β”œβ”€β”€ πŸ“ Staging Sub-OU (Pre-production mirrors, automated integration pipelines)
└── πŸ“ Production Sub-OU (Strict approvals, break-glass only IAM, automated drift alerting)
  • Centralized Billing: Payer account consolidates enterprise volume discounts and Reserved Instance / Savings Plans sharing without hosting EC2 or Lambda workloads.
  • Log Immutability: All VPC Flow Logs, CloudTrail audit trails, and EKS audit logs are pushed to the Log Archive Account where bucket policies reject object deletion or modification, satisfying SOC2 and ISO27001 compliance.
  • Delegated Administrator: Configure Security Hub and GuardDuty delegated admin accounts to inspect member accounts without requiring root access to the master organization payer.

4. Service Control Policies (SCPs): The Non-Negotiable Guardrails

Section titled β€œ4. Service Control Policies (SCPs): The Non-Negotiable Guardrails”

Service Control Policies (SCPs) define the maximum available permissions for accounts within an OU. Even the Root User of a member account cannot bypass an explicit Deny in an SCP.

[ Root Management Account ]
β”‚ (Applies SCP Guardrails)
β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Member Accountβ”‚ ──> Account Root User or AdministratorAccess IAM Role
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
β–Ό
[ Attempt: Disable CloudTrail ] ──> ❌ DENIED by SCP!

Enterprise SCP Pattern 1: Denying Non-Approved AWS Regions (Data Sovereignty)

Section titled β€œEnterprise SCP Pattern 1: Denying Non-Approved AWS Regions (Data Sovereignty)”

Prevents developers and attackers from launching compute instances in unmonitored geographic regions:

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "DenyUnapprovedRegions",
"Effect": "Deny",
"NotAction": [
"iam:*",
"organizations:*",
"route53:*",
"cloudfront:*",
"support:*",
"wafv2:*",
"health:*"
],
"Resource": "*",
"Condition": {
"StringNotEquals": {
"aws:RequestedRegion": [
"ap-south-1",
"us-east-1"
]
}
}
}
]
}

Enterprise SCP Pattern 2: Guarding CloudTrail and Security Guardrails

Section titled β€œEnterprise SCP Pattern 2: Guarding CloudTrail and Security Guardrails”

Blocks any IAM identity from stopping audit logging, deleting logs, or leaving AWS Organizations:

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "PreventDisablingSecurityControls",
"Effect": "Deny",
"Action": [
"cloudtrail:StopLogging",
"cloudtrail:DeleteTrail",
"cloudtrail:UpdateTrail",
"config:DeleteConfigRule",
"config:StopConfigurationRecorder",
"guardduty:DeleteDetector",
"guardduty:DisassociateFromMasterAccount",
"organizations:LeaveOrganization"
],
"Resource": "*"
}
]
}

5. Incident Triage Runbook: AccessDenied on KMS Decrypt During Deployments

Section titled β€œ5. Incident Triage Runbook: AccessDenied on KMS Decrypt During Deployments”

A Lambda function or ECS task fails on startup with KMS.AccessDeniedException: User: arn:aws:iam::111122223333:role/app-task-role is not authorized to perform: kms:Decrypt on resource: arn:aws:kms:ap-south-1:444455556666:key/abc-123.

[ Application Task Role (Account A) ]
β”‚
β–Ό (Calls kms:Decrypt)
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Cross-Account KMS Key (Account B) β”‚ ──> ❌ Fails if Key Policy lacks Account A!
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

For cross-account KMS operations, permission must be granted in both places:

  1. The caller account’s IAM Role Policy must permit kms:Decrypt.
  2. The KMS Key Policy in the resource account must explicitly grant permission to the external account or role.

Inspect the key policy using AWS CLI:

Terminal window
aws kms get-key-policy \
--key-id abc-123 \
--policy-name default \
--output json

Verify that the policy contains a statement delegating access to Account A:

{
"Sid": "AllowExternalAccountUsage",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111122223333:root"
},
"Action": [
"kms:Decrypt",
"kms:DescribeKey"
],
"Resource": "*"
}

If the key policy is correct, check CloudTrail in the KMS account to verify whether the application is omitting or passing mismatched encryption context:

Terminal window
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=EventName,AttributeValue=Decrypt \
--max-results 10 \
--query 'Events[*].CloudTrailEvent' --output text | jq .

Look for errorMessage: "The ciphertext references a key that either does not exist or you are not authorized to access it." This error frequently masks an Encryption Context mismatch. Ensure your code passes identical context key-value pairs during decryption.


Q1: Why does AWS Organizations SCP evaluate before IAM policies, and can an AdministratorAccess role override it?

SCPs define the maximum boundary of permissible API calls for an account or OU. In the AWS policy evaluation logic, an explicit Deny in an SCP immediately results in an unconditional Deny. Because SCPs sit above the account boundary, even the account Root user or an IAM principal with full AdministratorAccess (*:*) cannot bypass or override an SCP restriction.

Q2: An S3 bucket encrypted with SSE-KMS needs cross-account replication. Why does cross-account KMS decryption fail when using the default AWS Managed Key `aws/s3`?

AWS Managed Keys (aws/s3, aws/ebs, etc.) are owned and maintained by AWS for a specific account. Their key policies cannot be modified by customers, and they explicitly prohibit cross-account principals. To replicate encrypted S3 objects or share encrypted snapshots across accounts, you must re-encrypt the data using a Customer Managed Key (CMK) with a resource policy allowing the destination account principal.

Q3: What happens to data encrypted via envelope encryption if the KMS Key is scheduled for deletion with a 7-day waiting period?

Once a KMS key is scheduled for deletion, its status transitions to PendingDeletion, and it immediately rejects all cryptographic operations (including kms:Encrypt, kms:Decrypt, and kms:GenerateDataKey). While the ciphertext and encrypted data keys remain intact on disk, they become cryptographically unreadable. If the deletion is not canceled via aws kms cancel-key-deletion before the waiting period expires, the master key is destroyed permanently, rendering all associated encrypted datasets irrevocably lost.