Skip to content
About Me
----

Observability & Auditing: CloudWatch, CloudTrail & Systems Manager

Architect Key Takeaways

Master enterprise AWS observability and auditing: Amazon CloudWatch metrics/logs, immutable CloudTrail API auditing with log file validation, and AWS Systems Manager (SSM) Session Manager.

Observability & Auditing: CloudWatch, CloudTrail & SSM

Section titled β€œObservability & Auditing: CloudWatch, CloudTrail & SSM”

In enterprise DevSecOps, you cannot defend or optimize what you cannot observe. When a zero-day exploit strikes or an outage paralyzes production, engineers cannot rely on guesswork.

AWS divides observability and operational management into three pillars: Real-Time Performance Metrics & Alarms (CloudWatch), Cryptographic Audit Trails (CloudTrail), and Zero-Trust Infrastructure Management (Systems Manager).


1. CloudWatch vs. CloudTrail: The Fundamental Distinction

Section titled β€œ1. CloudWatch vs. CloudTrail: The Fundamental Distinction”

A foundational distinction every cloud architect and SRE must master:

Operational Dimension Amazon CloudWatch AWS CloudTrail
Primary Question β€œWhat is the health and performance of my systems right now?” β€œWho made which API request, from where, and at what time?”
Data Format Numerical Time-Series Metrics, Logs & Alarms Immutable JSON Event Records
Data Ingestion CPU utilization, RAM usage, 5xx HTTP errors, Disk I/O CreateBucket, AuthorizeSecurityGroupIngress, RunInstances
Primary Consumers SREs, DevOps Engineers, Performance Architects Security Operations Center (SOC), Compliance Auditors
Action Mechanism Triggers Auto Scaling policies, SNS alerts, Lambda remediation Triggers EventBridge security alerts, SIEM ingestion (Splunk)

CloudTrail records every single API activity across your AWS account. To ensure audit trail records remain admissible in legal audits and resilient against compromised administrator accounts:

[ Developer / Hacker API Call ]
β”‚
β–Ό
[ AWS CloudTrail Engine ]
β”‚
β–Ό (Captures JSON event metadata)
[ Amazon S3 Central Log Bucket (Dedicated Security Account) ]
β”‚
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β–Ό β–Ό
[ SHA-256 Digest ] [ S3 Object Lock (WORM) ]
(Log Validation) (Compliance Mode: Cannot Delete!)
  1. Multi-Region Organization Trail: Enabled centrally from the AWS Organizations Management account. Automatically captures API actions in all current and future AWS regions across all member accounts.
  2. Log File Validation Enabled: CloudTrail hashes log files using SHA-256 digital signatures and produces hourly digest files (.digest). If an attacker alters or deletes a single log entry in S3, the validation check fails immediately, exposing the tamper attempt.
  3. Dedicated Security Account Destination: Ship logs directly to an isolated Log Archive AWS Account where developers and workload administrators have zero IAM permissions.

3. AWS Systems Manager (SSM) Session Manager: Eliminating SSH & Bastion Hosts

Section titled β€œ3. AWS Systems Manager (SSM) Session Manager: Eliminating SSH & Bastion Hosts”

In traditional cloud operations, connecting to an EC2 instance required:

  • Opening inbound Port 22 (SSH) or Port 3389 (RDP) on firewalls.
  • Maintaining expensive β€œBastion Jump Boxes” exposed to internet port scanners.
  • Generating, distributing, and rotating fragile private SSH key pairs (.pem files).

AWS Systems Manager Session Manager replaces this legacy paradigm with Zero-Trust IAM-driven access:

[ Engineer Browser / CLI ] ──> [ AWS SSM Control Plane (TLS Port 443) ] ──> [ amazon-ssm-agent on EC2 ]
  • Zero Open Inbound Ports: Security Groups need zero open inbound ports (0.0.0.0/0:22 is removed!). The amazon-ssm-agent initiates outbound polling to the SSM service over TLS (port 443).
  • No SSH Key Management: Access is governed entirely by standard IAM policies and corporate Single Sign-On (SSO) with Multi-Factor Authentication (MFA).
  • Complete Keystroke Session Logging: Every single shell command typed during an active terminal session is encrypted and streamed in real-time to Amazon S3 and Amazon CloudWatch Logs for forensic auditing.

Incident: Detecting Unauthorized Security Group Modifications via CloudTrail

Section titled β€œIncident: Detecting Unauthorized Security Group Modifications via CloudTrail”

Scenario: At 2:00 AM, an alert fires indicating port 22 or 3389 was opened to 0.0.0.0/0 on a production database security group.

Investigation Runbook:

  1. Query CloudTrail using AWS CLI to immediately identify the caller identity and IP:
    Terminal window
    aws cloudtrail lookup-events \
    --lookup-attributes AttributeKey=EventName,AttributeValue=AuthorizeSecurityGroupIngress \
    --max-results 1
  2. Inspect the CloudTrail JSON output:
    • userIdentity.arn: Reveals whether an engineer’s credentials or an automated CI/CD role executed the change.
    • sourceIPAddress: Identifies the originating public IP address.
    • requestParameters: Details the exact CIDR and port opened.
  3. Automatically remediate using AWS Config and EventBridge: Trigger a Lambda function that automatically revokes non-compliant security group ingress rules within 5 seconds of the CloudTrail event.

1. Why is CloudTrail Log File Validation essential during a forensic breach investigation?

Answer: When sophisticated attackers breach an environment, their first action is often attempting to delete or modify CloudTrail logs in S3 to cover their tracks. Log File Validation uses cryptographic SHA-256 hashing. If any log file is deleted, modified, or appended out-of-band, running aws cloudtrail validate-logs flags a cryptographic signature mismatch, proving the logs were tampered with.

2. How does Amazon CloudWatch Agent capture memory (RAM) usage on an EC2 instance when standard CloudWatch metrics cannot?

Answer: Standard CloudWatch metrics are collected at the hypervisor layer outside the virtual machine (capturing CPU utilization, disk I/O, and network packet counts). The hypervisor cannot see inside the guest operating system’s internal RAM allocation. The Unified CloudWatch Agent runs inside the guest OS as a daemon, reading internal kernel memory buffers and streaming detailed metrics (RAM utilization, swap usage, disk space) into CloudWatch over HTTPS.