CloudFormation policy compliance monitoring: leveraging CloudTrail and Athena

Introduction: This post details an implementation that monitors who or what changed AWS resources created by CloudFormation. By leveraging AWS Config, CloudTrail, Athena, and Lambda, changes can be tracked, logs can be analysed, and compliance reporting can be automated. The collected data is stored in Amazon S3, making it accessible for audits and compliance verification. About the Project: This post builds upon my earlier article on monitoring CloudFormation stack drift using AWS Config Rules. In this enhanced version, the monitoring capabilities are extended by: Tracking user activities affecting CloudFormation resources. Logging change details to Amazon S3 via CloudTrail. Processing and querying logs using AWS Athena. Automating remediation through AWS Systems Manager and Lambda. Core Components: AWS Config Rule: Monitors stacks for drift using CLOUDFORMATION_STACK_DRIFT_DETECTION_CHECK. Systems Manager Automation Runbook: Invokes a Lambda function for compliance checks. Remediation Action: Executes the InvokeLambdaFromConfig automation document. Amazon S3 Bucket: Stores logs from CloudTrail and Athena query results. Athena Table: Organises and queries raw log data. CloudTrail Trail: Captures AWS API activity logs. Lambda Function: Extracts CloudFormation resource names and queries Athena for recent changes. Infrastructure Scema: Configuration in infrastructure/monitoring_stack_cloudtrail.yaml CloudFormation template: AWSTemplateFormatVersion: '2010-09-09' Description: CloudTrail setup for monitoring CFN stack modifications Parameters: AthenaDatabaseName: Type: String Description: Athena database name for running queries Default: 'cloudtrail_logs' StackNameToMonitor: Type: String Description: CloudFormation stack name to monitor Default: 'base-infrastructure' MaximumExecutionFrequency: Type: String Description: The maximum frequency with which drift in CloudFormation stacks need to be evaluated Default: 'One_Hour' Resources: ################################# # CloudTrail and Athena ################################# CloudTrailLogsBucket: Type: AWS::S3::Bucket Properties: BucketName: !Sub "aws-cloudtrail-logs-${AWS::AccountId}" VersioningConfiguration: Status: Enabled LifecycleConfiguration: Rules: - Id: ExpireLogs Status: Enabled ExpirationInDays: 365 CloudTrailLogsBucketPolicy: Type: AWS::S3::BucketPolicy Properties: Bucket: !Ref CloudTrailLogsBucket PolicyDocument: Version: "2012-10-17" Statement: - Sid: "AWSCloudTrailAclCheck" Effect: Allow Principal: Service: cloudtrail.amazonaws.com Action: s3:GetBucketAcl Resource: !Sub "arn:${AWS::Partition}:s3:::aws-cloudtrail-logs-${AWS::AccountId}" Condition: StringEquals: AWS:SourceArn: !Sub "arn:${AWS::Partition}:cloudtrail:${AWS::Region}:${AWS::AccountId}:trail/monitoring-cfn-policy-compliance" - Sid: "AWSCloudTrailWrite" Effect: Allow Principal: Service: cloudtrail.amazonaws.com Action: s3:PutObject Resource: !Sub "arn:${AWS::Partition}:s3:::aws-cloudtrail-logs-${AWS::AccountId}/AWSLogs/${AWS::AccountId}/" Condition: StringEquals: AWS:SourceArn: !Sub "arn:${AWS::Partition}:cloudtrail:${AWS::Region}:${AWS::AccountId}:trail/monitoring-cfn-policy-compliance" s3:x-amz-acl: "bucket-owner-full-control" - Sid: "AthenaQueryResultPutObject" Effect: Allow Principal: Service: athena.amazonaws.com Action: s3:PutObject Resource: !Sub "arn:${AWS::Partition}:s3:::aws-cloudtrail-logs-${AWS::AccountId}/athena-results/" Condition: StringEquals: aws:SourceArn: !Sub "arn:${AWS::Partition}:athena:${AWS::Region}:${AWS::AccountId}:workgroup/primary" CloudTrail: Type: AWS::CloudTrail::Trail Properties: TrailName: monitoring-cfn-policy-compliance S3BucketName: !Ref CloudTrailLogsBucket IncludeGlobalServiceEvents: true IsMultiRegionTrail: true EnableLogFileValidation: false IsOrganizationTrail: false IsLogging: true AthenaDatabase: Type: AWS::Glue::Database Properties: CatalogId: !Ref AWS::AccountId DatabaseInput: Name: !Ref AthenaDatabaseName AthenaTable: Type: AWS::Glue::Table Properties: CatalogId: !Ref AWS::AccountId

Mar 13, 2025 - 22:23

CloudFormation policy compliance monitoring: leveraging CloudTrail and Athena

Introduction:

This post details an implementation that monitors who or what changed AWS resources created by CloudFormation. By leveraging AWS Config, CloudTrail, Athena, and Lambda, changes can be tracked, logs can be analysed, and compliance reporting can be automated. The collected data is stored in Amazon S3, making it accessible for audits and compliance verification.

About the Project:

This post builds upon my earlier article on monitoring CloudFormation stack drift using AWS Config Rules. In this enhanced version, the monitoring capabilities are extended by:

Tracking user activities affecting CloudFormation resources.
Logging change details to Amazon S3 via CloudTrail.
Processing and querying logs using AWS Athena.
Automating remediation through AWS Systems Manager and Lambda.

Core Components:
AWS Config Rule: Monitors stacks for drift using CLOUDFORMATION_STACK_DRIFT_DETECTION_CHECK.
Systems Manager Automation Runbook: Invokes a Lambda function for compliance checks.
Remediation Action: Executes the InvokeLambdaFromConfig automation document.
Amazon S3 Bucket: Stores logs from CloudTrail and Athena query results.
Athena Table: Organises and queries raw log data.
CloudTrail Trail: Captures AWS API activity logs.
Lambda Function: Extracts CloudFormation resource names and queries Athena for recent changes.
Infrastructure Scema:

Configuration in infrastructure/monitoring_stack_cloudtrail.yaml CloudFormation template:

    AWSTemplateFormatVersion: '2010-09-09'
    Description: CloudTrail setup for monitoring CFN stack modifications

    Parameters:
      AthenaDatabaseName:
        Type: String
        Description: Athena database name for running queries
        Default: 'cloudtrail_logs'
      StackNameToMonitor:
        Type: String
        Description: CloudFormation stack name to monitor
        Default: 'base-infrastructure'
      MaximumExecutionFrequency:
        Type: String
        Description: The maximum frequency with which drift in CloudFormation stacks need to be evaluated
        Default: 'One_Hour'

    Resources:
    #################################
    # CloudTrail and Athena
    #################################
      CloudTrailLogsBucket:
        Type: AWS::S3::Bucket
        Properties:
          BucketName: !Sub "aws-cloudtrail-logs-${AWS::AccountId}"
          VersioningConfiguration:
            Status: Enabled
          LifecycleConfiguration:
            Rules:
              - Id: ExpireLogs
                Status: Enabled
                ExpirationInDays: 365

      CloudTrailLogsBucketPolicy:
        Type: AWS::S3::BucketPolicy
        Properties:
          Bucket: !Ref CloudTrailLogsBucket
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Sid: "AWSCloudTrailAclCheck"
                Effect: Allow
                Principal:
                  Service: cloudtrail.amazonaws.com
                Action: s3:GetBucketAcl
                Resource: !Sub "arn:${AWS::Partition}:s3:::aws-cloudtrail-logs-${AWS::AccountId}"
                Condition:
                  StringEquals:
                    AWS:SourceArn: !Sub "arn:${AWS::Partition}:cloudtrail:${AWS::Region}:${AWS::AccountId}:trail/monitoring-cfn-policy-compliance"
              - Sid: "AWSCloudTrailWrite"
                Effect: Allow
                Principal:
                  Service: cloudtrail.amazonaws.com
                Action: s3:PutObject
                Resource: !Sub "arn:${AWS::Partition}:s3:::aws-cloudtrail-logs-${AWS::AccountId}/AWSLogs/${AWS::AccountId}/*"
                Condition:
                  StringEquals:
                    AWS:SourceArn: !Sub "arn:${AWS::Partition}:cloudtrail:${AWS::Region}:${AWS::AccountId}:trail/monitoring-cfn-policy-compliance"
                    s3:x-amz-acl: "bucket-owner-full-control"
              - Sid: "AthenaQueryResultPutObject"
                Effect: Allow
                Principal:
                  Service: athena.amazonaws.com
                Action: s3:PutObject
                Resource: !Sub "arn:${AWS::Partition}:s3:::aws-cloudtrail-logs-${AWS::AccountId}/athena-results/*"
                Condition:
                  StringEquals:
                    aws:SourceArn: !Sub "arn:${AWS::Partition}:athena:${AWS::Region}:${AWS::AccountId}:workgroup/primary"

      CloudTrail:
        Type: AWS::CloudTrail::Trail
        Properties:
          TrailName: monitoring-cfn-policy-compliance
          S3BucketName: !Ref CloudTrailLogsBucket
          IncludeGlobalServiceEvents: true
          IsMultiRegionTrail: true
          EnableLogFileValidation: false
          IsOrganizationTrail: false
          IsLogging: true

      AthenaDatabase:
        Type: AWS::Glue::Database
        Properties:
          CatalogId: !Ref AWS::AccountId
          DatabaseInput:
            Name: !Ref AthenaDatabaseName

      AthenaTable:
        Type: AWS::Glue::Table
        Properties:
          CatalogId: !Ref AWS::AccountId
          DatabaseName: !Ref AthenaDatabase
          TableInput:
            Name: !Sub "aws_cloudtrail_logs_${AWS::AccountId}"
            TableType: EXTERNAL_TABLE
            Parameters:
              classification: cloudtrail
            StorageDescriptor:
              Columns:
                - Name: eventVersion
                  Type: string
                - Name: userIdentity
                  Type: struct,sessionIssuer:struct,ec2RoleDelivery:string,webIdFederationData:struct>>>
                - Name: eventTime
                  Type: string
                - Name: eventSource
                  Type: string
                - Name: eventName
                  Type: string
                - Name: awsRegion
                  Type: string
                - Name: sourceIpAddress
                  Type: string
                - Name: userAgent
                  Type: string
                - Name: errorCode
                  Type: string
                - Name: errorMessage
                  Type: string
                - Name: requestParameters
                  Type: string
                - Name: responseElements
                  Type: string
                - Name: additionalEventData
                  Type: string
                - Name: requestId
                  Type: string
                - Name: eventId
                  Type: string
                - Name: resources
                  Type: array>
                - Name: eventType
                  Type: string
                - Name: apiVersion
                  Type: string
                - Name: readOnly
                  Type: string
                - Name: recipientAccountId
                  Type: string
                - Name: serviceEventDetails
                  Type: string
                - Name: sharedEventID
                  Type: string
                - Name: vpcEndpointId
                  Type: string
                - Name: tlsDetails
                  Type: struct
              Location: !Sub "s3://aws-cloudtrail-logs-${AWS::AccountId}/AWSLogs/${AWS::AccountId}/CloudTrail/"
              InputFormat: com.amazon.emr.cloudtrail.CloudTrailInputFormat
              OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
              SerdeInfo:
                SerializationLibrary: org.apache.hive.hcatalog.data.JsonSerDe

    #################################
    # Lambda function
    #################################
      LambdaExecutionRole:
        Type: AWS::IAM::Role
        Properties:
          RoleName: LambdaAthenaQueryExecutionRole
          AssumeRolePolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Principal:
                  Service: 
                    - lambda.amazonaws.com
                    - athena.amazonaws.com
                Action:
                  - sts:AssumeRole
          Policies:
            - PolicyName: CloudFormationDescribe
              PolicyDocument:
                Version: "2012-10-17"
                Statement:
                  - Effect: Allow
                    Action:
                      - cloudformation:DescribeStackResources
                    Resource: "arn:aws:cloudformation:*"
            - PolicyName: AthenaQueryPolicy
              PolicyDocument:
                Version: "2012-10-17"
                Statement:
                  - Effect: Allow
                    Action:
                    - athena:StartQueryExecution
                    - athena:GetQueryExecution
                    - athena:GetQueryResults
                    - athena:GetWorkGroup
                    - athena:GetDataCatalog
                    - athena:GetTableMetadata
                    - glue:GetDatabase
                    - glue:GetTable
                    - glue:GetPartitions
                    Resource: "*"
                  - Effect: Allow
                    Action:
                    - s3:PutObject
                    - s3:GetObject
                    - s3:ListBucket
                    - s3:GetBucketLocation
                    - s3:PutObjectAcl
                    Resource: 
                      - !Sub "arn:${AWS::Partition}:s3:::${CloudTrailLogsBucket}"
                      - !Sub "arn:${AWS::Partition}:s3:::${CloudTrailLogsBucket}/*"
                  - Effect: Allow
                    Action:
                      - lambda:AddPermission
                    Resource: "*"
                  - Effect: Allow
                    Action:
                      - logs:CreateLogGroup
                      - logs:CreateLogStream
                      - logs:PutLogEvents
                    Resource: "*"

      CheckCloudTrailLogsLambda:
        Type: AWS::Lambda::Function
        Properties:
          FunctionName: CheckCloudTrailLogsLambda
          Runtime: nodejs22.x
          Handler: index.handler
          Role: !GetAtt LambdaExecutionRole.Arn
          Timeout: 120
          MemorySize: 256
          Environment:
            Variables:
              STACKS_TO_MONITOR: !Ref StackNameToMonitor
              ATHENA_DATABASE: !Ref AthenaDatabase
              ATHENA_TABLE: !Ref AthenaTable
              S3_OUTPUT_BUCKET: !Ref CloudTrailLogsBucket
          Code:
            ZipFile: |
              const { AthenaClient, StartQueryExecutionCommand } = require("@aws-sdk/client-athena");
              const { CloudFormationClient, DescribeStackResourcesCommand } = require("@aws-sdk/client-cloudformation");

              const athena = new AthenaClient({});
              const cloudformation = new CloudFormationClient({});

              exports.handler = async (event) => {
                  console.log("Event received:", JSON.stringify(event, null, 2));

                  const stacks = process.env.STACKS_TO_MONITOR.split(",");
                  const tableName = process.env.ATHENA_TABLE;
                  const databaseName = process.env.ATHENA_DATABASE;
                  const s3Bucket = process.env.S3_OUTPUT_BUCKET;

                  let resourceNames = [];

                  // Extract resource names from the CloudFormation stacks
                  for (const stack of stacks) {
                      const stackResources = await cloudformation.send(
                          new DescribeStackResourcesCommand({ StackName: stack })
                      );

                      stackResources.StackResources.forEach(resource => {
                          if (resource.PhysicalResourceId) {
                              resourceNames.push(resource.PhysicalResourceId);
                          }
                      });
                  }

                  // Construct Athena query
                  let whereClause = resourceNames.map(name => `resource.arn LIKE '%${name}%'`).join(" OR ");
                  let queryString = `
                      SELECT 
                          userIdentity.userName AS username,
                          eventName AS action,
                          eventTime AS timestamp,
                          resource.arn AS resource_arn,
                          sourceIPAddress AS request_source,
                          userAgent AS user_agent
                      FROM ${tableName}
                      CROSS JOIN UNNEST(resources) AS t(resource)
                      WHERE (${whereClause})
                      AND eventName IS NOT NULL
                      AND userIdentity.userName IS NOT NULL
                      AND from_iso8601_timestamp(eventTime) >= current_timestamp - INTERVAL '1' HOUR
                      ORDER BY from_iso8601_timestamp(eventTime) DESC;
                  `;

                  // Run the Athena query
                  const params = {
                      QueryString: queryString,
                      QueryExecutionContext: { Database: databaseName },
                      ResultConfiguration: { OutputLocation: `s3://${s3Bucket}/athena-results/` }
                  };

                  try {
                      const command = new StartQueryExecutionCommand(params);
                      const queryExecution = await athena.send(command);
                      console.log("Query started:", queryExecution.QueryExecutionId);
                      return { status: "Query started successfully", queryExecutionId: queryExecution.QueryExecutionId };
                  } catch (error) {
                      console.error("Error running query:", error);
                      throw error;
                  }
              };

      LambdaPermissionForConfig:
        Type: AWS::Lambda::Permission
        Properties:
          FunctionName: !Ref CheckCloudTrailLogsLambda
          Action: lambda:InvokeFunction
          Principal: config.amazonaws.com

    #################################
    # Config Rule
    #################################
      IamRoleForConfig2:
        Type: AWS::IAM::Role
        Properties:
          RoleName: CfnDriftDetectionForCloudTrail
          Description: IAM role for AWS Config to access CloudFormation drift detection
          AssumeRolePolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Principal:
                  Service: config.amazonaws.com
                Action:
                  - sts:AssumeRole
          ManagedPolicyArns:
            - arn:aws:iam::aws:policy/ReadOnlyAccess
          Policies:
            - PolicyName: CloudFormationDriftDetectionpolicy
              PolicyDocument:
                Version: "2012-10-17"
                Statement:
                  - Effect: Allow
                    Action:
                      - cloudformation:DetectStackResourceDrift
                      - cloudformation:DetectStackDrift
                      - cloudformation:DescribeStacks
                      - cloudformation:DescribeStackResources
                      - cloudformation:BatchDescribeTypeConfigurations
                      - cloudformation:DescribeStackResourceDrifts
                      - cloudformation:DescribeStackDriftDetectionStatus
                    Resource: !Sub "arn:${AWS::Partition}:cloudformation:${AWS::Region}:${AWS::AccountId}:*"

      ConfigRuleCheckCloudTralLogs:
        DependsOn:
        - LambdaPermissionForConfig
        Type: AWS::Config::ConfigRule
        Properties:
          ConfigRuleName: ConfigRuleCheckCloudTrailLogs
          Description: AWS Config rule to detect drift in CFN stacks and check CloudTrail logs
          Scope:
            TagKey: stack-name
            TagValue: !Ref StackNameToMonitor
          Source:
            Owner: AWS
            SourceIdentifier: CLOUDFORMATION_STACK_DRIFT_DETECTION_CHECK
          MaximumExecutionFrequency: !Ref MaximumExecutionFrequency
          InputParameters:
            cloudformationRoleArn: !GetAtt IamRoleForConfig2.Arn

      IamRoleForRemediation:
        Type: AWS::IAM::Role
        Properties:
          RoleName: AwsConfigRemediationActionInvokeLambda
          Description: IAM role for AWS Config remediation action to invoke Lambda function
          AssumeRolePolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Principal:
                  Service:
                    - config.amazonaws.com
                    - ssm.amazonaws.com
                Action:
                  - sts:AssumeRole
          Policies:
            - PolicyName: InvokeLambdaPolicy
              PolicyDocument:
                Version: '2012-10-17'
                Statement:
                  - Effect: Allow
                    Action:
                      - lambda:InvokeFunction
                    Resource: !GetAtt CheckCloudTrailLogsLambda.Arn

      SsmDocumentInvokeLambda:
        Type: AWS::SSM::Document
        Properties:
          DocumentType: Automation
          Name: InvokeLambdaFromConfig
          Content:
            schemaVersion: "0.3"
            description: "SSM Automation document to invoke a Lambda function"
            parameters:
              AutomationAssumeRole:
                type: String
                description: (Optional) The ARN of the role that allows Automation to perform the actions
                default: !GetAtt IamRoleForRemediation.Arn
            mainSteps:
              - name: InvokeLambda
                action: aws:invokeLambdaFunction
                inputs:
                  FunctionName: !Ref CheckCloudTrailLogsLambda
                  Payload: '{}'
                  InvocationType: Event
                  LogType: None
                maxAttempts: 2
                timeoutSeconds: 30
                onFailure: Abort
                isCritical: true
            assumeRole: !GetAtt IamRoleForRemediation.Arn

      RemediationActionInvokeLambda:
        Type: AWS::Config::RemediationConfiguration
        Properties:
          ConfigRuleName: !Ref ConfigRuleCheckCloudTralLogs
          TargetType: SSM_DOCUMENT
          TargetId: !Ref SsmDocumentInvokeLambda
          Automatic: true
          MaximumAutomaticAttempts: 2
          RetryAttemptSeconds: 30
          Parameters:
            AutomationAssumeRole:
              StaticValue:
                Values:
                  - !GetAtt IamRoleForRemediation.Arn

Prerequisites:

Ensure the following prerequisites are in place:

An AWS account with sufficient permissions to create and manage resources.
The AWS CLI installed on the local machine.
CloudFormation infrastructure deployed from my previous post (if applicable).

Deployment:

Deploy the CloudFormation Stack.

    aws cloudformation create-stack \
        --stack-name monitoring-policy-compliance \
        --template-body file://infrastructure/monitoring_stack_cloudtrail.yaml \
        --capabilities CAPABILITY_NAMED_IAM \
        --disable-rollback

2.Test the resources deployed with the stack. Change value of the resource from the base-infrastructure stack and evaluate the drift detection rule to verify functionality.

    aws ssm put-parameter --name "ConnectionToken" --value "secret_token_value_2" --type "String" --overwrite

    aws configservice start-config-rules-evaluation --config-rule-names ConfigRuleCheckCloudTrailLogs

3.After drift detection runs, review Athena query results stored in S3 under /athena-results in .csv file.

    aws s3 ls s3:///athena-results/ --recursive

    aws s3 cp s3:///.csv ./

Here is an example of logs from this file:

)

4.Cleanup Resources. After testing, stop the CloudTrail Trail logging, delete all data from the S3 bucket, and delete the CloudFormation stack.

    aws cloudtrail stop-logging --name monitor-cfn-policy-compliance

    aws s3 rm s3:// --recursive

    aws cloudformation delete-stack --stack-name monitoring-policy-compliance

Conclusion:

Implementing this solution provides visibility into changes affecting CloudFormation-managed resources. This enhances security, compliance tracking, and audit readiness. The ability to log and query user actions simplifies responses to compliance requests from clients, regulators, or security teams.

If you found this post helpful and interesting, please click the reaction button below to show your support. Feel free to use and share this post.