AWS Cloud Security Detection Pipeline
A serverless, event-driven pipeline on AWS that monitors an account's activity logs and automatically flags suspicious or high-risk behavior — analyzing new logs the moment they land, with no server to run.
01 Overview
Cloud accounts generate an enormous volume of activity logs, and the signal that matters — someone using the root account, an IAM policy quietly widening, a bucket going public — is buried in thousands of routine events. This pipeline solves the "nobody is reading the logs" problem: it ingests AWS CloudTrail activity, runs a set of security detection rules against every event automatically, and writes structured reports flagging what a reviewer should actually look at, ranked by severity.
02 Architecture
The system is fully event-driven — no server polling or running on a schedule. Each stage triggers the next:
03 Technologies & why
- AWS Lambda — serverless removes any always-on host to manage or pay for; runs only when a log arrives, which fits a bursty event stream.
- Amazon S3 + event notifications — the natural CloudTrail delivery target, and its native trigger makes the pipeline react in real time instead of on a timer.
- CloudTrail — the authoritative record of who did what in the account; the correct source of truth for account-level detection.
- Python — fast to write JSON parsing and rule logic in, and first-class in the Lambda runtime.
04 Implementation
The Lambda parses raw CloudTrail JSON and evaluates each event against six rule-based checks spanning four categories — authentication, IAM, privilege escalation, and public-resource exposure:
| ID | Rule | Severity |
|---|---|---|
| R-01 | Repeated failed console login attempts from the same source | Medium |
| R-02 | Root account usage | High |
| R-03 | Unusual / high-risk API activity | Medium |
| R-04 | IAM permission or policy changes tied to potential privilege escalation | High |
| R-05 | Overly permissive IAM policies | Medium |
| R-06 | S3 buckets or resources configured with public access | High |
Each matched event is tagged with a severity level and collected into a structured report, so the output is triaged rather than a flat dump. The pipeline has been run against a dataset of 10,000+ CloudTrail events.
05 Challenges
06 Solutions
07 Results
The pipeline analyzes 10,000+ CloudTrail events through six rule-based checks across authentication, IAM, privilege-escalation, and public-resource-exposure categories, producing severity-ranked reports. Because it's fully event-driven, new logs are analyzed automatically with no manual step.
08 Limitations
09 Future improvements
10 Source
The full source is on GitHub.