AWS EFS Analyzer: Optimizing Storage Costs with Amazon Q Developer

This is a submission for the Amazon Q Developer "Quack The Code" Challenge: Crushing the Command Line What I Built I created the AWS EFS Analyzer, a command-line tool designed to help AWS users optimize their Elastic File System (EFS) storage costs. The tool identifies cost-saving opportunities by analyzing file access patterns and recommending transitions to more cost-effective storage tiers—like moving infrequently accessed files from Standard to Infrequent Access or Archive. AWS estimates that customers can save up to 92% this way. Manual identification is tedious and error-prone—EFS Analyzer automates the process, delivering actionable insights with ease. Development Prompt Python Script for EFS Storage Optimization Analysis Create a Python script that analyzes an Amazon EFS (Elastic File System) mount point to identify cost optimization opportunities. The script should: Scan an EFS mount point recursively, with support for parallel processing to handle large file systems efficiently Categorize files based on last access time (7, 14, 30, 60, 90 days, 1 year, 2 years, and older) Calculate total storage size for each access time category Estimate storage costs across different EFS tiers (Standard, Infrequent Access, Archive) Generate detailed reports in both HTML and plain text formats showing: File access statistics by time category Current storage costs (assuming all in Standard tier) Potential optimized costs using appropriate tiers based on access patterns Projected monthly savings Recommendations for tier transitions Key Requirements: Handle large file systems efficiently using parallel processing Provide real-time progress tracking with completion percentage and ETA Automatically exclude system directories (/proc, /sys, /dev, etc.) to prevent infinite recursion Detect and avoid symbolic link loops Support command-line options for: Specifying the EFS mount point Setting parallel processing degree (default: number of CPU cores) Excluding specific directories Setting maximum scan depth Controlling whether to follow symbolic links Redirecting warnings and errors to a log file Display a clean progress bar that shows: Percentage completion Number of directories processed Number of files scanned Estimated time remaining Generate comprehensive reports that include: File access statistics by category Storage size distribution Current vs. optimized cost analysis Tier distribution recommendations Potential monthly savings Iterative Improvements with Amazon Q Developer Throughout the development process of the AWS EFS Analyzer, I worked iteratively with Amazon Q Developer to refine and enhance the tool. Each step built upon the last, resulting in a more robust and user-friendly solution. Key Iterative Enhancements Here are some examples of follow-up prompts that guided the improvement process: Handle System Directory Permissions Prompt: “Fix the permission errors when scanning system directories like /proc.” Result: Amazon Q quickly identified the issue and implemented error handling to avoid crashes when encountering protected system directories. Add User Confirmation for Resource-Intensive Operations Prompt: “Add a confirmation prompt that warns about CPU usage and asks for user confirmation before proceeding.” Result: The tool now includes a warning prompt that ensures users are aware of potential performance impacts during scanning. Improve Documentation Prompt: “Improve documentation with comprehensive docstrings for all functions and classes.” Result: Enhanced readability and maintainability through clear, detailed docstrings and usage explanations. Introduce a Clear Banner Message Prompt: “Add a banner with clear information about the tool's purpose and usage instructions.” Result: Users now see a helpful banner on launch that explains what the tool does and how to use it effectively. Warn About CPU Usage Prompt: “Add a warning about CPU usage during parallel scanning and recommend running during non-peak hours.” Result: A precautionary message was added to guide users on optimal usage timing and prevent performance bottlenecks.

May 9, 2025 - 07:12

AWS EFS Analyzer: Optimizing Storage Costs with Amazon Q Developer

This is a submission for the Amazon Q Developer "Quack The Code" Challenge: Crushing the Command Line

What I Built

I created the AWS EFS Analyzer, a command-line tool designed to help AWS users optimize their Elastic File System (EFS) storage costs.

The tool identifies cost-saving opportunities by analyzing file access patterns and recommending transitions to more cost-effective storage tiers—like moving infrequently accessed files from Standard to Infrequent Access or Archive. AWS estimates that customers can save up to 92% this way.

Manual identification is tedious and error-prone—EFS Analyzer automates the process, delivering actionable insights with ease.

Development Prompt

Python Script for EFS Storage Optimization Analysis

Create a Python script that analyzes an Amazon EFS (Elastic File System) mount point to identify cost optimization opportunities. The script should:

Scan an EFS mount point recursively, with support for parallel processing to handle large file systems efficiently

Categorize files based on last access time (7, 14, 30, 60, 90 days, 1 year, 2 years, and older)

Calculate total storage size for each access time category

Estimate storage costs across different EFS tiers (Standard, Infrequent Access, Archive)

Generate detailed reports in both HTML and plain text formats showing:

File access statistics by time category

Current storage costs (assuming all in Standard tier)

Potential optimized costs using appropriate tiers based on access patterns

Projected monthly savings

Recommendations for tier transitions

Key Requirements:
Handle large file systems efficiently using parallel processing

Provide real-time progress tracking with completion percentage and ETA

Automatically exclude system directories (/proc, /sys, /dev, etc.) to prevent infinite recursion

Detect and avoid symbolic link loops

Support command-line options for:

Specifying the EFS mount point

Setting parallel processing degree (default: number of CPU cores)

Excluding specific directories

Setting maximum scan depth

Controlling whether to follow symbolic links

Redirecting warnings and errors to a log file

Display a clean progress bar that shows:

Percentage completion

Number of directories processed

Number of files scanned

Estimated time remaining

Generate comprehensive reports that include:

File access statistics by category

Storage size distribution

Current vs. optimized cost analysis

Tier distribution recommendations

Potential monthly savings

Iterative Improvements with Amazon Q Developer

Throughout the development process of the AWS EFS Analyzer, I worked iteratively with Amazon Q Developer to refine and enhance the tool. Each step built upon the last, resulting in a more robust and user-friendly solution.

Key Iterative Enhancements

Here are some examples of follow-up prompts that guided the improvement process:

Handle System Directory Permissions
- Prompt: “Fix the permission errors when scanning system directories like /proc.”
- Result: Amazon Q quickly identified the issue and implemented error handling to avoid crashes when encountering protected system directories.
Add User Confirmation for Resource-Intensive Operations
- Prompt: “Add a confirmation prompt that warns about CPU usage and asks for user confirmation before proceeding.”
- Result: The tool now includes a warning prompt that ensures users are aware of potential performance impacts during scanning.
Improve Documentation
- Prompt: “Improve documentation with comprehensive docstrings for all functions and classes.”
- Result: Enhanced readability and maintainability through clear, detailed docstrings and usage explanations.
Introduce a Clear Banner Message
- Prompt: “Add a banner with clear information about the tool's purpose and usage instructions.”
- Result: Users now see a helpful banner on launch that explains what the tool does and how to use it effectively.
Warn About CPU Usage
- Prompt: “Add a warning about CPU usage during parallel scanning and recommend running during non-peak hours.”
- Result: A precautionary message was added to guide users on optimal usage timing and prevent performance bottlenecks.