Optimising Cloud Costs and Security: With Scientific Research Platform Using AWS Manager Store

In today’s cloud-driven world, managing sensitive credentials securely while keeping costs in check presents a significant challenge for organisations. Our recent publishers project demonstrates how thoughtful implementation of AWS services can achieve both goals. This article explores how we leveraged AWS Secrets Manager and AWS Systems Manager Parameter Store to securely manage database credentials and API keys while optimising cloud expenditure for our green technology research platform. The Publishers Project: Architecture and Requirements Our publishers project is a comprehensive platform that retrieves scientific articles about green technology from multiple publisher’s researcher using their respective APIs. The workflow follows these steps: Lambda functions query publisher APIs using specific keywords. Retrieved articles are processed and uploaded to Amazon S3 buckets. Article Metadata is inserted into a PostgreSQL database for indexing and research. AWS Step Functions orchestrate the entire workflow to avoid Lambda timeouts This data pipeline requires secure management of two types of sensitive information: Database credentials for PostgreSQL Multiple publisher-specific API keys Understanding AWS Secrets Management Options in Depth AWS offers two primary services for managing sensitive information, each with distinct capabilities and cost profiles. Understanding their similarities and differences is crucial for optimal implementation. AWS Secrets Manager: AWS Secrets Manager helps you manage, retrieve, and rotate database credentials, application credentials, OAuth tokens, API keys, and other secrets throughout their lifecycles. Many AWS services store and use secrets in Secrets Manager. Secrets Manager helps you improve your security posture, because you no longer need hard-coded credentials in application source code. Storing the credentials in Secrets Manager helps avoid possible compromise by anyone who can inspect your application or the components. You replace hard-coded credentials with a runtime call to the Secrets Manager service to retrieve credentials dynamically when you need them. AWS Systems Manager: AWS Systems Manager helps you centrally view, manage, and operate nodes at scale in AWS, on-premises, and multi-cloud environments. With the launch of a unified console experience, Systems Manager consolidates various tools to help you complete common node tasks across AWS accounts and Regions. Parameter Store: Parameter Store, a tool in AWS Systems Manager, provides secure, hierarchical storage for configuration data management and secrets management. You can store data such as passwords, API keys, database strings, Amazon Machine Image (AMI) IDs, and license codes as parameter values. You can store values as plain text or encrypted data. Similarities Between Secrets Manager and Parameter Store Encryption Capabilities Both services leverage AWS KMS to encrypt values, providing an additional security layer beyond IAM permissions. This encryption is essential for compliance requirements and protecting sensitive information. Key/Value Store Structure Both implement a key/value store model, allowing values to be organised with prefix schemas like application/environment/parametername. In our research platform, we organise our parameters logically: /publisher/green-technology/api-key /publisher/climate-chnage/api-key /publisher/global-warming/api-key /publisher/Sustainable-development/api-key /publisher/clean-energy/api-key CloudFormation Integration Both services integrate with CloudFormation, enabling infrastructure-as-code without embedding secrets in templates. This allows our CI/CD pipeline to reference secure values rather than hardcoding them. Versioning Support Both services maintain version history of secrets, allowing you to track changes and restore previous values if needed. This provides an audit trail of credential modifications. AWS Secrets Manager AWS Secrets Manager is a specialised service designed specifically for high-security storage of sensitive credentials with advanced features: Automatic secret rotation: Automatically updates credentials on a schedule. Cross-account access: Shares secrets across multiple AWS accounts. Fine-grained permissions: Controls access through IAM policies. Integration with AWS services: Seamlessly works with RDS, Redshift, and other services. Higher cost: $0.40 per secret per month + $0.05 per 10,000 API calls. Larger storage capacity: Can store secrets up to 10KB in size. Multiple region replication: Enables disaster recovery and multi-region applications. Service limits: Supports up to 500,000 secrets per region per account. AWS Systems Manager Parameter Store Parameter Store provides hierarchical configuration data management with tiered pricing: Free standard tier: Stores up to 10,000 parameters at no cost. Hierarchical organisation: Uses p

Apr 26, 2025 - 08:11

Optimising Cloud Costs and Security: With Scientific Research Platform Using AWS Manager Store

In today’s cloud-driven world, managing sensitive credentials securely while keeping costs in check presents a significant challenge for organisations. Our recent publishers project demonstrates how thoughtful implementation of AWS services can achieve both goals. This article explores how we leveraged AWS Secrets Manager and AWS Systems Manager Parameter Store to securely manage database credentials and API keys while optimising cloud expenditure for our green technology research platform.

The Publishers Project: Architecture and Requirements

Our publishers project is a comprehensive platform that retrieves scientific articles about green technology from multiple publisher’s researcher using their respective APIs. The workflow follows these steps:

Lambda functions query publisher APIs using specific keywords.
Retrieved articles are processed and uploaded to Amazon S3 buckets.
Article Metadata is inserted into a PostgreSQL database for indexing and research.
AWS Step Functions orchestrate the entire workflow to avoid Lambda timeouts

This data pipeline requires secure management of two types of sensitive information:

Database credentials for PostgreSQL
Multiple publisher-specific API keys

Understanding AWS Secrets Management Options in Depth

AWS offers two primary services for managing sensitive information, each with distinct capabilities and cost profiles. Understanding their similarities and differences is crucial for optimal implementation.

AWS Secrets Manager: AWS Secrets Manager helps you manage, retrieve, and rotate database credentials, application credentials, OAuth tokens, API keys, and other secrets throughout their lifecycles. Many AWS services store and use secrets in Secrets Manager.

Secrets Manager helps you improve your security posture, because you no longer need hard-coded credentials in application source code. Storing the credentials in Secrets Manager helps avoid possible compromise by anyone who can inspect your application or the components. You replace hard-coded credentials with a runtime call to the Secrets Manager service to retrieve credentials dynamically when you need them.

AWS Systems Manager: AWS Systems Manager helps you centrally view, manage, and operate nodes at scale in AWS, on-premises, and multi-cloud environments. With the launch of a unified console experience, Systems Manager consolidates various tools to help you complete common node tasks across AWS accounts and Regions.

Parameter Store: Parameter Store, a tool in AWS Systems Manager, provides secure, hierarchical storage for configuration data management and secrets management. You can store data such as passwords, API keys, database strings, Amazon Machine Image (AMI) IDs, and license codes as parameter values. You can store values as plain text or encrypted data.

Similarities Between Secrets Manager and Parameter Store

Encryption Capabilities
Both services leverage AWS KMS to encrypt values, providing an additional security layer beyond IAM permissions. This encryption is essential for compliance requirements and protecting sensitive information.

Key/Value Store Structure
Both implement a key/value store model, allowing values to be organised with prefix schemas like application/environment/parametername. In our research platform, we organise our parameters logically:

/publisher/green-technology/api-key
/publisher/climate-chnage/api-key
/publisher/global-warming/api-key
/publisher/Sustainable-development/api-key
/publisher/clean-energy/api-key

CloudFormation Integration
Both services integrate with CloudFormation, enabling infrastructure-as-code without embedding secrets in templates. This allows our CI/CD pipeline to reference secure values rather than hardcoding them.

Versioning Support
Both services maintain version history of secrets, allowing you to track changes and restore previous values if needed. This provides an audit trail of credential modifications.

AWS Secrets Manager

AWS Secrets Manager is a specialised service designed specifically for high-security storage of sensitive credentials with advanced features:

Automatic secret rotation: Automatically updates credentials on a schedule.
Cross-account access: Shares secrets across multiple AWS accounts.
Fine-grained permissions: Controls access through IAM policies.
Integration with AWS services: Seamlessly works with RDS, Redshift, and other services.
Higher cost: $0.40 per secret per month + $0.05 per 10,000 API calls.
Larger storage capacity: Can store secrets up to 10KB in size.
Multiple region replication: Enables disaster recovery and multi-region applications.
Service limits: Supports up to 500,000 secrets per region per account.

AWS Systems Manager Parameter Store

Parameter Store provides hierarchical configuration data management with tiered pricing:

Free standard tier: Stores up to 10,000 parameters at no cost.
Hierarchical organisation: Uses path-like structures for better organisation.
Optional encryption: Can use AWS KMS for sensitive parameters.
No automatic rotation: Requires manual management of parameter updates.
Lower cost: Free for standard parameters, $0.05 per 10,000 API calls for advanced tier.
Storage limitations: 4KB for standard parameters, 8KB for advanced parameters.
No built-in cross-region replication: Requires custom implementation.
Service limits: 10,000 standard parameters per region per account.

Our Cost-Optimised Implementation Strategy

After analysing our requirements, we implemented a hybrid approach using both services strategically:

Database Credentials in AWS Secrets Manager
We store PostgreSQL database credentials in AWS Secrets Manager because:

Database credential rotation: Secrets Manager can automatically rotate database credentials, reducing security risks.
Single point of access: Our application only retrieves database credentials during initialisation.
IAM integration: Secrets Manager integrates directly with our RDS instance.
Critical security: Database access requires the highest level of protection.

The implementation in our Lambda functions uses the following approach:

def get_db_credentials():
    secret_name = "publishers_db_credentials"
    region_name = "eu-west-2"
    session = boto3.session.Session()
    client = session.client(service_name='secretsmanager', region_name=region_name)
    try:
        get_secret_value_response = client.get_secret_value(SecretId=secret_name)
        secret = json.loads(get_secret_value_response['SecretString'])
        return {
            "host": secret["postgresql"]["host"],
            "port": secret["postgresql"]["port"],
            "dbname": secret["postgresql"]["dbname"],
            "user": secret["postgresql"]["user"],
            "password": secret["postgresql"]["password"]
        }
    except ClientError as e:
        print(f"Error retrieving secret: {e}")

        raise

API Keys in AWS Systems Manager Parameter Store

We store the publisher API keys in Parameter Store because:

Higher retrieval frequency: Our Lambda functions retrieve API keys for every article request.
Multiple distinct keys: We manage five different publisher API keys.
Hierarchical organisation: We organise keys by publisher in a structured hierarchy.
Cost efficiency: Parameter Store’s free tier significantly reduces operational costs.
Adequate security: Parameter Store provides sufficient encryption for API keys.

Our implementation uses a clear hierarchical structure:

def get_springer_api_key():
    ssm = boto3.client('ssm')
    try:
        response = ssm.get_parameter(
            Name='publisher/green-technology/api-key',
            WithDecryption=True
        )
        return response['Parameter']['Value']
    except ClientError as e:
        raise Exception(f"Failed to retrieve Climate Change API key: {str(e)}")

Cost Analysis: The Financial Impact of Our Strategy
Let’s analyse the cost implications of our approach:

Scenario 1: Everything in Secrets Manager

If we stored all credentials (1 database secret + 5 API keys) in AWS Secrets Manager:

6 secrets × $0.40 per month = $2.40 per month.
Assuming 1 million API calls per month: (1,000,000 ÷ 10,000) × $0.05 = $5.00 per month.
Total: $7.40 per month.

Scenario 2: Our Hybrid Approach

Database credentials in Secrets Manager: 1 secret × $0.40 = $0.40 per month.
API keys in Parameter Store: Free (within standard tier).
API calls to Secrets Manager (less frequent): (10,000 ÷ 10,000) × $0.05 = $0.05 per month.
API calls to Parameter Store (more frequent): Free (standard tier).
Total: $0.45 per month.

Annual savings: $83.40

This cost optimisation becomes even more significant as the number of API keys increases or when API call volume grows.

Choosing Between Secrets Manager and Parameter Store: Decision Criteria

When deciding which service to use for your specific use case, consider these criteria:

Choose Secrets Manager if:

You need automatic rotation of credentials.
Cross-account access is required.
You’re using AWS services with built-in Secrets Manager integration.
You need multi-region replication.
Compliance requirements mandate advanced security features.
You need to store secrets larger than 4KB.

Choose Parameter Store if:

Cost optimisation is a priority.
You’re storing a mix of configuration data and credentials.
You don’t require automatic rotation.
You’re working within a single AWS account.
Your secrets are smaller than 4KB (standard tier).
You have a large number of parameters to store.

Alternative Solutions

While AWS provides robust solutions for secrets management, there are alternatives worth considering:

Hashicorp Vault: An open-source solution offering advanced secret management with pluggable authentication.
Azure Key Vault: Microsoft’s equivalent service for managing keys, secrets, and certificates.
Azure App Configuration: Similar to Parameter Store but for Azure environments.

For our scientific research platform, AWS services provided the best integration with our existing infrastructure.

AWS Best Practices for Credential Management
Our implementation follows AWS best practices for credential management:

1. Separation of Sensitive Information from Application Code
By storing credentials in dedicated services, we’ve completely separated sensitive information from our application code. This improves security and simplifies credential rotation without code changes.

2. Appropriate Service Selection
We’ve followed AWS’s guidance by selecting:

Secrets Manager for critical credentials requiring rotation (database credentials).
Parameter Store for configuration data and less critical secrets (API keys).

3. Hierarchical Organisation
We’ve implemented a structured parameter hierarchy in Parameter Store:

/publisher/green-technology/api-key
/publisher/climate-change/api-key
/publisher/global-warming/api-key
/publisher/Sustainable-development/api-key
/publisher/clean-energy/api-key

This organisation improves manageability and allows for more granular IAM permissions.

4. Least Privilege Access
Our Lambda functions have IAM roles with permissions only to:

Access specific secrets or parameters.
Access only the necessary actions (GetSecretValue, GetParameter).
Operate only in the required region.

5. Encryption for All Sensitive Data
We ensure that all sensitive data is encrypted:

Secrets Manager automatically encrypts all secrets with AWS KMS.
Parameter Store API keys use the SecureString type with KMS encryption.

Conclusion: Balancing Security and Cost Efficiency

Our green technology research platform demonstrates that with thoughtful service selection, organisations can achieve both robust security and cost efficiency. By leveraging AWS Secrets Manager for critical database credentials and AWS Systems Manager Parameter Store for API keys, we’ve implemented a solution that:

Follows AWS security best practices.
Reduces monthly operational costs by over 90%.
Maintains appropriate security levels for different types of credentials.
Scales efficiently as our API usage grows.

This hybrid approach represents an optimal balance between security and cost-effectiveness, proving that with careful planning, organisations don’t need to compromise on either factor. For any cloud application managing multiple types of credentials, we recommend conducting a similar analysis to determine the most appropriate credential management strategy based on security requirements and cost constraints.

As cloud architectures become increasingly complex, strategic service selection becomes more critical for maintaining both security and cost efficiency. Our scientific research platform serves as a practical example of how this balance can be achieved, demonstrating that well-architected applications can be both secure and cost-effective.

Thank you for following along. I hope you find this valuable. Build On!

If you found this helpful, don’t forget to give this article a like and follow me for more tips and insights! Your support means a lot.