The Art Of Chaos Engineering : LFX 2025 Experience

I was selected for the LFX Mentorship Program 2025 under LitmusChaos where my project was CNCF - LitmusChaos: CI/CD Integration, SDK Development & Chaos-CI-Lib Revamp During my mentorship, I submitted a total of 7 PR's to 4 repositories of LitmusChaos with approximately 14k lines of code. A detailed gist containing all of my contribution PR's can be found here. Introduction I am Akash Singh, a third year engineering student and Open Source Contributor from Bangalore. Here is my LinkedIn, GitHub and Twitter I go by the name SkySingh04 online. Application Process The LFX Mentorship application journey began in December 2024 when applications opened for the 2025 Term 1 (March-May) cohort. The process was both exciting and nerve-wracking as I navigated through multiple project applications. Projects I Applied To Having prior open-source contributions gave me the confidence to apply to three CNCF projects: LitmusChaos: CI/CD Integration, SDK Development & Chaos-CI-Lib Revamp - This project aimed to revolutionize the CI/CD experience for chaos engineering by developing a dedicated SDK and modernizing the Chaos-CI-Lib for Litmus 3.x compatibility. KCL: KPM & LSP Integrated - Focused on strengthening the integration between Language Server Protocol (LSP) and KCL Package Manager (KPM) for better IDE experience. Karmada: Self-Signed Certificate Content Standardization - Aimed at enhancing Karmada's certificate system compliance by ensuring each component has distinct certificates. Application Requirements The LFX portal required two components for the project's I appiled to: Resume: Highlighting relevant technical skills and open-source contributions Cover Letter: Explaining my motivation and how my skills aligned with each project Note that these vary from org to org. Many orgs also conduct an interview round / give additional onboarding tasks that you have to complete. The Selection After weeks of anticipation, I received the selection email in late February 2025! The mentors - Shubham Chaudhary and [Vedant https://github.com/Jonsy13) from LitmusChaos - had evaluated all applications and selected me for the chaos engineering project. What made my application stand out was: My existing contributions to Karmada and KCL, demonstrating familiarity with CNCF projects Strong Go programming skills essential for SDK development Understanding of Kubernetes and CI/CD pipelines Clear communication in my cover letter about how I could contribute to the project's goals What is Chaos Engineering? This mentorship was my first deep dive into Chaos Engineering and honestly I find it such a cool concept in Software Engineering and Testing in general. The way I explain it to my juniors at Point Blank is its' the process of experimenting on a distributed system to build confidence in the system's capability to withstand ✨circus✨ conditions in production. It's about intentionally injecting failures into your systems to discover weaknesses before they impact your customers and you get a call at 3am. It is as simple as a fire drill for your infrastructure - you don't wait for an actual fire to test your emergency procedures. Similarly, chaos engineering helps you proactively identify and fix potential issues (what I refer to as ✨circus✨ conditions). Me teaching my juniors at Point Blank about Chaos Engineering // Detect dark theme var iframe = document.getElementById('tweet-1928079304451313876-603'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1928079304451313876&theme=dark" } Why LitmusChaos? LitmusChaos is a CNCF incubating project that makes chaos engineering easy for developers and SREs. What makes it special: Cloud-Native: Built specifically for Kubernetes environments Extensible: Large library of pre-built experiments GitOps Friendly: Declarative chaos workflows Enterprise Ready: RBAC, audit logs, and multi-tenancy support When I first encountered LitmusChaos, I was fascinated by how it democratizes chaos engineering, making it accessible to teams of all sizes. My Project: Modernizing LitmusChaos from 1.0 to 3.0 Approach My mentorship project focused on three critical areas: 1. SDK Development I built a comprehensive Go SDK from scratch that provides a programmatic interface to LitmusChaos. This SDK enables developers to: Authenticate and manage projects Create and execute chaos experiments Manage infrastructure and environments Monitor experiment status in real-time 2. CI/CD Integration I revamped the Chaos-CI-Lib to align with Litmus 3.0, transitioning from direct Kubernetes API calls to a cleaner SDK-based approach. This included: Refactoring the entire codebase from 1.0 approach to 3.0 approach that uses our SDK. Removing direct invocations of K8s API Calls with our SDK approach. R

Jun 1, 2025 - 13:10

The Art Of Chaos Engineering : LFX 2025 Experience

I was selected for the LFX Mentorship Program 2025 under LitmusChaos where my project was CNCF - LitmusChaos: CI/CD Integration, SDK Development & Chaos-CI-Lib Revamp

During my mentorship, I submitted a total of 7 PR's to 4 repositories of LitmusChaos with approximately 14k lines of code. A detailed gist containing all of my contribution PR's can be found here.

Introduction

I am Akash Singh, a third year engineering student and Open Source Contributor from Bangalore.
Here is my LinkedIn, GitHub and Twitter

I go by the name SkySingh04 online.

Application Process

The LFX Mentorship application journey began in December 2024 when applications opened for the 2025 Term 1 (March-May) cohort. The process was both exciting and nerve-wracking as I navigated through multiple project applications.

Projects I Applied To

Having prior open-source contributions gave me the confidence to apply to three CNCF projects:

LitmusChaos: CI/CD Integration, SDK Development & Chaos-CI-Lib Revamp - This project aimed to revolutionize the CI/CD experience for chaos engineering by developing a dedicated SDK and modernizing the Chaos-CI-Lib for Litmus 3.x compatibility.
KCL: KPM & LSP Integrated - Focused on strengthening the integration between Language Server Protocol (LSP) and KCL Package Manager (KPM) for better IDE experience.
Karmada: Self-Signed Certificate Content Standardization - Aimed at enhancing Karmada's certificate system compliance by ensuring each component has distinct certificates.

Application Requirements

The LFX portal required two components for the project's I appiled to:

Resume: Highlighting relevant technical skills and open-source contributions
Cover Letter: Explaining my motivation and how my skills aligned with each project

Note that these vary from org to org. Many orgs also conduct an interview round / give additional onboarding tasks that you have to complete.

The Selection

After weeks of anticipation, I received the selection email in late February 2025! The mentors - Shubham Chaudhary and [Vedant https://github.com/Jonsy13) from LitmusChaos - had evaluated all applications and selected me for the chaos engineering project.

What made my application stand out was:

My existing contributions to Karmada and KCL, demonstrating familiarity with CNCF projects
Strong Go programming skills essential for SDK development
Understanding of Kubernetes and CI/CD pipelines
Clear communication in my cover letter about how I could contribute to the project's goals

What is Chaos Engineering?

This mentorship was my first deep dive into Chaos Engineering and honestly I find it such a cool concept in Software Engineering and Testing in general. The way I explain it to my juniors at Point Blank is its' the process of experimenting on a distributed system to build confidence in the system's capability to withstand ✨circus✨ conditions in production. It's about intentionally injecting failures into your systems to discover weaknesses before they impact your customers and you get a call at 3am.

It is as simple as a fire drill for your infrastructure - you don't wait for an actual fire to test your emergency procedures. Similarly, chaos engineering helps you proactively identify and fix potential issues (what I refer to as ✨circus✨ conditions).

Me teaching my juniors at Point Blank about Chaos Engineering
// Detect dark theme var iframe = document.getElementById('tweet-1928079304451313876-603'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1928079304451313876&theme=dark" }

Why LitmusChaos?

LitmusChaos is a CNCF incubating project that makes chaos engineering easy for developers and SREs. What makes it special:

Cloud-Native: Built specifically for Kubernetes environments
Extensible: Large library of pre-built experiments
GitOps Friendly: Declarative chaos workflows
Enterprise Ready: RBAC, audit logs, and multi-tenancy support

When I first encountered LitmusChaos, I was fascinated by how it democratizes chaos engineering, making it accessible to teams of all sizes.

My Project: Modernizing LitmusChaos from 1.0 to 3.0 Approach

My mentorship project focused on three critical areas:

1. SDK Development

I built a comprehensive Go SDK from scratch that provides a programmatic interface to LitmusChaos. This SDK enables developers to:

Authenticate and manage projects
Create and execute chaos experiments
Manage infrastructure and environments
Monitor experiment status in real-time

2. CI/CD Integration

I revamped the Chaos-CI-Lib to align with Litmus 3.0, transitioning from direct Kubernetes API calls to a cleaner SDK-based approach. This included:

Refactoring the entire codebase from 1.0 approach to 3.0 approach that uses our SDK.
Removing direct invocations of K8s API Calls with our SDK approach.
Running and testing over 13 experiments end to end with the refactoring.

3. Template Optimization

Updated GitHub Actions and GitLab CI templates to work seamlessly with Litmus 3.0, adding:

Enhanced configuration options
Better infrastructure management
Comprehensive probe configurations

My Experience: From Zero to Chaos Engineer

The Learning Curve

Coming into this project, I had experience with Go and Kubernetes, but chaos engineering was new to me. The first few weeks were intense - understanding GraphQL APIs, diving deep into Kubernetes operators, and grasping the architecture of a distributed chaos engineering platform.

Key Achievements

Built a production-ready SDK: The litmus-go-sdk now serves as the foundation for all programmatic interactions with LitmusChaos
Refactored and Rewrote Chaos-Ci-Lib: Rewrote and released the new version of chaos-ci-lib with our Litmus 3.0 approach using our fancy new SDK.
Modernized CI/CD workflows: Reduced setup complexity for users integrating chaos testing into their pipelines on both Github Actions and Gitlab Pipelines.

Gratitude

This incredible journey wouldn't have been possible without:

Shubham Chaudhary (@ispeakc0de) - For his patient guidance and code reviews that taught me industry best practices
Vedant Shrotria (@Jonsy13) - For helping me understand the deeper architecture of LitmusChaos
Sayan Mondal (@S-ayanide) - For coordinating the mentorship program and ensuring smooth progress

Special thanks to the entire LitmusChaos community for being welcoming and supportive throughout my journey.

Conclusion

The LFX mentorship program has been the most rewarding experience of my year. It has transformed me from a beginner in chaos engineering to a confident contributor capable of making meaningful improvements. This experience enhanced my technical understanding of chaos engineering and helped me get insights into my code and infrastructure. I have learned that anything is achievable with the right amount of energy and time.

My journey with LitmusChaos doesn't end here - I'm committed to continuing my contributions to the project. I plan to maintain the SDK, help onboard new contributors, work on new features, and contribute to the ecosystem's growth. The mentorship might be ending, but my involvement with the LitmusChaos community is just beginning!

Don't hesitate to apply for this mentorship for the next quarter - it could be the start of your own incredible open-source journey!