My Journey in AWS Cloud..
When you work in Cloud technologies and help your clients/product owners to deliver their business features/ products, with the time we are living in, specially during pandemic, it's super critical that you deliver in time with well architected design that includes best operationalization, security, reliability, performance/scale, lower cost etc . Key things not to forget in your design and not limited to:
- infrastructure with best security
- monitoring of your product
- performance and scalability
- disaster recovery
If you look at the areas that development team needs to cover, on top of business feature request, one of the most under rated non functional area is Infrastructure management.
I started my could development journey around 2014 time frame, and mainly in AWS cloud. First thing that I learnt on aws cloud was, how easy it was to set up any linux machine a.k.a EC2 instance. That feeling didn't stay for more then 3-4 weeks and soon I realize, there is more to cloud offering and specially when you deliver working solution. The term the full stack developer has a depth to it and we always tend to miss infrastructure management as one of the key skills part of full stack developer.
Back in 2014, when I designed and implemented my first solution on AWS, infrastructure point of view I realized, I can not be create and maintain infrastructure via console that AWS provides. I learned about AWS development tool called "cloud formation" that I can use to deliver my infrastructure.
Now when I was learning AWS cloud, it was overwhelming to understand how you develop in cloud vs your data center, and I started valuing the work done by CM ( configuration management ) team even more.
In my company, I have good friends in CM group and they started showing me what all challenges they faced when they deliver a stable environments. This was early in 2015 time.
In my mind, I started mapping them back to cloud delivery. One of my good friend to whom I have given really hard time by asking number of questions around governance of deployment was nice to answer all questions. Sometime he said, don't shoot the messenger. Means it's over kill, we don't need to have that :) David Platt, I miss you my friend. At the same time he explained what all mistakes we as a developer make and how they put controls in place so they can detect and notify us about all the crazy thing we do as a developer.
Back to cloud journey, I came up with rule for my cloud development in 2016 that I will not deploy any service or I say object on AWS with the help of aws console. May be when I am learning or doing quick check things, it's okay. For any real work I will use Cloud formation (a.k.a CFN).
If you are in AWS cloud development, you will say what's big deal in that, it's one of the great development tool AWS has, to manage your infrastructure. I agree at the same time, when you start deploying object with CFN and create crazy infrastructure, you will feel the pain.
CFN is declarative way of creating your infrastructure. It is either JSON based or YAML based, both are not easy to master personally for me. I spend almost two and half year and in server-less paradigm, server-less framework helped a lot, at the same time when you have multiple CFN that shares objects part of your solution, it becomes more and more challenging to write those complex CFN.
Moving towards topic of this Blog...
Infrastructure as a code with CDK
During all these time, I was progressing in my professional path, and time became most valuable asset for me, I needed to move smartly.
In 2019, when I had to do crazy POC for some of the proofing work on my design and that involved number of AWS objects DynamoDB tables, KMS, S3, SQS (simple and FIFO with DL queue), lambdas, SNS, SNS subscriptions and SQS connecting to lambda. One thing was clear, writing CFN for this will be night mare, specially some of the objects are referred conditionally from other stacks (yes you can use import values from other stack)
Glimpse of design that needed to be POC for many reason:
Now take this diagram and create infrastructure for all components with the help of CFN, you will easily spend a week to get it right. And if you get lost in yaml definition, good luck for few more days
So here is my honest take on CFN:
1. CNF is good, at the same time having this kind of structure POCed
2. Even developed and maintenance will require crazy accuracy with YAML syntax and great depth of knowledge and remember the time part (which is bit difficult to achieve).
3. Now add Cloud best practices on top of your CFN, and how to manage that in your project, so your team is making no mistakes, specially it's critical when it comes to security best practices.
The great thing happened early 2019, AWS released an open source project called CDK cloud development kit. One of my colleague, told me about CDK in one of the blueprint meetings. It was in beta mode I looked at it and ignored. Having time crunch I decided to look at that again, and here comes best learning, cdk is GA and had good amount of services covered, specially all you see in my design here.
I gave it a try and the same infrastructure you see in diagram, I was able to create that in 4 hours.
More info on CDK To Be Added....
CDK Advantage
- Use logic (if statements, for-loops, etc) when defining your infrastructure
- Use object-oriented techniques to create a model of your system
- Define high level abstractions, share them, and publish them to your team, company, or community
- Organize your project into logical modules
- Share and reuse your infrastructure as a library
- Testing your infrastructure code using industry-standard protocols
- Use your existing code review workflow for Infrastructure
- Code completion within your IDE
- Declarative is good, Programmatic is great, fits well with Developer DNA
Declarative vs Programmatic
- Unit Test your Infrastructure
Object Oriented Framework
Multi Language Support
- Typescript, Javascript, python, java, c#
- JSii + TS compiler
- JavaScript Interoperability interface
