About Me

My photo
Reston, Virginia, United States
Mr Sheth is a wanna be technical evangelist with more than 17 years experience (# of years is just a number) , he is focused on functional programming and Cloud technologies. Mr Sheth works with a mission driven organization and is focused on helping students and educators to do their day to day work with ease and smartness. Currently he is working with Collegeboard as Cloud Architect with Enrollment Group. He started with core product that delivers Estimated Family Contribution (INASaurs team) and now helping powerfaids group with their cloud move. Recent achievements: - Helping engineer to learn new technologies in his group. Designed CDK workshop for programmatic infrastructure deployment. - Event driven task management system. - Designed passive monitoring system that notifies DevOps team with in 3 secs of event occurrence with complete details about the event. To enable this monitoring, it requires minimal changes for existing component or new components. This also gives complete visibility in deployed serverless stack as well as react/JavaScript based single page app.

Monday, February 8, 2021

Infrastructure as a code with CDK

My Journey in AWS Cloud..


When you work in Cloud space and helping your client/product Owens to deliver their products. With the time we are living in, specially during pandemic, it's super critical that you deliver in time with the features requested. Other things not to forget as true technologies, do not compromise in all non functional areas like:
  • security
  • monitoring your solution
  • performance 
  • disaster recovery
  • and many more...
If you look at the areas that development team needs to cover, on top of feature request, is growing with cloud solutions. One of the most under rated non functional area is Infrastructure management (even I missed that in the list of non functional requirements :) )

I started my could development journey around 2014 time frame, and mainly in AWS cloud. First thing that I learnt on aws cloud was, how easy it was to set up any linux machine. That feeling didn't stay for more then 3-4 weeks  and soon I realize, there is more to cloud offering and specially when you deliver working solution. The full stack development has a depth to it.

Looking from infrastructure point of view, infrastructure can not be created and maintained via console that AWS provides in real projects. Soon I came to know about AWS development tool called "cloud formation" that I can use to deliver my infrastructure. 
Now when I was learning AWS cloud, it was overwhelming to understand how you develop in cloud vs your data center, and I started valuing the work done by CM ( configuration management ) team even more. 

In my company, I have good friends in CM group and they started showing me what all challenges they faced when they deliver a stable environments. This was early in 2015 time.

In my mind, I started mapping them back to cloud delivery. One of my good friend to whom I have given really hard time by asking number of questions around governance of deployment was nice to answer all questions. Sometime he said, don't shoot the messenger. Means it's over kill, we don't need to have that :) David Platt, I miss you my friend. At the same time he explained what all mistakes we as a developer make and how they put controls in place so they can detect and notify us.

Back to cloud journey came up with rule for my cloud development in 2016 that I will not deploy any service or I say object on AWS with the help of aws console. May be when I am learning or doing quick check things, it's okay. For any real work I will use Cloud formation (a.k.a CFN). 

If you are in AWS cloud development, you will say what's big deal in that, it's one of the great development tool AWS has to manage your infrastructure. I agree at the same time ( don't want to use BUT), when you start deploying object with CFN and create crazy infrastructure, you will feel the pain. CFN is declarative way of creating your infrastructure. It is either JSON based or YAML based, both are not easy to master personally for me. I spend almost two and half year and in server-less paradigm, server-less framework helped a lot, at the same time when you have combination of objects part of your solution, it becomes more and more challenging.

Moving towards topic of this Blog... 

Infrastructure as a code with CDK


During all these time, I was progressing in my professional path, and time was becoming a top most consideration for me, I needed to move smartly. 

In 2019, when I had to do crazy POC for some of the proofing work on my design  and that involved number of AWS objects DynamoDB tables, KMS, S3, SQS (simple and FIFO with DL queue), lambdas, SNS, SNS subscriptions and SQS connecting to lambda. One thing was clear, writing CFN for this will be night mare, specially some of the objects are referred conditionally from other stacks (yes you can use import values from other stack)

Glimpse of design that needed to be POCed for many reason:
 



Now take that design and write CFN, you will easily spend a week to get it right. And if you get lost in yaml definition, good luck for few more days 

So here is what I feel, CNF is good, at the same time having this kind of structure POCed or even developed and maintain will require crazy accuracy with YAML syntax and great depth of knowledge and remember the time part.

Now add Cloud best practices on top of your CFN, and how to manage that in your project so your team is making no mistakes (leaks) when it comes to security practices.

The great thing happened early 2019, AWS released an open source project called CDK cloud development kit. One of my colleague, I am not able to find him (sorry) told me about CDK in one of the buleprint meetings. It was in beta mode I looked at it and ignored. Having time crunch I decided to look at that again, and wola, they were GA and had good amount of services covered, specially all you see in my design here. 

I gave it a try and the same infrastructure you see in diagram, I was able to create that in 4 hours. 


More info on CDK To Be Added....


CDK Advantage

  • Use logic (if statements, for-loops, etc) when defining your infrastructure
  • Use object-oriented techniques to create a model of your system
  • Define high level abstractions, share them, and publish them to your team, company, or community
  • Organize your project into logical modules
  • Share and reuse your infrastructure as a library
  • Testing your infrastructure code using industry-standard protocols
  • Use your existing code review workflow for Infrastructure
  • Code completion within your IDE
  • Declarative is good, Programmatic is great, fits well with Developer DNA

Declarative vs Programmatic

  • Unit Test your Infrastructure 

Object Oriented Framework

Multi Language Support

  • Typescript, Javascript, python, java, c#
  • JSii + TS compiler
  • JavaScript Interoperability interface