Should we test AWS CDK code?

AWS CDK is a framework for defining cloud infrastructure in code. CDK allows defining CloudFormation templates using convenient programming concepts. We instantiate classes provided by CDK and follow all its requirements, and CDK builds the CloudFormation template for us. With CDK, you cannot do more than with vanilla CloudFormation, but the former allows enjoying the conciseness and power of programming technics.

CDK is written in TypeScript. Therefore, I would recommend implementing infrastructure code in TypeScript if you are concerned about testing.

Out of the box, CDK provides three classes of tests: snapshot tests, tests based on fine-grained assertions and validation tests. I recommend checking the AWS blog article for an intro on CDK testing.

When and why do we need any of these tests?

What is the difference between running a snapshot test and running cdk diff or cdk deploy --no-execute commands? Snapshot test and cdk diff command compares the saved CloudFormation template with a template generated by the application. When does it make sense to reach the “current” template produced by the program with the “previous” template saved in the snapshot?

I can think about two use cases when this might be helpful.

First, when you do not apply the changes immediately but release your infrastructure changes on a schedule. You might have tested changes that have affected the saved snapshot and not tested new changes in the code. When running a snapshot test, you check the latest changes on top of the changes tested and approved before.

Even in this case, you have to deploy your changes on the test cloud. Because the fact that the program can generate the template does not guarantee that CloudFormation can apply the changes. Changes might require resource recreation, and this might be a disaster if they go unnoticed. You check with the manual for every item you will update or try it on the test cloud.

Second, when you have continuous integration tools and want to make sure unchecked changes are not applied to the production cloud by mistake. In this case, testing against saved snapshots before deployment might be a reasonable precaution.

In other scenarios, it might be worth thinking if the work needed to keep your snapshot updated is worth the benefits of running snapshot tests. For example, it might appear that the only thing you do with snap-shot tests is updating them before every commit, almost without thinking.

Broadly, a CDK program might be a library of patterns (reusable high-level components) or a setup for your project cloud (collection of patterns and components configured for your unique project requirements). In the former case, unit tests with fine-grained assertions are must-haves. In the latter case, this might be overkill. So why do we want to double test thoroughly tested patterns? If we have complicated business logic in our program, then we should test that logic.

The same goes for validation tests. In the former case, they might be of tremendous help. In the latter case, this might be overkill and extra work, which would never be helpful.

The good thing about tests is that they might help avoid making mistakes, help to write new code, and protect from regressions. The bad thing is that they slow development and, in general, make the program harder to change. Because a one-line change in the code might require changing tens of lines of tests and/or test presets. The decision to commit to tests should be thoroughly considered and proved needed.