I learned the following lessons the hard way and I believe -despite being very obvious- most of teams tend to ignore these hints specially teams with low resources / capacity.
1. Always keep an alert on your cloud provider bill.
- You will be one day senior enough that you will be trusted for your team’s / company’s cloud infrastructure bill. Please, don’t forget to put a budget alert.
- Feature is available on azure, AWS and I’m not sure if it’s available for alibaba cloud and IBM cloud but there must be some sort of alternative for these platforms.
2. Always have a test environment that’s completely separate from production and avoid making the production environment / production accounts accessible in development.
Making production environment too accessible for your development team this may risk (top of my head):
- Junk data in production
- Loss of data in production
- Loss of credit
- Going out of API quota.
3. Don’t Exclude your Ops and DevOps from QA steps.
When your backend developers has some green tests locally that doesn’t necessarily mean that you made it. you have to actually get the card done by going to production with the correct config-maps, envs, service accounts and feature-flags and (whatever environment dependent change) that may make this code break) is in place then you may cross your fingers and celebrate.
In DevOps some teams (and I used to be part of these teams) usually tend to celebrate DevOps efforts very early be and exclude it from the traditional QA.
In short, you will regret it and I’m sure because I’ve been there. normal backend/front-end task usually can be recoverable but a back-up script that have an issue is not recoverable and won’t be noticeable until it’s too late.
I’m pretty sure that many of the readers of this small article may have more lessens that they learned the hard way; Please, Don’t hesitate to share them with me in the comments. you may save a life!