The development environment of your application can and probably will be broken and frequently changed and deployed to. However, deploying your application to stage should be as similar as possible to your production deployment. The stage deployment should follow these rules:
- The deployment should be immutable.
- Configuration must be done via code.
- Deployment to production should be possible by just substituting environment variables
- The infrastructures should be similar
- Ideally the stage will have mock user data and/or integration with other systems.
Immutable deployment
By immutable deployment we mean, that code (e.g. terraform or CloudFormation) is used to deploy the infrastructure. All configuration is done automatically during this deployment process and no manual interaction should be done on VMs/containers (e.g. administrator accessing VM to do some manual changes from shell). If system must be patched, the infrastructure is torn down and rebuild with those patches. Of course the update of infrastructure is done by rolling update (i.e. components are update in such fashion that there is no service outage). Some strategies also do blue/green deployments so servers of one color are always active and servers of the other color can be updated.
Configuration via code
To configure deployed servers use tools, that can be run automatically and perform configuration otherwise done by administrators. This way you make sure that every time the VM is deployed, the same configuration is applied. Furthermore, the servers can be configured in parallel. Automating configuration also means, you can narrow who sees secrets and has access to sensitive data. Engineers can develop configuration code without needing access to sensitive secrets (SSH keys, passwords, SSL private keys, etc.).
Variable substitution
When stage and production differ only by deployment variable substitution, you will have more certainty that setup you tested in stage will behave same in the production. This separation is also important from a security perspective. The secrets for production deployment will almost certainly enable anyone that has them to access or change your production environment.
Infrastructure similarity
For small deployments it’s not hard to have stage and production environments be identical. However, for production environments having 100s of servers it wouldn’t make sense to inflate stage environment to the same scale. In this case your stage environment will have to be scaled down representation of the production. This part might seem more like an art than an exact science. Good representation is usually achieved after few iteration and it’s continuous effort.
Mocked user data
To perform E2E tests you will probably need some test data. Since stage and production environments are similar you might be tempted to use data from production. This isn’t recommended as stage environment despite the best effort, might be unsafe and larger number of people might have access to it. By mocking data you will be able to perform E2E tests without fear of real user data leaks in case something goes wrong.