Continuous testing - creating a testable CI/CD pipeline

For continuous testing, focus on confidence, implementation, maintainability, monitoring and speed (CIMMS):

  1. For greater confidence, pair testers with developers as they write code to review unit tests for coverage and to add service tests for business logic and error handling.
  2. To implement, use tools that support rapid feedback from fast running of repeatable tests. For service-level tests, inject specific responses/inputs into Docker containers or pass stubbed responses from integration points. For integration tests, run both services in paired Docker containers within the same network. Limit full-environment tests.
  3. Ensure tests are maintained and up to date. Create tests with human-readable logging, meaningful naming and commented descriptions.
  4. To monitor, use testing tools that integrate into CI/CD pipeline tools to make failures/successes visible and even send emails out automatically. In production, labeling logs to trace a user’s path and capturing system details of the user environment allows easier debugging.
  5. For speed, keep the test suite minimal. Let each test focus on only one thing and split tests to run in parallel if need be. Segregate to test only for changed areas and ignore those with no cross-dependencies.
  • Avoid automating everything. Run manual exploratory tests at each stage to understand new behaviours and determine which of those need automated tests.
  • When pushing to a new environment, test environmental rollback. Reversing changes should not impact users or affect data integrity. Test the rollout process itself for production and run smoke tests. Continue to monitor by running known error conditions and ensure monitoring captures those with sufficient information for easy debugging.

Full post here, 7 mins read