How we improved the observability of a Go project
- Add comprehensive info-level logging coverage to make each step of processing more observable. Structure logging with log levels using the logrus package to eliminate noise and improve searchability within logs, and impose consistency.
- Create a structured error type to include a generic map of data. This will help you diagnose what exactly went wrong at the top level when error handling, indicating which errors are fatal and which indicate you should retry processing.
- Use Go’s context package, threading it through all request handlers and message processors for greater control over your services, making for a more reliable shutdown in case of new deployment or scaling events.
- Combine the detailed errors and logging into a common abstraction to reduce code repetition. You can add functions for creating errors and logs with basically the same interface, in turn using the context data.
- Add service-level indicators (SLIs) for high-level functions in asynchronous processing to reveal end-to-end latency of different kinds of processing to end-users.
Full post here, 6 mins read