Distributed Logging Architecture in the Container Era
- Logging is a cross-cutting concern in any application. For distributed application, it's better to have shared logging technology across all the services. Log aggregators are a solution for polyglot systems as they have connectors to most languages.
- Logging infrastructure must be searchable. What's the point of logging everything without being able to answer queries like "Which service throws the most errors?"
- Using a single correlation ID across services allows you to filter log messages from all the sources. This makes debugging a lot easier if you can view all the code paths that a request touched.
- Include a lot more context in each log message. Data such as username, service name, timestamp etc are very handy to scan through logs quickly.
- Network failure is an inevitability when dealing with log aggregators. Some of the ways to handle this are using local disk to log or a fallback service.
- With privacy and GDPR in full force, be careful to not log personally identifiable information.
Full Post here, 9 mins read