Author: Naveen Kumar
Observability is crucial because it enables teams to understand, monitor, and troubleshoot complex distributed systems effectively. By implementing structured logging, establishing Service Level Objectives (SLOs), measuring Service Level Indicators (SLIs), defining key performance metrics, implementing distributed tracing, monitoring dependencies, and tracking errors and exceptions, teams can gain comprehensive insights into their system’s behavior and performance. These practices are essential because they allow for proactive issue detection, faster problem resolution, improved system reliability, and better alignment with business goals. Ultimately, designing for observability helps organizations maintain high-quality services, respond quickly to incidents, and make data-driven decisions for continuous improvement.
Implement structured logging
Structured Logging involves recording log events in a well-defined, machine-readable format such as JSON. Instead of writing log messages as plain text, log data is organised into key-value pairs, making it easier to search, filter, and analyse.
Some key aspects of Structured Logging for Observability:
Consistency - Logs consistently follow a predefined structure, enabling easier parsing by log aggregation tools. For example, instead of a log saying:
Error: User not found for ID 123
You could instead write the log as:
{“level”: “error”, “message”: “User not found”, “user_id”: 123, “timestamp”: “2024-09-30T10:12:00Z”}
Searchability - With structured logs, querying becomes efficient since logs can be indexed by specific fields, allowing for advanced filtering, aggregation, and correlation.
Automation & Alerting - Because logs are machine-readable, monitoring systems can more easily process log data to trigger alerts or respond to incidents (e.g. alert on a high frequency of certain error types).
Contextual Insights - We should include more context in structured logs (e.g. request ID, transaction ID, correlation ID, etc) that help in tracing the flow of events across distributed systems, especially in microservices architectures.
Establish SLO and Measure SLI/Error Budgets
One of the critical aspects of effective monitoring is setting clear, measurable goals for system performance. This is done through the establishment of Service Level Objectives (SLOs) and the tracking of Service Level Indicators (SLIs), as well as maintaining an error budget.
- Define SLOs Service Level Objectives are measurable goals that reflect the performance and availability of a service as perceived by the users. SLOs should align with business goals and user expectations, balancing reliability with feature velocity. When setting SLOs:
Involve stakeholders to define acceptable performance levels.
Focus on key metrics like availability, latency, or throughput, depending on your service’s nature.
SLOs should be realistic and achievable yet ambitious enough to push the team toward excellence.
Some examples of SLOs:
99.9% uptime for a web application in a calendar month
99% of requests completed within 1 second
Less than 0.1% of requests resulting in 5xx errors over a 24-hour period
- Measure SLIs Service Level Indicators are the quantifiable metrics that indicate how well your service meets the defined SLOs. They measure specific aspects of system performance, such as:
Latency (time taken to process a request)
Availability (percentage of time the system is operational)
Error rate (percentage of failed requests) Ensure SLIs are collected in real-time and accurately reflect the user experience, as this data will drive decision-making and alerting.
- Error Budgets The error budget is the allowable margin for errors within the defined SLOs. It represents the difference between 100% uptime (perfect service) and the agreed-upon SLO target. For example, if the SLO is set to 99.9% uptime, the error budget is 0.1%.
Define and expose key performance metrics for key business transaction flows
Effective Observability starts with a clear understanding of the business transactions that are critical to the application. Some best practices include:
Identify Key Business Transactions and Define Key Metrics for them
Start by identifying the essential performance metrics for each business transaction flow. These metrics provide insights into how those business transactions are performing. For example, if we are expecting 2.7 million voter registrations on Polling Day, tracking the number of successful voter registrations, number of failed voter registration and number of voter registrations requiring exception flows would give us much better operational awareness and how well the system is supporting those business transactions.
Capture business-specific events as structured events in logs or custom metrics
Capture business-specific events (e.g. number of leave submissions, number of claim submissions, number of voter registrations, number of student assignment submissions, etc) as custom metrics or log them as structured events. Custom metrics can be used for dashboarding and structured events can be used for debugging or investigations using Observability approaches.
Implement distributed tracing
In modern architecture, especially microservices architecture, a single user request often traverses multiple services. This creates complexity in diagnosing issues like slow response times or errors. Traditional logging alone is insufficient to provide end-to-end visibility of how requests flow through the system. To address this, distributed tracing is a crucial observability practice that helps track requests across services, providing a clear picture of their journey and the performance of each step.
Some best practices include:
Instrumentation of All Critical Services
Ensure that all services, including databases (e.g. what database queries are performed and the response time) and external APIs (e.g. what external APIs are invoked and their status and response time) are instrumented for tracing. If there are ‘breakages’ along the way, it would affect the effectiveness of distributed tracing.
Use Unique Trace IDs
Assign a unique trace ID to each incoming request. This ID should be propagated across all service boundaries and included in logs, metrics, and traces to enable correlation.
Maintain Context Propagation
As requests move through services, maintain and propagate the trace context (trace ID, span ID, etc) to maintain continuity.
Define Key Spans and Tags
Break down traces into spans, representing different parts of the request lifecycle. Use meaningful names for spans and tag them with relevant metadata (e.g. error codes, user IDs, and request payloads) to provide deeper insights.
Establish Dependency Monitoring
In modern distributed systems, applications often rely on various third-party services and external APIs. These dependencies can significantly impact system performance and availability. Some best practices include:
Monitor Third-Party API Performance and Availability
Monitoring the performance and availability of Third-Party APIs ensures that any degradation in the services would not go unnoticed.
Track Response Time - Measure the response time of external APIs to detect any slowness.
Monitor API uptime - Keep track of the availability of Third-Party APIs to identify outages or service disruptions.
Set thresholds and alerts - Establish baseline performance metrics for external APIs and set up automated alerts when they deviate from expected performance.
Track External Service Dependencies
Applications often depend on external services such as SingPass/CorpPass, MyInfo, payment gateways, etc. It is essential to have visibility into these dependencies to prevent cascading failures across the system. Key actions include:
Map out dependencies - Create a clear visualisation of the application’s external service dependencies to understand the potential impact of any failures.
Monitor service health - Track the health of critical services
Perform regular dependency reviews - Conduct periodic reviews of external services to assess their stability, performance and overall impact on your system.
- Implement Circuit Breakers and Monitor their Status
Circuit breakers are used to detect and mitigate failures in external services by temporarily halting traffic to problematic services, preventing cascading failures. Steps to consider include:
Monitor Circuit Breaker States - Track whether circuit breakers are in closed (normal), open (error), or half-open (recovery) state to detect dependency issues early.
Set Alerts for Circuit Breaker Events - Configure notifications when a circuit breaker is triggered to allow timely investigation of service disruptions.
Analyze Circuit Breaker Data - Use data from circuit breakers to analyze trends and identify recurring external service failures, so that you can take proactive measures.
Track Errors and Exceptions
Effective monitoring involves not only tracking performance metrics but also monitoring of errors and exceptions across all layers of the application. Timely detection and resolution of errors are critical to minimizing downtime and ensuring smooth application functionality.
Centralise Error Logs Across All Application Components
To gain a comprehensive view of your system’s health, centralize error logging across all services, microservices, and application layers. Use a centralized logging solution that aggregates logs from different components, allowing for easier tracking and analysis of errors. This helps prevent fragmented error reporting and provides a single source of truth for troubleshooting and debugging.
Set up Alerts for Critical Errors and Exceptions
Not all errors are equally important. Establish critical errors and exceptions that could impact system performance or user experience. Set up real-time alerts for these issues so your team can respond quickly. Ensure that these alerts are context-rich, providing sufficient information to enable a quick root-cause analysis and action. It’s also important to prioritize alerts to avoid alert fatigue and focus on the most impactful issues.
Track Error Rates and Trends Over Time
Monitoring individual errors in real-time is important, but tracking error rates and trends over time offers deeper insights into system stability. By analyzing error patterns, you can identify recurring problems, evaluate the effectiveness of bug fixes, and preemptively address issues before they escalate. Trend analysis also helps in identifying patterns (e.g. specific errors happening during certain days or timings), capacity planning, revealing potential bottlenecks or stress points that may require optimization.
Implement Log Correlation for Easier Troubleshooting
Isolated logs provide limited context, making troubleshooting complex issues in distributed systems more challenging. Implement log correlation, which links related logs from different services and components. This correlation allows for better understanding of how an error in one part of the system might cascade and affect other components. Tools that automatically trace request paths and errors across distributed systems can greatly speed up incident response and resolution.
