Author: Naveen Kumar
Applications should be designed with Operations in mind so that there can be better operational readiness and a smoother transition from development to operations. By having certain design considerations upfront such as verbose error messages and externalising of configurations, it can help in easier debugging and faster resolution and execution of configuration changes.
Design for Easy Debugging and Issue Resolution
Efficient debugging and issue resolution are critical for maintaining system reliability and reducing downtime. By embedding debugging-friendly practices in application design, teams can quickly identify, analyze, and fix issues before they escalate. Below are some best practices to ensure your application is easy to debug and resolve issues.
Verbose Error Messages - Ensure that error messages are detailed and provide enough context to guide troubleshooting. Do note that you need to balance this with security. i.e. Do not divulge sensitive information in error messages. This includes information that reveals system information such as version of middleware being used, application stacktrace, etc as these allows attackers to learn what versions of software components are being used and try to exploit vulnerabilities that exist in these components.
One possible solution is to include an unique error ID or transaction ID in the error message. Then in the error logs on the server, include the same error ID or transaction ID. When users report issues and they provide screenshots showing the error ID or transaction ID in the error message, we can then use it to quickly correlate to log entries containing details such as exception stacktraces.
Centralized Logging - Centralized logging systems consolidate logs from multiple services and components into a single location, for example storing CloudWatch logs in a common AWS account. This makes it easier to follow requests across distributed systems and track down issues faster.
Correlation IDs: Use correlation IDs to trace the lifecycle of a request through the system. When an error occurs, having a unique identifier makes it easier to trace what happened across different microservices or layers.
Structured logging: Use structured logging (e.g., JSON format) to ensure consistency and make logs easier to parse and analyze. Refer to 6.3 Design for Observability for more info about structured logging.
Example: In a microservices architecture, when a user makes a request that touches several services, a correlation ID (e.g.,
request-12345) can be attached to each log entry. This ID allows tracing the full flow of the request across services, simplifying root cause analysis.Environment Parity - Maintain environment parity (development, staging, production) to ensure that issues encountered in production can be replicated and resolved in lower environments. Refer to Automate Environment Setup for more info about environment parity.
Externalize all configurations
Managing application configurations externally ensures operational flexibility and reduces risks associated with hard-coded values. This practice also allows operations teams to manage environments effectively and make changes without requiring code modifications or redeployment. Here are some best practices to follow:
Use Environment Variables for Configuration Where Possible - Storing configuration values such as database connections, API keys, and service endpoints in environment variables is a widely accepted practice. This approach ensures that sensitive data isn’t stored within the codebase, improving security. It also makes it easy to manage configurations across different environments (development, testing, production) by simply updating environment-specific variables.
Ensure that all sensitive information (e.g., credentials, tokens) is only accessible via secure, encrypted environment variable storage mechanisms. Consider using secrets management tools like AWS Secrets Manager. For other non-sensitive information, can consider using tools like AWS Systems Manager’s Parameter Store.
Implement Feature Flags for Easy Enabling/Disabling of Features - Feature flags (or toggles) are an excellent way to externally control the activation of new features without requiring code changes or redeployment. They provide the flexibility to perform controlled rollouts, A/B testing, or immediate disabling of malfunctioning features. Feature flags also enable safe experimentation in production environments. Ensure feature flags can be toggled at runtime, without restarting or redeploying the application, and ensure there is an easy rollback path in case a feature causes issues. Refer to Keep the Codebase in an Always Releasable State for more info about feature flags.
Design for Easy Configuration Updates Without Redeployment - Ensure the application has built-in mechanisms to refresh configurations dynamically, either by polling for changes or subscribing to configuration updates. Gracefully handle configuration changes, ensuring they don’t introduce instability during runtime.
Design for Disaster Recovery
Disaster Recovery (DR) is a critical aspect of application design that ensures business continuity in the face of unexpected events. It is an important consideration even if the application is not CII/SII.
- Backup and Restore - Ensure that the application design incorporates immutable backup capabilities. Immutable backups are backups that prevent unauthorised modifications to backup data. The team should check/test the following:
Check that the backup frequency and backup mechanism (e.g. point-in-time recovery) aligns with the required Recovery Point Objective.
Verify the backup schedule and retention policies are configured correctly.
Test backups to ensure that they can be restored successfully.
Check how the operations team is monitoring the backup jobs and whether backup failures are addressed promptly.
IM8 Policy: BR-1: Backup
Regularly backup all important data and systems, and store backups in a secure and separate location.
IM8 Policy: BR-2: Recovery Testing
Conduct regular testing of recovery processes to ensure their effectiveness.
IM8 Policy: BR-3: Backup Retention
>Prevent backups from being modified or deleted or as stipulated in the agency’s data retention policies.
- Disaster Recovery Plan - Ensure that a comprehensive disaster recovery plan is in place and it should be aligned with the agency’s broader Business Continuity Plan (BCP). The team should check the following:
Check that Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO) are clearly defined and achievable.
Check that the DR Plan includes detailed steps for various disaster scenarios, including timing and sequence.
Check that roles and responsibilities are clearly assigned for DR activities and communication protocols and escalation paths are established.
The IT team should arrange for disaster recovery exercises periodically and participate and observe the conduct of these DR exercises.
- Data Replication - The correct implementation of the data replication is a critical component in ensuring the RPO can be met. The team should check the following:
Examine the replication strategy, especially in assessing the synchronisation method used and whether the method can achieve the required RPO.
Ensure that replication includes not just data, but also application configurations and dependencies.
Implement monitoring and alerting for replication lag or failures.
Monitor that the bandwidth for the replication channel is sufficient.
Check that data consistency and integrity is maintained across replicated sites.
- Backup Asset Health and Security - The team should check the following to ensure the health and security of backup assets:
Ensure restoration tests are conducted regularly to check that backup images can be restored correctly.
Perform test restorations in an isolated environment to prevent any accidental production impact.
Perform malware scans on backup data to prevent any spread of malware infections during restoration.
Implement regular vulnerabilities scan on backup systems and ensure results are reviewed and acted upon.
