Author: Toh Kim Huat
“How do I know things are working? How can I be the first to find out if things stop working?”
Once software is running it’s important to know that it’s running well and is meeting it’s objectives and business needs. Your system should collect and publish logs and metrics, and provide appropriate alerts, that keep your teams constantly informed of its current state.
This is not a day 2 consideration. Often teams will leave the configuration of production monitoring and alerting as a task to be completed close to go-live. As with everything that is left until the last minute, it will be harder to do and therefore done less well, than if it was a routine consideration. Instead, part of your definition of done or your NFRs should include the requirement that new features, and any important or interesting metrics associated with them, will be observable in production.
Design Applications for Operations
Applications should be designed with Operations in mind so that there can be better operational readiness and a smoother transition from development to operations (ideally we should not even feel such transition taking place). By having certain design considerations upfront such as verbose error messages and externalising of configurations, it can help in easier debugging and faster resolution and execution of configuration changes.
Automate Operations
The shift to “shift-left” infrastructure and Infrastructure-as-Code (IaC) offers numerous advantages over manual configuration, including improved consistency, version control, automation, and enhanced disaster recovery capabilities. By leveraging automation tools for various operational aspects (e.g. patch management, asset inventory management, account and access management and logs monitoring), agencies can streamline operations, reduce human error, and more effectively manage their cloud infrastructure, aligning with best practices for cloud governance and addressing common audit findings and security concerns in Government IT systems.
Design for Observability
Observability enables teams to understand, monitor, and troubleshoot complex distributed systems effectively through practices like structured logging, SLOs, SLIs, performance metrics, distributed tracing, and error tracking. These practices are crucial for proactive issue detection, faster problem resolution, improved system reliability, and better alignment with business goals, ultimately helping organisations maintain high-quality services and make data-driven decisions.
Establish Monitoring
A comprehensive monitoring strategy established at inception is crucial for effective system oversight, aligning with business objectives and defining clear performance metrics. This proactive approach enables early issue detection, facilitates rapid incident response, and provides actionable insights for continuous improvement, ensuring all stakeholders have visibility into system health and performance.
Drive Continuous Operation Improvements
Continuous operation improvements encompass regular reviews of metrics, tools, and scalability, post-incident analyses, performance optimization, cloud cost management, and capacity planning. These practices collectively ensure system reliability, efficiency, and alignment with business needs while maintaining optimal performance and cost-effectiveness.
