How should metrics and alerting be designed for production?

For production, define the requirements and failure modes for metrics and alerting, automate repeatable configuration, apply least privilege, validate inputs and contracts, set sensible limits and timeouts, add structured logs and metrics, monitor important outcomes, test recovery paths, and document operational ownership. Review the design regularly as scale and dependencies change.