How should system prompts be applied effectively in Prompt Engineering?

For production use, define requirements first, choose the simplest appropriate design, validate edge cases, automate repeatable behavior, add security and observability, test failure paths, and document important tradeoffs for system prompts.