How should asynchronous messaging be applied in a production system?

Make consumers idempotent, define retry and dead-letter policies, version schemas, and monitor lag.