10 principles for building resilient payment systems (by Shopify)
Shopify has some precious tips for building resilient payment systems.
Lower the timeouts, and let the service fail early The default timeout is 60 seconds. Based on Shopify’s experiences, read timeout of 5 seconds and write timeout of 1 second are decent setups.
Install circuit breaks Shopify developed Semian to protect Net::HTTP, MySQL, Redis, and gRPC services with a circuit breaker in Ruby.
Capacity management If we have 50 requests arrive in our queue and it takes an average of 100 milliseconds to process a request, our throughput is 500 requests per second.
Add monitoring and alerting Google’s site reliability engineering (SRE) book lists four golden signals a user-facing system should be monitored for: latency, traffic, errors, and saturation.
Implement structured logging We store logs in a centralized place and make them easily searchable.
Use idempotency keys Use the Universally Unique Lexicographically Sortable Identifier (ULID) for these idempotency keys instead of a random version 4 UUID.
Be consistent with reconciliation Store the reconciliation breaks with Shopify’s financial partners in the database.
Incorporate load testing Shopify regularly simulates the large volume flash sales to get the benchmark results.
Get on top of incident management Each incident channel has 3 roles: Incident Manager on Call (IMOC), Support Response Manager (SRM), and service owners.
Organize incident retrospectives For each incident, 3 questions are asked at Shopify: What exactly happened? What incorrect assumptions did we hold about our systems? What we can do to prevent this from happening?
We use analytics to understand the usage, to improve user experience and to measure the performance of our website. We anonymise any information we may collate so we can’t identify you personally.
Learn More.