System Design Basics for Production Applications
System design is not just an interview topic. These are the decisions that are expensive to reverse once an application is live.
By Uttam Thapa · · System Design
⚡ Executive Summary (TL;DR)
System design is not a whiteboard exercise reserved for interviews — it is the small set of decisions that are expensive to reverse once real users arrive. This covers the ones that actually matter at production scale: understanding the read and write pattern before choosing an architecture, letting the database enforce correctness, classifying failures so a non-critical dependency cannot take down checkout, and deferring every split until there is a concrete reason for it.
Figure 1: The boundaries worth drawing early — and the ones worth postponing.
Introduction
System design is often presented as something you only need for interviews at large companies. In practice, the decisions it covers show up the moment you build anything that real people use.
This article covers the system design thinking I apply when building production applications. It is not about designing for millions of users on day one. It is about making the small number of decisions that are expensive to reverse later.
Start With the Constraints, Not the Diagram
The most common mistake is drawing boxes before understanding what the system actually has to do.
Before any architecture decision, I try to answer four questions.
- What is the read and write pattern? Mostly reads, mostly writes, or balanced?
- What has to be correct immediately, and what can be eventually correct?
- What is the acceptable failure mode when a dependency is down?
- What is realistically going to grow first, data volume or request volume?
Most architectures fail not because they were too simple, but because they were complex in the wrong place.
Separation of Concerns
Every production application I build follows the same layering on the backend.
Routes -> request handling and validation
Controllers -> business logic orchestration
Services -> database access and external calls
Middleware -> authentication, logging, error handling
The value of this is not tidiness. It is that each layer can be changed independently.
- Swapping a payment provider touches the service layer only.
- Adding rate limiting touches middleware only.
- Changing a response shape touches the controller only.
When those responsibilities are mixed together, every change has an unpredictable blast radius.
Where State Lives
The hardest problems in any system are usually about state, not about traffic.
Rules I Follow
- The database is the source of truth. Caches are disposable.
- Anything the client sends can be wrong or replayed. Validate on the server.
- Money and inventory changes belong in a transaction, never in application memory.
- If two requests can race, assume they will.
This last point matters more than it seems. A read-then-write check that looks correct in testing will eventually run twice at the same moment in production. The fix is usually a database constraint rather than more application code.
Designing for Failure
External services will be unavailable at some point. The question is what your system does when that happens.
Failure Categories
- Retryable: a timeout on an idempotent request. Retry with backoff.
- Fatal: invalid credentials. Fail loudly at startup rather than at request time.
- Degraded: a non-critical service, such as analytics. Continue without it.
Classifying failures this way early prevents the most frustrating category of bug: an unimportant dependency taking down an important flow.
Validate Configuration at Startup
A pattern I now apply to every service is validating environment configuration before the application accepts traffic.
Load environment
-> Validate required variables
-> Fail fast with a clear message if missing
-> Start server
Without this, a missing variable becomes a 500 error on a user action, often days later. With it, the failure happens at deploy time where it belongs.
Idempotency
If an operation can be triggered twice, it should produce the same result both times.
This applies to more than payments.
- Form submissions where the user refreshes.
- Webhooks, which providers deliberately re-send.
- Background jobs that retry after a crash.
The usual implementation is a unique key on the operation and a database constraint that rejects duplicates. Let the database enforce it rather than checking first in application code.
Choosing Boundaries
The decision to split a system into services is often made too early.
Reasons That Justify a Split
- Genuinely different scaling profiles.
- Different teams owning different lifecycles.
- A hard isolation requirement, such as compliance.
Reasons That Do Not
- The codebase feels large.
- Microservices are considered modern.
- It might be needed later.
A well-layered single service is easier to operate than four small ones connected by a network you now have to reason about.
Observability
You cannot debug what you cannot see. The minimum I set up on any production service is:
- Structured request logging, without sensitive values.
- A health endpoint that reports the environment.
- Errors logged with enough context to reproduce them.
- Clear separation between operational errors and unexpected ones.
Operational errors are expected and can be shown to the user. Unexpected errors should be logged in full and shown to the user as a generic message, so internal details never leak.
Key Takeaways
- ✓Understand the read and write pattern before choosing an architecture.
- ✓Layer the backend so changes stay contained.
- ✓Let the database enforce correctness, not application checks.
- ✓Classify failures so unimportant dependencies cannot take down important flows.
- ✓Validate configuration at startup, not at request time.
- ✓Split a system only when there is a concrete reason to.
Good system design is mostly about deferring complexity until it is genuinely required, and being deliberate about the few decisions that are hard to undo.
Frequently asked questions
Do small applications really need system design?
Yes, but only for the decisions that are expensive to reverse: where state lives, what the database enforces, and where the boundaries sit. Everything else can be deferred until there is evidence you need it.
When should you split a monolith into services?
When you have a concrete reason — an independent scaling profile, a separate deployment cadence, or a team boundary. Splitting for tidiness buys you distributed-systems problems in exchange for nothing.
What is the most common system design mistake?
Designing for imagined scale instead of the actual read and write pattern. The second most common is putting correctness rules in application code that the database could enforce for you.
Home · Projects · Blog · Services · Résumé · Contact