Applied AI System Design Under Customer Constraints

The Applied AI system design round is not the generic distributed-systems interview. The constraints come from a specific customer, and the constraints are the test. Instead of "design for a million users," you get "this runs inside the customer's VPC, their data cannot leave its region, a security team will review the architecture, and answers must return within a latency budget." Your job is to design a working system, often a retrieval pipeline over the customer's own data, inside those walls, and to treat evaluation as the customer's acceptance test rather than an afterthought. The most common failure is designing the happy path and bolting the constraints on at the end. This piece covers how to design from the constraints outward.

The constraints are the problem, not the footnotes

In a normal system design round, the hard part is scale. Here, the hard part is the customer's environment. A frontier model that works in a public demo is the easy part, and it has been for a while. The interesting question is how it survives contact with a real enterprise: their network, their data rules, their reviewers, their performance expectations.

So lead with the constraints. Before you sketch a single box, ask what they are, because they reshape every later decision. The standard set in this round:

• Deployment boundary. Does this run in the customer's VPC, on-premises, or in your cloud with a private link to theirs. On-prem or VPC-only changes which models you can even use and how you index data.
• Data residency. Must the data stay in a region or country. If so, every component that touches it inherits that rule.
• Compliance and review. A security team will examine this. Some industries add audit and retention requirements on top.
• Latency budget. A hard number, often in the low seconds, that constrains how many model calls and retrieval hops you can afford.
• Acceptance criteria. What the customer must see before they trust and deploy the system.

If you treat these as footnotes, you have misread the question. To calibrate against the broader design patterns this round assumes you know, the system design questions are the core practice set, and the must-know set is the fastest way to find the gaps before the loop.

Designing inside a VPC or on-prem

The deployment boundary is usually the first real decision, because it cascades. If the system must run inside the customer's VPC or on-premises, you cannot casually call a public model endpoint or ship their data to an external service. That pushes you toward a model you can deploy in their environment, or a vendor offering with a private deployment and a contractual data boundary. State the assumption out loud and design to it.

From there, build for an environment you do not fully control. Private networking, no unplanned egress, and explicit integration points with systems that were never meant to talk to each other. You are not expected to recite a specific cloud's product catalog. You are expected to reason about the patterns: private endpoints, where each component lives, and how data moves (or pointedly does not move) between them. Naming a managed service is fine, but the score is in the reasoning.

Data residency, compliance, and access

You are not a lawyer and the panel knows it. What they want is a design that behaves as if the data cannot leave its boundary, with the controls a security reviewer will look for. Three things carry most of the signal.

First, residency by construction. If the data must stay in a region, every component that processes it lives there too, including the index, any caches, and logs that might capture content. Do not let a convenience feature quietly copy data out of bounds.

Second, identity and least privilege. The system should respect the customer's existing permission model rather than reading everything as one superuser. A retrieval system that returns documents a given user is not allowed to see is a security incident, not a bug. Carry the user's access scope through retrieval so results are filtered by what that user may actually read.

Third, auditability. Keep a trail of what was accessed and why, and apply retention rules the customer can verify. Show that you would pull in their security and compliance experts rather than guessing at regulations. The AI security and governance questions cover the access, audit, and data-handling concerns that come up here in depth.

The retrieval pipeline over the customer's data

The modal Applied AI design problem is a retrieval system over the customer's own corpus: their documents, tickets, code, or records, made answerable by a model. Walk it end to end, but keep it grounded in the constraints you already named.

Ingestion and indexing: how their data gets in, how often it refreshes, and how you handle formats and permissions at index time so access control is baked in rather than patched on. Chunking and embeddings: enough to show you understand that retrieval quality depends on how you split and represent the documents, without burning your whole window on it. The vector database choice, constrained by the deployment boundary, since an on-prem requirement rules out a hosted-only option. Retrieval and ranking: how you fetch candidates and improve precision, and where a reranking step earns its latency cost given the budget.

Then the latency budget forces honest trade-offs. Every model call and retrieval hop spends from a fixed account. Caching common queries, limiting reranking depth, and deciding when one well-grounded model call beats a multi-step agent loop are exactly the choices the panel wants to see you make on purpose. For the retrieval and agent-specific depth, the RAG and agent design questions are the closest match to this part of the round.

Evaluation as customer acceptance

This is where strong candidates separate themselves: frame evaluation as the customer's acceptance test, not an academic metric. The customer will not deploy a system they cannot trust, so "how do you know it works" is effectively a contractual question.

Say the word acceptance. Define the metric from the business objective you scoped, not a generic benchmark. Hold out a slice of the customer's real data and queries, set a bar that means "good enough to ship" for this use case, and measure against it before go-live. Crucially, include the unhappy cases: the queries that should return nothing, the ambiguous ones, the ones that have historically gone wrong. Customers trust a system more when you show how you measured its failures, not just its wins. A live eval harness that keeps running after deployment, so quality regressions surface before the customer notices, is the detail that signals you have actually shipped to an enterprise.

Budget your time and lead with constraints

A practical split for the round: roughly a third on constraints and high-level architecture, a third on the retrieval and evaluation pipeline over the customer's data, and a third on security, access, and operability. If you spend the whole window on chunking strategy, you will fail on the parts that actually differentiate this interview.

The through-line is simple. Start from the customer's walls, design within them deliberately, and prove the result works on their own data. A slicker pipeline that quietly assumes an open internet and a public endpoint loses to a plainer one that respects the boundary, the budget, and the acceptance bar.

The one-line version

In Applied AI system design, the constraints are the problem. Design from the VPC, the residency rule, the compliance review, and the latency budget outward, build a retrieval pipeline that respects the customer's access model, and treat evaluation as the acceptance test that gets you signed off. Do that and you look like someone who has shipped into a real enterprise, which is exactly the signal the round exists to find.