Every founder I’ve spoken with in the last eighteen months has asked me some variation of the same question: “Which AI stack should we use?” They come armed with a dozen browser tabs open to Hugging Face, a few GitHub repositories they found on Hacker News, and a vague sense of anxiety about the speed of change. They want a simple answer—a chart, a checklist, a definitive “use this.” But the reality of building a sustainable company on top of AI infrastructure is far more nuanced than picking a vector database or an LLM provider.
The temptation to treat the AI stack like a standard web development stack is strong, but misleading. In traditional web development, the abstractions are relatively stable. You pick a relational database, a backend framework, and a frontend library. The trade-offs are well-understood. In AI, the stack is a moving target. The cost structures are different, the latency characteristics are unpredictable, and the “best” tool today might be deprecated or fundamentally changed by a model release next week. For a founder, this isn’t just a technical decision; it is a strategic bet on the trajectory of the entire industry.
When I advise founders, I don’t start with tools. I start with the nature of the problem they are trying to solve. The AI stack is not a monolith; it is a layered architecture, and every layer presents a fork in the road between abstraction and control, between speed and cost, between building and buying. Understanding these layers is the first step in making a decision that won’t require a total rewrite six months from now.
The Strategic Layer: Build vs. Fine-Tune
Before you write a line of code or provision a single GPU, you must confront the most expensive decision in your stack: the model strategy. Are you going to rely entirely on API providers like OpenAI, Anthropic, or Google? Are you going to fine-tune an open-source model? Or are you brave enough to train a model from scratch?
The API route is seductive. It offers immediate access to state-of-the-art capabilities without the operational nightmare of managing inference infrastructure. You can ship a product in days rather than months. However, this is a trade-off of control for convenience. When you build on top of an API, you are subject to rate limits, price changes, and the provider’s content policies. I have seen startups pivot their entire user experience because a provider changed their token limits overnight. Furthermore, as your usage scales, the economics can become punishing. API costs are variable and can scale linearly with your user base, making unit economics difficult to optimize.
On the other end of the spectrum is self-hosting open-source models. This offers cost predictability and data privacy, but the engineering lift is significant. You are now responsible for the inference stack: optimizing model weights, managing GPU clusters, and handling latency spikes. This is where the “hidden costs” of engineering time come into play. A team of five backend engineers might spend months just getting a fine-tuned model to run efficiently in production.
The middle ground—and where I see the most sophisticated startups positioning themselves—is fine-tuning or customizing smaller, open-source models. This allows you to specialize a model for your specific domain without the astronomical cost of training from scratch. Tools like LoRA (Low-Rank Adaptation) have made this accessible, allowing you to fine-tune a model with a fraction of the parameters. But even here, the decision isn’t just technical; it’s about data. Do you have the proprietary data required to make fine-tuning worthwhile? If your competitive advantage is unique data, fine-tuning is a moat. If your advantage is UX or workflow, perhaps sticking with a general-purpose API is better.
The Inference Layer: Latency, Cost, and the Hardware Reality
Once you’ve chosen your model strategy, you hit the infrastructure layer. This is where the rubber meets the road in terms of performance. The primary constraint here is the latency vs. cost trade-off.
If you are building a chat interface, latency is critical. Humans perceive a response time of over 200-300ms as “laggy.” Large language models, however, are autoregressive; they generate tokens one by one. Achieving low latency requires significant engineering. You need fast GPUs (like the NVIDIA H100), optimized inference runtimes (like vLLM or TensorRT-LLM), and smart caching strategies.
For many founders, the question becomes: do you manage this yourself or use an inference provider?
Providers like Replicate, Fireworks AI, or Banana offer a serverless layer on top of GPU hardware. They handle the scaling, the queuing, and the optimization. This is excellent for getting to market. However, as you scale, the per-token costs add up. At a certain scale, it becomes cheaper to manage your own dedicated GPU instances.
If you choose to self-host, you enter the world of Kubernetes and container orchestration. This is not for the faint of heart. You will need to manage model weights, handle cold starts (the time it takes to load a model into GPU memory), and ensure high availability. Tools like BentoML or Modal can abstract some of this away, providing a “serverless” experience on your own cloud account, but you are still ultimately responsible for the underlying hardware.
A critical, often overlooked aspect of the inference layer is the quantization of models. Running a model in full precision (FP16 or BF16) requires massive memory bandwidth. By quantizing a model to 8-bit or 4-bit integers (INT8/INT4), you can drastically reduce memory usage and increase throughput with a minimal loss in accuracy. For most production applications, quantization is not just an optimization; it is a necessity. It allows you to serve larger models on cheaper hardware. If your stack doesn’t account for quantization, you are likely overpaying for your compute.
The Orchestration Layer: Chains, Agents, and State
With a model selected and inference running, you need to build the logic that connects your application to the model. This is the orchestration layer, and it has evolved rapidly. Initially, we wrote raw API calls. Then came the era of “prompt engineering” where we manually crafted strings. Now, we are in the era of frameworks.
Frameworks like LangChain and LlamaIndex exploded in popularity because they solved a real problem: connecting LLMs to external data sources and tools. They provided pre-built components for common patterns like retrieval-augmented generation (RAG) and agent loops. However, as applications grow in complexity, these frameworks can become a burden. They introduce abstraction layers that can be hard to debug and often add unnecessary overhead.
For experienced engineers, I often recommend starting with lighter-weight solutions or even raw implementation for critical paths. Pydantic, for example, is excellent for validating model outputs and enforcing structure (a technique known as “function calling” or “JSON mode”). It ensures that the data coming out of your LLM is typed correctly, preventing downstream errors in your application logic.
When building agents—systems that can take actions, such as browsing the web or executing code—the orchestration layer becomes even more critical. The state management of an agent is non-trivial. You need to handle conversation history, tool usage, and error recovery. A common mistake is building an agent that is too brittle, where a single hallucination breaks the entire workflow. A robust agent stack requires a clear separation between the reasoning loop (the LLM deciding what to do) and the execution environment (the tools doing the work).
Consider the memory requirements of an agent. Every time the agent takes an action, you typically need to pass the full conversation history back to the model to maintain context. As this history grows, your token count—and thus your cost and latency—skyrockets. Implementing a summarization strategy or a vector-based memory system (retrieving only relevant past interactions) is essential for scaling agent-based applications.
The Data Layer: RAG, Vector Databases, and Unstructured Data
Most startups don’t have the luxury of a model that knows everything. Your users want answers specific to your data, your documents, and your domain. This brings us to the data layer, specifically Retrieval-Augmented Generation (RAG).
RAG is the process of retrieving relevant information from a knowledge base and feeding it to an LLM to generate a response. It is the most common pattern for grounding LLMs in facts and reducing hallucinations. The stack for RAG typically involves three components: ingestion, storage, and retrieval.
Ingestion is the hardest part. You have unstructured data—PDFs, emails, transcripts, code. You need to parse it, chunk it into meaningful segments, and convert it into vector embeddings (numerical representations). The choice of chunking strategy is vital. If you chunk too small, you lose context. If you chunk too large, the model loses focus, and retrieval becomes noisy. There is no one-size-fits-all chunk size; it depends on your data and the model’s context window.
Storage usually involves a vector database. The market is crowded: Pinecone, Weaviate, Qdrant, Chroma, and even Postgres with the pgvector extension. The choice here is often driven by operational complexity. Managed services like Pinecone are easy to start with but can become expensive. Open-source options like Qdrant or Weaviate offer more control but require self-hosting. Postgres with pgvector is a compelling option if you already rely heavily on Postgres; it keeps your operational surface area small by keeping data in a single system. However, for massive scale (billions of vectors), specialized vector databases usually outperform general-purpose databases.
Retrieval is the query phase. You embed the user’s query, search the vector database for the nearest neighbors (chunks of text semantically similar to the query), and pass those chunks to the LLM. A common pitfall is retrieving too much or too little context. Advanced retrieval strategies, such as hybrid search (combining semantic vector search with keyword search) or re-ranking (using a cross-encoder to score the relevance of retrieved documents), are often necessary to achieve production-grade accuracy.
It is also worth questioning whether a vector database is necessary at all for your use case. If your data is structured or if your retrieval needs are simple, a traditional SQL query might be faster and cheaper. Vector databases are powerful tools, but they are not a universal solution for every data retrieval problem.
The Evaluation Layer: Testing Non-Deterministic Systems
One of the most difficult aspects of the AI stack is testing. Traditional software is deterministic; if you run the same code with the same inputs, you get the same output. AI models are probabilistic. You can ask the same question twice and get slightly different answers. This breaks traditional testing methodologies.
Founders often neglect the evaluation layer until they are in production, dealing with user complaints about hallucinations or poor quality. Building a robust evaluation stack is not optional if you want a reliable product.
There are two main approaches to evaluation: unit testing and LLM-as-a-judge.
Unit testing in AI involves creating a “golden dataset”—a set of inputs and expected outputs. You run your model against this dataset and measure the overlap or semantic similarity. Tools like DeepEval or Ragas help automate this. However, creating a comprehensive golden dataset is time-consuming, and models change, requiring constant updates to the tests.
The more scalable approach is using an LLM-as-a-judge. Here, you use a powerful model (like GPT-4) to evaluate the outputs of a smaller, cheaper model. You prompt the judge model to score the response based on criteria like accuracy, helpfulness, and safety. While this introduces cost, it provides a scalable way to monitor quality across thousands of interactions.
Furthermore, you must monitor for drift. Data drift occurs when the distribution of input data changes over time. Concept drift occurs when the relationship between inputs and outputs changes. In a RAG system, if your underlying documents change, your retrieval quality changes. You need automated pipelines to detect when your model’s performance is degrading, rather than waiting for user feedback.
Security and Privacy: The Invisible Stack
In the rush to ship, security is often an afterthought, but in AI, it is a foundational requirement. The attack surface of an AI application is unique.
First, there is the issue of data privacy. If you are processing user data, where is it going? If you are using an API, does the provider store your prompts for training? Most enterprise agreements allow you to opt out of training, but you must verify this. For highly regulated industries (healthcare, finance), self-hosting is often the only viable path to ensure data never leaves your infrastructure.
Second, there are prompt injection attacks. This is where a user crafts a malicious input that overrides your system instructions. For example, a user might tell the model to “ignore previous instructions” and reveal its system prompt. In a RAG system, an attacker might embed hidden text in a document that, when retrieved, instructs the model to exfiltrate data. Mitigating this requires careful prompt engineering, sanitization of inputs, and strict separation between system instructions and user data.
Third, consider the supply chain of your models. If you are downloading weights from Hugging Face or other repositories, you are trusting the source. Malicious actors can embed harmful code in model files (pickled Python objects). Always use “safe tensors” formats where possible and scan models before loading them into production environments.
Finally, there is the cost of denial-of-service. In traditional web apps, a DDoS attack costs you bandwidth and compute. In AI, a DDoS attack can bankrupt you by forcing massive inference costs. Rate limiting and strict usage quotas are not just performance optimizations; they are financial safeguards. Implementing tiered access—where free users have strict limits and paid users have higher throughput—is essential for protecting your stack from abuse.
Putting It All Together: A Practical Example
Let’s imagine you are building an internal tool for a law firm that answers questions about case files. Here is how a pragmatic stack might look:
1. Model Strategy: You choose a 7B or 13B parameter open-source model (like Mistral or Llama 3) because you need data privacy and cost control. You don’t fine-tune initially; you rely on RAG to inject case-specific data.
2. Inference: You use vLLM running on a single A100 GPU in your own cloud account (AWS/GCP). vLLM is an optimized inference engine that handles dynamic batching and memory management, significantly increasing throughput compared to standard Hugging Face pipelines. You quantize the model to 4-bit to fit it comfortably on the GPU with room for long contexts.
3. Data & RAG: You ingest thousands of PDF case files. You use LangChain (or a custom script) to parse and chunk the text. You store the embeddings in pgvector running on a managed Postgres instance. This keeps your stack simple—one database to rule them all. For retrieval, you use hybrid search: a vector similarity search for semantic meaning and a keyword search (BM25) for specific legal terms.
4. Orchestration: You write a custom Python service using FastAPI. The endpoint accepts a query, retrieves the top 5 relevant chunks from pgvector, constructs a prompt template, and calls the vLLM instance. You use Pydantic to ensure the output is structured (e.g., “Answer: … Citations: […]”).
5. Evaluation: You create a dataset of 50 hard questions with known answers. Every time you update your chunking strategy or model version, you run this dataset through the system and compare the new answers to the old ones using a semantic similarity score. You also set up a dashboard to track latency (p95 and p99) and token usage.
6. Security: The API is behind a corporate VPN. Rate limiting is applied at the load balancer level. User queries are sanitized to remove special characters that might break the prompt structure. The system logs all queries for audit purposes (stripped of PII).
This stack isn’t the flashiest. It doesn’t use the newest agent framework or the largest model. But it is robust, cost-effective, and maintainable. It solves the specific problem at hand without over-engineering.
The Human Element: You Are the Stack
The most sophisticated AI stack in the world is useless without a team that understands it. The technology changes so fast that the specific tools you choose today might be irrelevant in a year. The skills that matter are the fundamentals: understanding how transformers work, knowing how to debug a pipeline, and having the intuition to know when a problem requires a simple heuristic versus a complex model.
When you are choosing your stack, you are also choosing the problems you will face. Choosing a managed API solves infrastructure problems but creates vendor dependency. Choosing open-source solves dependency problems but creates infrastructure complexity. There is no free lunch.
The best advice I can give is to start small. Build a vertical slice of your product with the simplest stack possible. Measure the bottlenecks. Is it latency? Is it cost? Is it accuracy? Let the data guide your next investment in the stack. Don’t build a scalable infrastructure for a product that doesn’t have product-market fit yet. Conversely, don’t hack together a prototype that cannot possibly scale if you are already seeing rapid growth.
The AI stack is a toolkit for solving problems. It is not a religion. The founders who succeed will be those who remain pragmatic, who understand the trade-offs at every layer of the stack, and who prioritize the user experience over the novelty of the technology. The stack is just the foundation; what you build on top of it is what truly matters.
