Key Takeaways
- • Sequoia Capital surveyed 33 companies across its network, from seed-stage startups to large public enterprises, and found that 65% had LLM applications in production, up from 50% two months earlier.
- • 94% of surveyed companies use a foundation model API, with OpenAI's GPT the clear favorite at 91%, while Anthropic usage rose to 15% over the prior quarter.
- • 88% expect a retrieval mechanism such as a vector database to remain a key part of their stack, and 38% are interested in orchestration frameworks such as LangChain.
- • Only 15% have built custom language models from scratch or from open source, though custom model training has increased meaningfully in recent months.
- • Sub-10% of respondents are seeking tools to monitor LLM outputs, cost, or performance, a gap Sequoia expects to grow as large companies and regulated industries adopt the technology.
Summary
Sequoia Capital published findings from interviews with 33 companies across its network on how they are building applications on large language models (LLMs), according to the report dated June 14, 2023. Sequoia reports that nearly every company in its network is integrating language models into products, with examples spanning code (Sourcegraph, Warp, GitHub), data science (Hex), marketing (HubSpot, Jasper, Writer), legal (Ironclad, Harvey), and consumer payments (Klarna). The firm cautions that the stack is still evolving rapidly, and that practitioners lack high confidence in the end-state architecture.
The emerging stack centers on language model APIs, retrieval, and orchestration. Of the companies surveyed, 65% had applications in production, compared with 50% two months earlier, and 94% used a foundation model API. Vector databases were cited by 88% as a likely long-term component, used to improve output quality, reduce hallucinations, and address data freshness. Purpose-built vector databases such as Pinecone, Weaviate, Chroma, Qdrant, and Milvus are favored by startups, while larger companies tend to rely on tools from their cloud providers. Sequoia notes that context windows are expanding, citing OpenAI's expansion to 16K and Anthropic's 100K-token window, which could eventually embed retrieval directly into foundation model services.
Sequoia identifies three ways to customize models: training from scratch, which it describes as the hardest path and cites BloombergGPT as an example; fine-tuning a base model, which some practitioners privately describe as harder than expected due to model drift; and using a pre-trained model with retrieved context, which it characterizes as the least difficult approach. Sequoia states that the LLM API and custom model stacks are likely to converge over time, and that interest in custom training has grown rather than declined despite broader API availability.
The report also identifies trust as a barrier to full adoption. Companies in regulated sectors such as fintech and healthcare are seeking tools for data privacy, security, copyright, and output monitoring. Robust Intelligence was cited as addressing these challenges, with customers including PayPal and Expedia. Sequoia further notes that ChatGPT consumer data is used for training by default, while ChatGPT Business and API data are not. Sequoia expects multimodal applications combining text, speech, and image generation to become more prevalent, and characterizes the infrastructure layer as early in its development.
The emerging stack centers on language model APIs, retrieval, and orchestration. Of the companies surveyed, 65% had applications in production, compared with 50% two months earlier, and 94% used a foundation model API. Vector databases were cited by 88% as a likely long-term component, used to improve output quality, reduce hallucinations, and address data freshness. Purpose-built vector databases such as Pinecone, Weaviate, Chroma, Qdrant, and Milvus are favored by startups, while larger companies tend to rely on tools from their cloud providers. Sequoia notes that context windows are expanding, citing OpenAI's expansion to 16K and Anthropic's 100K-token window, which could eventually embed retrieval directly into foundation model services.
Sequoia identifies three ways to customize models: training from scratch, which it describes as the hardest path and cites BloombergGPT as an example; fine-tuning a base model, which some practitioners privately describe as harder than expected due to model drift; and using a pre-trained model with retrieved context, which it characterizes as the least difficult approach. Sequoia states that the LLM API and custom model stacks are likely to converge over time, and that interest in custom training has grown rather than declined despite broader API availability.
The report also identifies trust as a barrier to full adoption. Companies in regulated sectors such as fintech and healthcare are seeking tools for data privacy, security, copyright, and output monitoring. Robust Intelligence was cited as addressing these challenges, with customers including PayPal and Expedia. Sequoia further notes that ChatGPT consumer data is used for training by default, while ChatGPT Business and API data are not. Sequoia expects multimodal applications combining text, speech, and image generation to become more prevalent, and characterizes the infrastructure layer as early in its development.
Go deeper than the headline
Research every firm in this story, track the ones that matter and get the daily briefing — free for 7 days.