Key Takeaways
- • A free LLM benchmark for portfolio construction ranked Claude Haiku first with a Sharpe ratio of 1.08, followed closely by GPT mini at 1.07 and Qwen in third place, all outperforming the S&P 500's 240.1% return over the comparison window in backtested results.
- • The benchmark uses a controlled methodology with identical market snapshots, rebalancing cadence, and scoring systems across five LLM models selecting from the S&P 500 universe, then passes allocations to a deterministic backtester to evaluate risk-adjusted performance rather than raw returns alone.
- • Risk-adjusted performance (Sharpe ratio ranking) produced materially different results than return-only leaderboards, with Qwen generating higher absolute returns than GPT mini but placing third due to slightly lower risk efficiency—demonstrating the critical importance of evaluation methodology in LLM investment benchmarking.
- • The creator explicitly cautions that backtests are "fragile" and "dangerous when treated as prophecy," emphasizing the benchmark's purpose as a controlled comparative arena rather than evidence that any model can reliably beat markets in production.
Summary
An independent researcher at QUINETICS published a novel benchmarking framework in May 2026 designed to evaluate large language models' ability to construct portfolios under controlled, repeatable conditions. Rather than testing LLMs on abstract reasoning tasks, the benchmark presents each model with identical market snapshots and directs them to construct long-only S&P 500 equity portfolios, then evaluates performance through deterministic backtesting with standardized metrics.
The methodology ranks model performance by Sharpe ratio—a risk-adjusted metric—rather than raw returns alone. This design choice proved consequential: Claude Haiku ranked first with a Sharpe ratio of 1.08, with GPT mini nearly tied at 1.07. Qwen generated higher absolute returns than GPT mini but placed third due to marginally lower Sharpe ratios, illustrating how evaluation methodology directly shapes competitive rankings. All five models tested exceeded the S&P 500 benchmark's 240.1% return over the comparison window in backtested results.
The creator emphasizes critical methodological constraints. The benchmark operates as a "controlled arena" with uniform rules, investment universe, rebalancing schedules, and scoring systems across all participants. This design addresses a persistent weakness in LLM benchmarking: most tests evaluate models on decontextualized tasks like code writing or puzzle-solving rather than repeatable decision-making under real market constraints. The researcher explicitly warns that backtests remain "fragile" and "dangerous when treated as prophecy," positioning the framework as a comparative tool rather than evidence of predictive market-beating ability in live trading environments.
The distinction between risk-adjusted and absolute return rankings carries material implications for institutional investors evaluating LLM-driven investment tools. A leaderboard ranked solely by returns would misrepresent the efficiency with which models generated those returns, potentially leading allocators toward strategies with unnecessary volatility or drawdown risk. By anchoring rankings to Sharpe ratios, the benchmark rewards models that produce consistent, repeatable returns relative to risk—a more rigorous standard for fiduciary evaluation.
The methodology ranks model performance by Sharpe ratio—a risk-adjusted metric—rather than raw returns alone. This design choice proved consequential: Claude Haiku ranked first with a Sharpe ratio of 1.08, with GPT mini nearly tied at 1.07. Qwen generated higher absolute returns than GPT mini but placed third due to marginally lower Sharpe ratios, illustrating how evaluation methodology directly shapes competitive rankings. All five models tested exceeded the S&P 500 benchmark's 240.1% return over the comparison window in backtested results.
The creator emphasizes critical methodological constraints. The benchmark operates as a "controlled arena" with uniform rules, investment universe, rebalancing schedules, and scoring systems across all participants. This design addresses a persistent weakness in LLM benchmarking: most tests evaluate models on decontextualized tasks like code writing or puzzle-solving rather than repeatable decision-making under real market constraints. The researcher explicitly warns that backtests remain "fragile" and "dangerous when treated as prophecy," positioning the framework as a comparative tool rather than evidence of predictive market-beating ability in live trading environments.
The distinction between risk-adjusted and absolute return rankings carries material implications for institutional investors evaluating LLM-driven investment tools. A leaderboard ranked solely by returns would misrepresent the efficiency with which models generated those returns, potentially leading allocators toward strategies with unnecessary volatility or drawdown risk. By anchoring rankings to Sharpe ratios, the benchmark rewards models that produce consistent, repeatable returns relative to risk—a more rigorous standard for fiduciary evaluation.
Go deeper than the headline
Research every firm in this story, track the ones that matter and get the daily briefing — free for 7 days.