Key Takeaways
- • Researchers from FINMA and the University of Liechtenstein found that large language models overestimate the quality of weak crypto token projects, with the error concentrated in the lowest-rated whitepapers.
- • Across 1,008 expert-LLM rating pairs covering 26 projects, the pooled correlation between human and LLM ratings was essentially zero, with a mean absolute error of 1.14 on a 0-to-5 scale.
- • Structured summarization combined with Chain-of-Thought prompting produced the lowest error (MAE 0.706, RMSE 0.967), though rank-correlation differences between configurations were not statistically resolvable.
- • Low-rated projects showed a mean signed error of -1.353, indicating systematic overestimation, and the same directional bias appeared in two small open-source models.
- • The authors recommend LLMs serve as screening and explanation tools with human experts retaining final due diligence judgment, particularly for borderline projects.
Summary
A study published in Knowledge and Information Systems, led by Yann Thorens of FINMA with Martin Angerer, Pejman Ebrahimi and Johannes Schneider of the University of Liechtenstein, tested whether large language models can help retail and professional investors assess token-offering whitepapers, a market characterized by thin disclosure standards and limited investor protection. The team used Design Science Research to build a prototype evaluation framework drawing on the Token Offerings Research Database, which contains more than 6,400 ICOs, STOs and IEOs. From this database they analyzed 29 projects, of which 26 formed the final sample after exclusions.
The researchers tested three content-processing approaches (partial document rating, structured summary rating and aspect-focused summary rating) alongside four prompting strategies: Zero-Shot, Few-Shot, Chain-of-Thought, and Chain-of-Thought with Context and Rubric. The design generated 1,139 evaluation runs, of which 1,008 complete expert-LLM pairs were analyzed. The authors noted that prompt sensitivity was pronounced: introducing specific risk criteria shifted one rating from a positive 4 to a critical 2.
The quantitative results showed a pooled correlation between human and LLM ratings that was essentially zero, with a mean absolute error of 1.14. The first two approaches overestimated investment quality relative to experts, with mean signed differences of -1.01 and -0.92. Structured summarization paired with Chain-of-Thought prompting achieved the lowest error and outperformed Zero-Shot and aspect-focused alternatives under project-clustered bootstrap comparisons. The stratified analysis found that error was heavily concentrated in weak projects, where the mean absolute error reached 1.549 and the mean signed error was -1.353. Highly rated projects showed lower absolute error but a mild underestimation bias of +0.435. The authors attribute this to generic summaries preserving promotional language when red flags are implicit rather than explicit.
A multi-model robustness check using Qwen2.5-0.5B and Qwen2.5-1.5B on 47 runs across 25 projects reproduced the overestimation pattern, though the authors describe this comparison as exploratory. The study also acknowledges limitations, including the effective sample of 26 projects, the lack of judge-level annotations, and the possibility that some model-human disagreement reflects genuine variability in early-stage venture assessment.
The authors recommend a hybrid workflow in which LLMs structure evidence, surface red flags and flag uncertainty, while final judgment remains with domain experts. Divergent cases among score, evidence and rationale should be escalated to human reviewers. Future work will test additional document types such as governance proposals, tokenomics documents and smart-contract audit reports, as well as judge models and agentic systems.
The researchers tested three content-processing approaches (partial document rating, structured summary rating and aspect-focused summary rating) alongside four prompting strategies: Zero-Shot, Few-Shot, Chain-of-Thought, and Chain-of-Thought with Context and Rubric. The design generated 1,139 evaluation runs, of which 1,008 complete expert-LLM pairs were analyzed. The authors noted that prompt sensitivity was pronounced: introducing specific risk criteria shifted one rating from a positive 4 to a critical 2.
The quantitative results showed a pooled correlation between human and LLM ratings that was essentially zero, with a mean absolute error of 1.14. The first two approaches overestimated investment quality relative to experts, with mean signed differences of -1.01 and -0.92. Structured summarization paired with Chain-of-Thought prompting achieved the lowest error and outperformed Zero-Shot and aspect-focused alternatives under project-clustered bootstrap comparisons. The stratified analysis found that error was heavily concentrated in weak projects, where the mean absolute error reached 1.549 and the mean signed error was -1.353. Highly rated projects showed lower absolute error but a mild underestimation bias of +0.435. The authors attribute this to generic summaries preserving promotional language when red flags are implicit rather than explicit.
A multi-model robustness check using Qwen2.5-0.5B and Qwen2.5-1.5B on 47 runs across 25 projects reproduced the overestimation pattern, though the authors describe this comparison as exploratory. The study also acknowledges limitations, including the effective sample of 26 projects, the lack of judge-level annotations, and the possibility that some model-human disagreement reflects genuine variability in early-stage venture assessment.
The authors recommend a hybrid workflow in which LLMs structure evidence, surface red flags and flag uncertainty, while final judgment remains with domain experts. Divergent cases among score, evidence and rationale should be escalated to human reviewers. Future work will test additional document types such as governance proposals, tokenomics documents and smart-contract audit reports, as well as judge models and agentic systems.
Go deeper than the headline
Research every firm in this story, track the ones that matter and get the daily briefing — free for 7 days.