Same Person, Different Advice: Auditing LLM Financial Advice with Repeated Risk-Tolerance Responses (OpenAlex (recherche mondiale))

Summary

• Across 800 real household profiles where only the risk answer changed, seven large language models shifted recommended stock allocations by 10 to 16 percentage points on average.p.1
• When the stated risk answer changed, recommended stock shares moved by 24 to 35 percentage points, compared to 0.6 to 4.2 points when repeating the exact same prompt.p.1
• Discounting the risk answer for its measured reliability according to classical test theory would cut advice swings by a factor of 2.4 to 3.3.p.1
• Prompting models with a noise warning steadies some models but shifts weight to age and raises stock recommendations for cautious clients unwilling to take any risk from 4% to 22%.p.1
• In a scripted clarification test, asking one clarifying question reduced the average unwanted swing from 33.5 points to 1.9 points.p.1

Key insights

1. LLM financial advisers rely predominantly on single stated-risk questions, making their recommendations highly sensitive to transient response error.p.1
2. In a crisis-free six-month laboratory retest, changes in risk rating alone shifted LLM stock recommendations by 11 to 18 points.p.1
3. Five of seven audited LLMs respond to an unannotated risk answer as strongly as when explicitly instructed that the answer is very reliable.p.1
4. Advice instability is unevenly distributed across demographic groups and is largest for clients under 40 years of age.p.1
5. System prompts warning models about measurement noise pull cautious clients toward generic middle allocations rather than tailoring to stable financial facts.p.1
6. Adding three supplemental investing questions reduces raw advice swings by 1.5 to 5.4 points for six models but narrows overall advice spread across clients similarly.p.11
7. Scripted clarifying conversations effectively isolate client misunderstandings from genuine preference changes without sacrificing responsiveness to deliberate updates.p.1