🧪 Test?View on arXiv
RAW: Refusal Aversion in the Wild, A Causal Measurement Method for Deployed LLMs
Not provided
user engagementLLM evaluationcausal inference
2611.21641
Builder Relevance
15h ago80%
Abstract
The paper introduces RAW, a method to measure the user-side cost of refusal in deployed LLMs, revealing significant impacts on user re-engagement.
Reality Card
Core Claim
The RAW method allows for the causal measurement of user disengagement due to refusal in deployed LLMs, costing 3 to 17 percentage points of subsequent re-engagement for about half of the models tested.
Method / Result
The method runs on any deployed model from logs alone for about $0.30 per 1,000 conversations.
Limitations
The method's applicability is limited to existing logs and may not be generalizable across all types of LLMs or user interactions.
Paper to code
Verified implementation resources so builders can test the paper’s claims instead of stopping at the abstract.
No verified implementation link has been attached yet. AIBuzzHub will keep this panel separate from unverified search results.