Artificiality's First Exam: An LLM-Referenced C-test for Supraintelligence
Author1, Author2, Author3, Author4, Author5
Abstract
This paper presents a novel benchmark for evaluating AI systems that may surpass current capabilities by utilizing Levin's Kt to generate high-difficulty test items for large language models.
Reality Card
The paper introduces the first test based on Levin's Kt using LLMs, enabling the generation of high-difficulty items that can be certified before being solvable by current AI or human testers.
A battery of certified items was administered to twelve LLMs, revealing agent characteristic curves that decrease with difficulty, indicating a new high-difficulty testing region.
The main limitation is the potential difficulty in reproducing the high-difficulty items and ensuring consistent certification across different LLMs.
Paper to code
Verified implementation resources so builders can test the paper’s claims instead of stopping at the abstract.