NeurIPS @ Cambridge

Panel session

The future of AI policy: anticipating challenges and driving positive change

Panellists: Emily Shuckburgh, Jess Montgomery, Neil Lawrence, Mateja Jamnik

Coffee

Keynote seminar

Anders C Hansen, Dept of Applied Maths and Theoretical Physics, University of Cambridge

Talk title: The consistent reasoning paradox, hallucinations and fallibility of super AI: The power of 'I don't know'

Abstract: We introduce the Consistent Reasoning Paradox (CRP), which applies to any artificial super intelligence (ASI) (surpassing human intelligence). Consistent reasoning, at the core of logical reasoning, is the ability to handle questions that are equivalent, yet described by different sentences ('Is 1 > 0?' and 'Is one greater than 0?').  The CRP asserts that any ASI, because it must attempt to consistently reason, will always be fallible — like a human. Specifically, the CRP states that there are problems, e.g. in basic arithmetic, where any ASI that always answers and strives to reason consistently will hallucinate (produce wrong, yet plausible answers) infinitely often. The paradox is that there exists a non-consistently reasoning AI — which is not on the level of human intelligence — that will be correct on the same set of problems. The CRP also shows that detecting these hallucinations, even in a probabilistic sense, is strictly harder than solving the original problems, and that there are problems that an ASI may answer correctly, but it cannot provide a correct logical explanation for the answer. Therefore, the CRP implies that any trustworthy AI (i.e., an AI that never answers incorrectly) that also reasons consistently must be able to say 'I don't know'. Moreover, this can only be done by implicitly computing a new concept that we introduce, termed the 'I don't know' function — something currently lacking in modern AI. In view of these insights, the CRP provides a glimpse into the behaviour of ASI. An ASI cannot be 'almost sure', nor can it always explain itself, and therefore to be trustworthy it must be able to say 'I don't know’.