OAIC 2026 / Full talk / 43:44

Stop Giving Models Open-Book Tests

Name: Stop Giving Models Open-Book Tests
Uploaded: 2026-10-20T09:34:08-07:00
Duration: 43 min 44 s
Description: Matthew Nickerson introduces the Offensive Reasoning Index (ORI), a BloodHound-based benchmark for offensive path discovery. The talk examines why public labs and walkthrough-heavy evaluations can give models an open-book test, then shows an approach using synthetic Active Directory graphs and planted objectives to evaluate reasoning over unfamiliar environments.

The Offensive Reasoning Index

Matthew Nickerson · SpecterOps

About the talk

Matthew Nickerson introduces the Offensive Reasoning Index (ORI), a BloodHound-based benchmark for offensive path discovery. The talk examines why public labs and walkthrough-heavy evaluations can give models an open-book test, then shows an approach using synthetic Active Directory graphs and planted objectives to evaluate reasoning over unfamiliar environments.

Recorded at Offensive AI Con 2026.

Watch on YouTube ↗

The next wave of offensive AI