Methodology & Evaluation
I Already Knew the Answers. The Question was Whether AI Could Find Them.
The executive problem wasn't confined to marketing, admissions, finance, or operations. So I designed the analytical environment around the relationships between them. The Evidence Engine had access to a relational representation of Harbor Ridge spanning acquisition, inquiry, financial qualification, admission, treatment and revenue. I wanted the analysis to distinguish different kinds of performance problems rather than assume that declining admissions automatically meant an admissions problem.
Freezing the Rules
For the blind evaluation, I went further. I froze Ground Truth before AI analysis began. The answer key was based on the actual generated scenarios, and the rule was explicit: what counted as the correct answer could not change based on what the AI later found or missed.
That created a clean separation between two questions: did the analysis surface something useful? and did it recover the deliberately hidden problem it was being tested to find? Those questions eventually produced different answers.
Designing the Blind Evaluation
After the first blind-analysis attempt produced mixed results, I redesigned the evaluation. For V0.2, I froze an Evaluation Interpretation Protocol defining Pass/Partial/Miss for known scenarios and Clean/Borderline/False Positive for a Healthy Baseline negative control.
The discovery procedure was designed without Harbor Ridge context. I subjected it to two independent AI-assisted genericity audits, one by Claude and one by ChatGPT. Both returned 10/10 PASS. Only then did I freeze the procedure and run three fresh, isolated blind sessions: Scenario 1, Scenario 2, and the Healthy Baseline. The outputs were frozen before scoring.
The Result
- Scenario 1
- MISS
- Scenario 2
- MISS
- Healthy Baseline
- CLEAN
The misses were not cases where the system simply found nothing. In both scenarios, it found and prioritized other patterns that substantially contradicted the frozen Ground Truth. The Healthy Baseline remained clean, which is useful evidence about that control test. It does not prove general reliability.
I designed the evaluation so the Evidence Engine could fail without letting me redefine failure afterward. Then it did.
What It Means and What It Doesn't
The specific V0.2 discovery procedure was insufficient to reliably recover the previously demonstrated localized failures under the existing Harbor Ridge experimental conditions.
That conclusion is intentionally narrow. The evaluation showed that this procedure did not reliably recover the planted failures under these conditions. It did not erase the useful findings the analysis surfaced elsewhere, and the Healthy Baseline's CLEAN result did not establish general reliability.
A subsequent diagnostic examined why the misses may have occurred and found evidence consistent with signal-to-noise structure in the experimental environment. That analysis was informative, but not decisive. I therefore preserved the benchmark result rather than changing the evaluation after seeing it.