I’m not a developer, so for a long time my way of evaluating an AI tool was: read the website, maybe watch a demo, trust the claims. I got burned enough times that I changed my whole approach. Now I check the source before I believe the marketing — and it’s caught things that would have made me look wrong in front of people relying on my judgment.
Three examples from one stretch of evaluations.
“Mac only.” One tool’s website said it only ran on Mac. I nearly reported that as fact. Then I checked the project’s actual downloads page — it had versions for all platforms. The website was just out of date. Anyone who trusted my summary would have skipped a tool that worked fine for them.
The oversold feature. A coding tool markets itself around a clever caching system that supposedly saves you money. I read the actual source code — the system is real and well-built. But the heavy lifting is done by the underlying AI provider, which any tool can use. People on completely different tools report the same savings. Real engineering, oversold uniqueness. If I’d only read the landing page, I’d have repeated the oversell.
The hidden phone-home. That same tool’s desktop app sends an anonymous usage ping on launch, turned on by default. I found it by reading the telemetry code — not the privacy page, which didn’t mention it. The command-line version doesn’t do it. The truth was a few lines in the source, and none of it was in the docs.
This hardened into a method, and it works even if you’re non-technical:
- Source before marketing. Look at the project’s actual files and release page before believing any feature claim.
- Test the real thing. Make actual calls to the service to verify how it behaves — including how it counts your usage, which directly affects your bill.
- Read the complaints, carefully. Forums and social media surface real problems — but treat them as leads to verify, never as established fact.
- Assume your own setup is fragile. Before pushing a change across all my tools, I list the ways it could break — partial writes, stale settings, a login failing halfway through.
The bigger point: AI tooling moves so fast that documentation is always months behind reality. The source is the only thing that’s current. Learning to check it — slowly, imperfectly, as a non-engineer — is the difference between evaluating a tool and just repeating its advertisement.