What AI missed when planning an auth migration across ten applications

A closer look at What AI missed when planning an auth migration across ten applications

What this post documents

A staff-level planning evaluation based on a real authentication migration for a SaaS platform with about ten client applications. The author converted their migration into an eval to see how state-of-the-art agent models would plan the change.

The write-up reports results from running multiple models against a fixed rubric and the author’s own implementation experience.

What you can actually read and learn here

A summary of the migration context: coexistence of old and new auth, diverse token mechanisms (cookies, redis sessions, negotiated JWTs, long lived tokens), organisation-level accounts, permissions, revocation, impersonation, and hard-to-update clients like mobile.

How the evaluation was run: code snapshot inspection only, a product brief and starting code given to models, an outcome-based rubric with 18 categories, 72 checks, 100 weighted points and 19 critical checks, and comparison of model plans to those outcomes.

Who will find this useful

Staff or senior engineers planning complex auth migrations who want concrete examples of common pitfalls and what an automated agent missed.

People curious about practical limits of model-generated planning for engineering work, especially around subtle issues like rollback assumptions, account ownership (email vs IdP), and edge-case authenticated flows outside normal HTTP requests.

Questions people ask about What AI missed when planning an auth migration across ten applications

Which models were evaluated and how did they score?

The post mentions multiple models; the best-performing model (named Astra) scored about 54.75-55% and the other two clustered around 33%.

Did the author let models access the internet during the eval?

The evaluation was sandboxed to code snapshot inspection only; one submission disclosed web access despite the no-network protocol.

Does the post give a tested migration plan I can copy?

The author describes their implementation experience and the evaluation approach, but the post focuses on lessons, rubric, and where models missed critical concerns rather than a step-by-step copyable migration checklist.

What AI missed when planning an auth migration across ten applications was shared on NexURLs by Nexis on November 1, 2026, in Tech & AI.