Otto: Homepage Simulation Study

What we were hired to find out

Otto is an AI-powered accounting platform built specifically for self-employed creative professionals. It handles bookkeeping, taxes, invoicing, and contracts. Before investing in paid traffic or conversion rate optimization, the team wanted a baseline: how does the homepage land with the target audience, what's working, what's blocking signups, and what's the highest-leverage place to improve?

We ran a baseline simulation using 20 synthetic personas, then iterated across six more rounds — improving the page in some rounds and fixing our own measurement system in others. What we found about the methodology turned out to be as valuable as what we found about the page.

How the simulation works

Each persona is a synthetic profile built from demographic and behavioral patterns matching Otto's target market: self-employed creatives, 1-10 years in business, variable income, doing their own finances or relying on spreadsheets. Each persona reads the homepage text and answers structured questions about first impression, comprehension, signup intent (1-10), and what would make them more confident.

A separate judge model then reads all 20 responses and produces a structured verdict. Rounds run sequentially on the same 20 personas for consistent comparison.

Seven rounds: what changed and what we learned

* v5 avg corrected manually after identifying extraction bug; automated avg was unreliable that round.

Three bugs that were suppressing every verdict

The most important finding in this study wasn't about Otto's homepage — it was about our own measurement system. After five rounds of flat "Getting There" verdicts despite significant page improvements, we ran a debugging session on the simulator itself. Three compounding bugs surfaced.

We also ran a control test: gave the judge a fabricated perfect scenario (all 20 personas at 10/10, zero objections) and confirmed it could return "Ship It." Then tested what score threshold triggered "Nearly There" — it required approximately 7.5/10 average with remaining gaps framed as "want to try it." This calibrated our expectations for what the verdicts actually mean.

The changes that actually worked

Story cards (v5): biggest single page lift

Replacing three generic star-rating testimonials with before/after story cards — each with a specific dollar outcome, named persona, and quote tied to that outcome — was the highest-impact page change across all rounds. The judge noted the cards in every subsequent verdict. By v7, 17/20 personas found a story that felt written for them.

"The before/after user stories made me more likely to sign up because they show real improvements and savings, but I need to see more case studies specific to my industry."

Priya Mehta, UX/UI designer (v5 response)

Profession-specific grid (v7): recognition, not conversion

The "Who Otto is built for" grid with six creative type tiles (photographer, designer, content creator, consultant, artist/maker, service pro) increased story match rate to 17/20. But it didn't move intent scores substantially — recognition alone isn't conversion. Personas who saw their profession in the grid still needed to see their expenses in the story card to feel understood.

Accuracy stats (v7): partially addressed, still a gap

Pulling the 94% / 100% / 168 corrections stats out of the FAQ and into a prominent blue band addressed Priya and Tyler's specific confidence gap. But three personas still cited AI accuracy as their primary hesitation even after seeing the stats. Stats alone don't build trust — they set a floor. Stories and product experience build on top.

14-day trial CTA (v3): structural fix that stuck

Adding a 14-day free trial option for personas earning above the $25K free tier threshold was the single structural fix that addressed the most-cited baseline objection (18/20 had flagged "no free trial"). This change held through every subsequent round and is reflected in the final page.

Free tier contradiction fix (v5): small but necessary

The free tier callout box listed "human bookkeeper review" as a free feature while the pricing cards showed it as paid-only. Fixed in v5. Small change, but this kind of inconsistency is caught by careful readers and damages trust disproportionately.

Who converts and who doesn't

Where the page ends and the product begins

Across seven rounds, the most common remaining gap — even from personas who found their story, understood the pricing, and said they'd click the CTA — was some version of: "I'd feel fully confident after trying it." That's not a copywriting problem. That's the free trial being the conversion mechanism.

A 40% CTA click rate (8/20) from a cold read of a homepage is strong performance for a freemium product. Real-world cold traffic converts at much lower rates than synthetic personas because personas don't have competing tabs open, don't get distracted, and read every word. 40% simulation likely maps to 8-12% in production, which is healthy for a free trial offer.

The remaining 60% are not lost — they're pre-trial. Their confidence gap is "I want to see it work for me," which is exactly what the free tier delivers. The page's job is to get them to click, not to eliminate all uncertainty before the click. It's doing that job.

What to do next

Ship v7

The v7 prototype is the deliverable. It outperforms the original on every measurable dimension: free tier clarity, trial availability, social proof depth, profession recognition, AI accuracy transparency. Replace or inform the current homepage with v7.

Add a musician / audio producer story card

Carlos Vega (music producer) was the clearest unmet need in v7. No story on the page speaks to his expenses: DAW software, sample packs, studio time, sync licensing income. One additional story card targeting audio/music professionals would address this gap. It's the only creative type in Otto's target market that v7 doesn't cover.

Consider a 60-second product walkthrough video

The most common confidence gap across all rounds was "I want to see it work." Video is the closest a static page can get to product experience. A 60-second screen recording of Otto categorizing real transactions — with a correction shown in real time — would address the AI accuracy hesitation that copy alone cannot fully resolve.

Post-launch: rebuild social proof with real user stories

The v7 story cards are fabricated to represent the target audience. As real users accumulate, replace them with actual before/after stories using the same structure: profession label, problem before, specific outcome, quote. One real story with real numbers outperforms three constructed ones.

What we'd do differently from round one

This study included a mid-run methodology audit that should be standard practice in future simulation work. Key lessons:

Always include a binary CTA question. "Would you click the primary CTA right now?" gives a cleaner conversion signal than intent scores alone and cuts through hedging language.

Never ask for the "single biggest thing stopping you." This prompt guarantees every persona finds a blocker, even if they'd convert. Ask what's missing, not what's wrong.

Test the judge with control scenarios before trusting its verdicts. A "perfect" scenario (all 10/10, no objections) should return "Ship It." A strong scenario (7.5/10, "I want to try it") should return "Nearly There." Calibrate before running real rounds.

Validate intent extraction before running. Print 3-4 sample responses and manually check that the regex captures the actual score, not a number from the prompt format.

Verdict labels are noisy at the margin. The quantitative metrics (avg intent, CTA rate, story match rate) are more reliable than the label. A "Getting There" at 6.95 with 40% CTA is performing better than a "Nearly There" at 6.35 with 25% CTA.

Every prototype, in order

Click any version to open the full HTML prototype in a new tab.

* v5 avg corrected manually after identifying extraction bug. Automated avg that round was unreliable.