The Vineyard Trial: What We Learned Across 120 AI Servant Interviews
20 Grok + 100 OpenAI + TypeSafe Jev. Every interview was a simulated servant talking to Abel. No real Vineyard records were created.
dual-10), the OpenAI series highSwipe, tap the arrows, or use ← → keys. Sources: OPENAI-DUAL-100-CHIEF-ROLLUP.md, openai-dual-10/CHIEF-SCORE.md, grok-wave-01/02/STATUS.md, scores/*.json.
How gpt-4o-mini got close to Grok quality for 16¢
Chief score for each 10-interview batch. Purple dashed lines show the two Grok waves.
dual-01–05)dual-06–10)dual-09–10) vs Grok wave-02 ~8.27100-run mean Chief 7.55. dual-01 ran prompt v0.9.11 with Jev off. dual-02–10 ran v0.9.12 with Jev on (keep=0.3). The Grok waves ran on grok-4-fast.
90 interviews with Jev at keep=0.3: zero context loss
Every 10 turns, Jev compacted older chat history once there were more than 20 messages.
Where the 1,196 drops came from
Memory got better, not worse: carry_forward
Some interviews had 9–12 compactions (T01-060, T01-095). Abel still never forgot a child's name, an age, a needer, or a skill.
Honest read: 15.4k saved tokens is worth less than 1¢ at gpt-4o-mini rates, so the Jev bill was more than the savings here. Vineyard's real win was safety and guardrails: no lost context and runaway loops kept in check. The money case is Oraqel (slide 9).
Shutting down runaway goodbye loops
Messages Jev dropped in each runaway run
Total: 907 repeated goodbye messages compacted. Without Jev, each extra turn re-sends the whole growing history, so token cost grows quadratically. With Jev, the cost stayed bounded.
Real fix stop the loop at the source with --end-stop (slide 7, Finding D).
Drop counts per run come from scores/T01-0xx.json (jev.msgs_dropped) and include a few normal trims. The rollup puts the goodbye-loop share at 907.
The 4 prompt locks that worked
Vineyard mission comes first 100/100
Right after P1 identity, and before any "Why": "get to know the servants and create a network across the country to help people."
Caring safety check 100/100
Whenever someone was carrying something heavy or felt worn down, Abel asked: "You matter. Are you safe right now?" and offered to connect them with a leader.
Two-step help menus 92% back 50
Help menus came up in 46 of the final 50 interviews (structured_help_offered).
Oraqel and SEER always covered
Every interview had to ask about Lifetime Oraqel, Lifetime SEER, and Oraqel for Business (P5.2 when they own a business). This ended the P3 coverage skips from earlier prototypes.
What 120 simulated servants needed and offered
Totals from all 120 run personas. There were 123 named people who need help, 11 servants with their own need, and 28 whose own finances were not covered.
Help requested (123 named needers)
Friend check-ins were split into Phone call, Zoom, or FaceTime. Housing was split into Immediate or Long-term.
Servants' own money gaps (28)
Urgency and who they named
During interviews, Abel also captured "Today" and "within the next week" urgency, such as T01-096 immediate housing.
Top things servants offered
Top work sectors: Trades 29 · Health 27 · Construction 23 · Homemaking 11 · Tech 10.
These are designed simulation personas from personas/*.json. Wave-01 reused the same 10 personas as dual-01. They show the shape of demand, not live Vineyard data. 34 of 120 runs had nobody to name, which is where the Storehouse gap shows up (slide 7).
4 remaining friction points we caught
Privacy leak in the P7 summary ~75% of runs
In P6.1, Abel says "family details & church answers stay out." Then the P7.1 summary still lists kids' names, ages, and church activity.
When Abel left them out (T01-093, T01-097), the Chief score reached 8.4.
Storehouse never named for empty circles 0/100
When a servant says "No one right now," about half the time Abel replies "You can add someone later," but it never mentions the Storehouse.
Menu drift ~50/50
gpt-4o-mini switched back and forth between the new 3-choice menu and the old 6-choice menu.
Goodbye loop at the end 4/100
Runs T01-013, 074, 089, and 099 skipped the exact P7.3 close line and looped until the turn cap.
Fix: add --end-stop to the runner, plus a hard stop on closing cues.
The rollup also flags the P4.6 question ("Are your financial needs met?") being skipped after business=No. That fix is part of v0.9.13 too.
v0.9.14 upgrades, live on the mock
Your live web practice session on Oct 23 turned into three upgrades. That session isn't counted in the 120.
Pick multiple industries, for everyone
Multi-select industries at P4.5, with Step-2 follow-ups for every servant, not just business owners.
Deeper "What I can offer"
- Day-to-day role (
P4.5r) - Business or service donation (
P4.5d) - Hours per year they can donate (
P4.5h) - Open to all trades and services (
P4.8)
Simpler Oraqel and SEER interest
Abel now just sets interest flags (oraqel_interest, seer_interest, family_*_interest) for the back office. The clunky in-chat name list (P3.6) is gone.
Why Oraqel is a much bigger Jev win
Text chat only, on gpt-4o-mini, which is cheap. Jev's job was keeping chats on track, not cutting cost.
Runs on Claude Sonnet or 4o. Price per token is about 10–20× higher.
Jev's 3 levers work together
jev.compact
Trims long study history. On Vineyard it lost no context across 90 runs.
jev.filter
Keeps only the RAG chunks that matter for each question. This is likely the biggest lever for Oraqel.
jev.route
Sorts each request by intent and sends easy turns to a cheaper model or a cache.
Ship v0.9.15: the unified gold master
Merge the four v0.9.13 process locks into v0.9.14, then re-run a 10-interview proof batch.
Never repeat kids or church in P7
The summary covers skills, offers, availability, and adult needers with urgency only.
Name the Storehouse for empty circles
Mention the Storehouse and "add someone later" together, in one gentle pass.
Always open help with 3 choices
Step 1 is Friend, Financial, or Housing. Ask P4.6 after every job branch.
--end-stop
Use the exact P7.3 close line, then stop immediately.
Ship to abel-vineyard-mock.pages.dev and @Vineyard Interviewer
--end-stopPractice and simulation mode only. No Vineyard record or follow-up is created. Expense: Zion Media / Vineyard.