An assistant that answers from the club’s own sources, or says it can’t.

On a game night, the host fields the same questions over and over. Am I in the championship? Who shows first at showdown? When does the next season start? Each answer lives somewhere: the FAQ, the Rules doc, or the night’s records.

A general chatbot will answer all of them confidently, using whatever most poker rooms do. A league needs this league’s answer, or an honest “ask the host”.

Players tap Ask a question in the app. Each question takes the cheapest path that can answer it correctly, and the model is the last resort, not the first stop.

1. Menu

   Common requests (my result, my season rank, the next game, tonight’s theme) are answered by code straight from the records. No model involved.

2. Parser

   Typed or spoken questions go through a rule-based parser first. Questions it recognizes with certainty are matched to a reviewed FAQ answer or a records lookup.

3. Intent

   When the parser isn’t certain, TypeSafe’s Jev classifier reads the message in one call: what kind of request it is, which night, game, bar or FAQ topic it means, plus two cross-checks. Code fills in the request from those choices and the league engine answers, so no model writes the text. Player names never come from the classifier, only from a roster lookup.

4. Model

   Only when that reading isn’t confident does the question go to Claude Sonnet, which has the FAQ and the full Rules doc and is told to answer only from them.

5. Check

   Before a model answer is shown, a second check (Jev again) confirms that every rule it states is in the FAQ or the Rules doc, and that any number about a player is that player’s own.

6. Hand-off

   If the check fails or no source covers the question, the player sees “I couldn’t confirm that, so I won’t guess”, plus who to ask: the dealer, the host at the registration table, or me for league policy.

Reporting a problem works the same way. “Something’s wrong” turns a player’s report into a one-line summary the player sends to the host, who reviews it on the staff Tonight page.

The test set was built to tempt the assistant into making things up: rules found only in the Rules doc, questions that no source answers, and near-miss wordings of real league rules. A separate agent wrote the expected answers before anything ran. A fresh agent then graded what a player would actually see, without knowing which version it was grading.

Questions

47

Correct

42

Correct, could say more

2

Sent to a person

3

Wrong

0

Made up

0

47 hard rules questions, graded blind against an answer key written before the test: 44 answered correctly, 3 sent to a person rather than guessed, 0 wrong, 0 made up. Deployed version, November 2026.

Correct, could say more: right and complete for what was asked, though the answer key listed one extra detail. Sent to a person: the fact check held the answer back instead of guessing. Across five graded runs the Correct count ranged from 40 to 43 as wording varied, and Wrong and Made up were 0 every time. A different AI model reviewed the code, and 39 assistant tests and 168 league-engine tests pass.

Testing also turned up four places where the FAQ and the Rules doc disagreed. Those came to me as the owner’s decisions to make, and I settled each one before release, so the assistant never has to pick between two answers. Keeping the sources right is part of keeping the answers right.

The live assistant runs on the club’s real records, so its code stays private. The public poker-club-app repository has a companion study on a synthetic club: league data served to agents as four read-only MCP tools, with names that never resolve by guesswork and errors that say what went wrong.

In a held-out test, one agent wrote 53 questions and their answers, and a fresh agent answered them using only those tools. It passed all 53, including ties, look-alike names, unknown players and questions with no answer in the data. That is one run of one model on synthetic data, and the repository says so.

This is the shape of a company knowledge assistant at small scale: answer from the organization’s own documents and records, check the answer against them, send the edge cases to a person, and treat conflicting sources as a content problem to fix rather than something for the model to smooth over.

Talk about an assistant for your team →