Learning Companion Agent

Why this exists

When I was preparing to write an AI companion requirement at work, I found many key questions a flowchart cannot answer: after a student photographs a problem, should the system explain immediately or first judge whether they have already answered; when a student keeps asking for the answer, how does the agent avoid being adversarial while still protecting the learning process; when do textbook knowledge, conversation history, and long-term memory enter context; when a tool fails or evidence is thin, how does the final reply degrade honestly.

So I did not start with an idealized PRD. I built the companion agent first. It is a working product-experiment prototype, meant so product, algorithm, and engineering can see real dialogue, state change, and failure modes before deciding what a formal product needs for capability and limits.

View the GitHub repository

From a chat box to a learning workbench

The companion agent started from free discussion, then gradually formed several learning forms that can be switched and continued:

• Free discussion: judge whether a textbook, memory, or an interactive resource is needed; answer directly when a tool is not needed.
• Photo explanation: recognize the item structure in one image, form an item list, then enter guided explanation item by item.
• Textbook preview: locate specified textbook content, generate 3–5 learning tasks unlocked in order, and save progress.
• Mind map: generate a knowledge structure from textbook evidence, then support follow-up questions from the map.
• Session and data views: keep multi-session, dialogue summaries, learning insight, long-term memory, and per-turn token / call information.

The change in product form matters: a learning agent is not only message bubbles. Item lists, task cards, mind maps, image annotation, and session state are interface evidence that the student understands “what the system is doing.”

Photo explanation: first understand which step the student is on

“Upload a problem — the model gives the answer” is fast to build and thin in educational value. I split photo explanation into intake, item split, choosing the current sub-item, recognizing student state, extracting knowledge points, round-by-round coverage, and ending the explanation.

The system has to tell unanswered, existing draft, supplementary material, a new item, and an unrelated image apart. For a student who has not answered, the first round is about the smallest actionable hint; for a student who has submitted a solution, the focus turns to checking reasoning and locating the error. When the student is stuck, hint strength can rise, but the agent still does not answer for them just because they keep asking.

That turned “Socratic teaching” from a prompt sentence into observable product state: current item, student stance, hint level, covered knowledge points, and exit conditions all have to be recorded and verified.

Textbook tools: RAG is not the default answer

The prototype has both vector retrieval and a tool that fetches OCR original text by textbook structure. After using it, the boundary was clear: for textbook preview, mind maps, and questions about a specified page, the main path should be deterministic fetch_textbook; vector retrieval is only auxiliary recall, and “semantically similar” cannot replace a specific textbook fact.

Tool results also have to be judged usable, missing parameters, wrong target, or irrelevant. Finding text is not the same as finding the right content. When evidence is unusable, the agent should add parameters, clarify the target, or say so plainly — not fill a fluent answer from training memory.

Memory: what to keep matters more than remembering more

I split memory into short-term dialogue context, reusable long-term facts, and learning insight formed after a conversation. Long-term facts must have a source, a scope, and a way to be corrected; an inference the model generated in one conversation cannot automatically become a stable fact.

That also changed evaluation. Memory is not only “can it be recalled.” It also has to verify restraint when it should not write, whether duplicates are merged, how conflicts are handled, and whether the final answer actually used the evidence that was retrieved.

Freeze product behavior before a refactor

When the prototype moved from several independent flows toward a unified agent runtime, I did not replace the old implementation with a new architecture immediately. I first inventoried F01–F23: for each capability, what is the entry, API, SSE event, storage, interface, and existing tests.

The audit found the capability table in the docs was already behind the code; multi-session, manual image annotation, and Context / CallSpan monitoring were real product behaviors added later. Unit-test coverage also does not equal end-to-end behavior. A runtime refactor still has to verify the whole chain of prompt, model adapter, events, storage, and web UI.

So this page will not describe Runtime V2 as done. It better shows how I work: establish a behavior baseline and acceptance language first, then discuss whether the architecture is better.

What I learned

1. Education-agent quality is a multi-dimensional constraint

Besides a correct answer, measure tool-choice accuracy, non-invocation accuracy, boundary discipline, parameter quality, factual grounding, and memory-write quality. Average correctness cannot cover a high-risk failure such as “answering for the student.”

2. Multimodal is not “pass one more image”

An image changes intent, session, and UI: it may be a new item, a student solution, supplementary material, or unrelated. A reliable flow has to recognize the image’s role in the current session before deciding whether to split items, continue explaining, or ask for confirmation.

3. Wrap open generation in a deterministic skeleton

Textbook location, item state, tool execution, result checking, and session persistence belong to a deterministic system; the model is better at semantic judgment, teaching language, and local planning. The clearer that boundary, the easier the product is to debug and evaluate.

4. A prototype’s value is exposing failure, not only demoing the happy path

Real dialogue produces recognition errors, missing tool parameters, the wrong textbook target, student pushback, and model over-explanation. Keeping those failures and tracing them to a concrete chain helps form a formal requirement more than a demo that is always right.

What it proves

The companion agent shows I can take a vague “build a learning agent” and split it into working product capabilities, tool and data limits, then converge the requirement through real use, code audit, and eval design. It is also a representative experiment of going from AI product design into agent runtime, context, memory, and evals.

Scope note: This is a product experiment and engineering prototype, not described as a commercial product. Screenshots are from the current local version. Runtime V2 is still an uncommitted refactor; this page treats the implemented baseline and the refactor method as evidence, and does not claim the migration is done.