What institutions should evaluate before adopting an AI tutor

Begin with the learning job

The phrase “AI tutor” can describe very different products. One may answer general questions from the open web. Another may work only with institution-approved course material. One may focus on practice exercises, while another helps students navigate lecture recordings.

Start by defining the job you need the product to do. Is the priority helping absent students catch up, reducing repeated questions after class, making technical lectures searchable, or giving learners a structured revision path? A clear job prevents a visually impressive chat experience from becoming the goal by itself.

The job should be specific enough to test. “Improve engagement” is difficult to evaluate in a pilot. “Help a student find the explanation of a concept across four recorded classes” gives the institution a concrete task and a visible outcome.

Inspect the knowledge boundary

Ask what the tutor is allowed to use when it responds. For course-specific support, the boundary should be visible. It may include lectures, transcripts, slides, notes, or approved resources. The institution should understand who adds that content and how it is associated with a course.

A defined knowledge boundary helps responses match the teaching context, but it also creates responsibilities. Outdated or incorrect material needs a path for review. Courses with several instructors may require clear ownership. Content that should not be available to a particular cohort needs appropriate controls.

Test what happens outside the boundary. If a student asks about material that has not been taught, does the product say that it lacks context, produce a generic answer, or redirect the question? The behaviour should match the institution’s expectations.

Look for source visibility, not only answer quality

A fluent answer can feel convincing even when it is incomplete. Source visibility gives learners and educators a way to inspect how the response relates to the course. The product should make the relevant lecture or material easy to open.

During evaluation, check whether citations lead to meaningful context. A timestamp should open near the relevant explanation. A document reference should identify the useful section. The learner should be able to tell where source material ends and generated synthesis begins.

This is not only a risk control. It is part of the learning design. Returning to the source helps students recover the surrounding explanation, examples, and assumptions that a short answer cannot contain.

Evaluate the path from answer to evidence. A source badge is useful only when the learner can inspect what sits behind it.

Evaluate educator control and everyday operations

A pilot often begins with a carefully prepared course. Production use is messier. Classes are rescheduled, recordings fail, content is revised, and educators have different working habits. The product needs an operating model that can handle those realities without creating a new administrative burden.

Ask how recordings enter the system, how a teaching team reviews material, how students gain access, and how content is corrected or removed. Understand which steps are automatic and which require an owner. Consider how the workflow fits the institution’s current learning platform rather than assuming staff will manage a separate content island.

Educator control should be practical. If every generated note requires a lengthy approval process, the system may never scale. If nothing can be reviewed, the institution may not have enough oversight. The right balance depends on the course and learner context.

• How are classes and course materials added?
• Who controls what learners can access?
• How are transcription or content errors corrected?
• How is outdated material removed from future answers?
• What happens when a recording or integration fails?

Test with a realistic, bounded pilot

A useful pilot includes real teaching material, a defined learner group, and a small set of learning tasks. It should include ordinary content, not only the cleanest lecture. Technical vocabulary, varied speaking styles, questions outside the material, and references to visual content all reveal how the system behaves.

Collect feedback from students and educators separately. Students can describe whether the product helps them find and understand material. Educators can judge whether answers reflect the course and whether the workflow is manageable. Programme teams can assess administration and support.

The decision should not rest on a single accuracy score or a generic claim. Look at relevance, source visibility, learner clarity, educator control, operational fit, and the product’s behaviour when it does not know. An AI tutor becomes useful when it strengthens a learning workflow the institution can actually own.