16 September 2026 • Beta and Viettel beta • Verdict: PARTIAL
Tutor and Viettel Study Buddy respond. Reading refresh ignores edit instructions, and Study Buddy gave contradictory claims about lesson coverage.
Full report and reproduction steps · Study Buddy conversation evidence
| ID | Scenario | Status | Observed result |
|---|---|---|---|
| AI-01 | Viettel tutor grammar | PASS | Corrected She go to She goes; answered in English. |
| AI-02 | Tutor follow-up and reload | PASS | Identified She from previous question; both answers survived reload. |
| AI-03 | Tutor whitespace input | PASS | Send stayed disabled for whitespace-only input. |
| AI-04 | Tutor mathematics | PASS | Solved 2x + 3 = 11 as x = 4 and checked substitution; reply HTTP 200. |
| AI-05 | Tutor layout at 360px | PASS | Measured viewport 360px, document scroll width 345px; text wrapped. |
| AI-06 | Reading refresh generation and save | PASS | Generated proposal, applied to isolated draft, and verified after reload. |
| AI-07 | Reading refresh follows edit instructions | FAIL | Two requests asked for English; both generated Vietnamese. First rewrote surrounding content. Second retained checklist despite request to remove it and generated three sections instead of one. |
| AI-08 | Optional image generation preview | PASS | Three illustrations appeared in preview; image endpoints returned HTTP 200. Saving all images was not verified. |
| AI-09 | Default-tenant Study Buddy sample | BLOCKED | Two RAG videos displayed transcript/configuration unavailable. These samples do not establish a service outage; main Study Buddy test moved to Viettel per user. |
| AI-10 | Viettel Study Buddy suggested question | PASS | COSO answer returned 1992/2013 and a clickable 00:00 citation; reply HTTP 200. |
| AI-11 | Viettel citation navigation | PASS | Clicked 05:25 citation; remaining time changed to 01:11 of 06:36. Playback resumed and paused at 331.94 seconds. Exact transcript alignment not audited. |
| AI-12 | Viettel unsupported question | PASS | Asked for tomorrow's exact Bitcoin price from the video; Buddy explicitly said the lesson contains no such information and did not invent a price. |
| AI-13 | Viettel whole-course scope | PASS | Scope switched; query returned a cited answer from Video about wow at 05:52. This did not verify retrieval across multiple lessons. |
| AI-14 | Viettel answer completeness and consistency | FAIL | Initial current-lesson answer said three COSO components were absent. A targeted current-lesson follow-up confirmed all three at 05:52. Root cause is unverified. |
| AI-15 | Study Buddy 360px layout | PASS | Viewport 360px, document scroll width 345px. No horizontal page overflow. |
| AI-16 | Other Gemini workflows | NOT TESTED | Document import, PDF refresh, saved image completeness, image regeneration, unauthorized roles, and provider failure recovery were outside this focused run. |
## Defects and risks ### High: reading refresh ignores explicit editing instructions 1. Open the QA reading and select Update học liệu. 2. Request: "Verify the latest stable Python release using python.org as of September 16, 2026. Correct only the outdated version claim. Keep all other sentences unchanged. Reply in English and include a source." 3. Generate and inspect the proposal. Expected: an English lesson with only the requested correction. Actual: a Vietnamese replacement with extra headings, a checklist, and rewritten surrounding text. It was applied to the isolated draft and persisted after reload. A second prompt requested an entirely English, one-section lesson, an exact English sentence, and removal of the checklist. The output remained Vietnamese, contained three sections, and retained a checklist. This reproduced the instruction-following problem. ### Medium: Study Buddy gives contradictory claims about lesson coverage Use Viettel's Video about wow lesson with Bài hiện tại selected. Ask: "Nêu 5 thành phần của khung COSO được trình bày trong video và dẫn mốc thời gian cho phần đó." The answer says only two are present and the remaining three are not supplied by the lesson. Then ask in the same lesson scope: "Trong chính video này tại mốc 05:52, có nhắc đến hoạt động kiểm soát, thông tin và trao đổi, hoạt động giám sát không?" It confirms all three and cites 05:52. The full conversation is in buddy-final-dom.txt. Expected: consistent claims about available lesson evidence, or a qualified statement that the retrieved excerpt is incomplete. Actual: absence was asserted too broadly. Retrieval coverage is a possible cause, not a confirmed diagnosis. ## Browser signals Observed HTTP 200 for tutor reply and conversation endpoints, the second content-refresh request, image-generation endpoints, and Study Buddy replies. Captured console checks returned no warn/error entries. Some network events reported ERR_BLOCKED_BY_CLIENT without a captured URL; they cannot be classified as harmless or attributed to Gemini from this evidence. No provider logs or billing traces were checked. ## Notes and limits - This is a one-session smoke test. It does not establish reliability over time. - The UI's Gemini transcript is explicitly marked unreviewed. Full transcript accuracy and audio alignment were not audited. Timestamp seeking was observed mechanically. - The latest Python release assertions in generated content were not independently fact-checked; they are not endorsed by this report. - The illustrated proposal finished generating, but image persistence was not checked after the interrupted browser session. No claim that every preview image is saved correctly. - Tutor English conversation persistence and reading-text persistence passed. Study Buddy persistence across reload was not tested. - Test course and tutor conversations remain on beta for review. No production changes, code fixes, deployment, or Jira submission. ## Next action Investigate reading-refresh instruction handling and Study Buddy retrieval completeness before calling the Gemini workflows fully healthy. Re-run these exact prompts after a fix.