Tom Braack:
Mirror Mirror on the Side: Help to Think, Right? Implementation of, Experimental Design and Feasibility Study to Evaluate LLM Socratic Questioning to Promote Critical Engagement in LLM-Assisted Dashboard Workflows
Contents
This thesis implements a Socratic Questioning intervention at decision points within an LLM-assisted data visualization workflow, the Grafana Assistant, and designs a between-subjects experimental protocol to evaluate its impact on self-reported enaction of critical thinking. A feasibility study (n=7) was conducted to validate the technical architecture and procedural design under realistic conditions. While the study confirmed the protocol is runnable, it identified risks and validity threats, including infrastructure instability, unintended unblinding through CoT output and Socratic Questioning on hallucinated matters. In consequence of the scope, no claims regarding the effectiveness of the Socratic intervention are made. Instead, this work contributes a tested, empirically refined protocol and prompt-based implementation of LLM Socratic Questioning as a cognitive forcing function for future evaluation.

