Structured Prompting and Automated Evaluation in Fixed Synthetic Japanese-Language Counseling Dialogues
arXiv:2507.02950v3 Announce Type: replace-cross Abstract: Large language models (LLMs) may support counseling training, yet evidence from Japanese-language interactions and automated quality ratings remains limited. We examined 18 fixed Japanese-language counseling transcripts generated through artificial intelligence (AI)-to-AI interactions under three counselor conditions: GPT-minimal (GPT-4-turbo with a minimal role instruction), GPT-SMDP (GPT-4-turbo with the Structured Multi-step Dialogue Prompt [SMDP]), and Claude-SMDP (Claude-3-Opus with SMDP). Fifteen counseling experts rated transcrip
The proliferation of advanced LLMs necessitates robust evaluation methods, particularly for specialized applications and non-English languages, making this research timely.
This research provides a framework for structured prompting and automated evaluation in AI counseling, which is critical for developing trustworthy and effective AI applications in sensitive domains, especially across diverse linguistic contexts.
The systematic comparison of LLMs with structured prompting in a specific, high-stakes application like counseling, alongside expert evaluation, advances the methodology for AI-to-AI interaction assessment and model fine-tuning.
- · AI developers
- · Healthcare AI companies
- · Japanese LLM developers
- · Counseling training institutions
- · Companies offering unvalidated AI counseling tools
- · Generic LLMs without specialized prompting
Improved methodologies for evaluating and deploying LLMs in specialized, culturally and linguistically specific applications like mental health support.
Increased trust and adoption of AI-powered tools in sensitive sectors, leading to a demand for more localized and validated AI solutions.
The acceleration of AI application development in non-English speaking markets, potentially reducing reliance on models primarily trained on English datasets.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.AI