Q1. Separation of Responsibility Between Model and Code
Choose one AI feature you recently implemented and explain which parts were handled by the model and which parts were guaranteed by standard code.
Besides improving prompts, what did you implement to enhance quality and reliability?
Q2. Handling Continuously Updating Knowledge
If you were designing a system to answer questions based on frequently updated internal documents with citations, how would you handle the following elements?
- Document chunking
- Retrieval
- Re-ranking search results
- Access control
- Reflecting updates and deletions of documents
Q3. Methods for Detecting Quality Degradation Due to Changes
When you change prompts, models, search methods, or tool definitions, how do you ensure that existing functionality hasn't broken?
What do you keep as test cases, and what metrics do you use to decide whether to release?
Q4. To What Extent Do You Trust AI-Based Evaluation?
When using AI as an evaluator, what kinds of biases or misjudgments occur?
How do you combine automated AI evaluation with human evaluation?
Q5. How to Safely Execute Operations with Side Effects
Suppose the AI calls tools that change external states, such as transferring money, sending emails, or creating tickets.
Design an execution flow that includes input validation, permission checks, user approval, retries, and prevention of duplicate execution.
Q6. How to Stop a Runaway AI
The AI has entered a loop of repeated searches or tool calls and won't finish processing.
How do you detect the loop, and under what conditions do you stop it?
How do you set budgets for time, execution count, tokens, cost, and tool usage?
Q7. Service Continuity During Model Failure
A high-performance model becomes unavailable due to latency, rate limits, or outages.
Under what conditions do you switch to a different model?
If the alternative model results in lower quality or functionality, how do you modify the user experience?
Q8. How to Trace a Single Failure
A user reports that "the response was slow and the content was incorrect."
How do you trace that specific processing instance?
Q9. Analysis of Increased Costs
AI usage fees increased from the previous month.
How do you identify which specific feature, customer, workflow, or processing stage caused the increase, rather than just which model?
What criteria do you use to judge if that cost is worth the business value?
Q10. What Was Learned from Production Failures?
Describe the most dangerous failure you've experienced in an AI feature you operated.
- What happened?
- How was it discovered?
- How did you stop the impact?
- How did you identify the cause?
- What did you change to prevent recurrence?
