Python Foundations for AI — Interview Q&A
Q: Why would you use asyncio instead of just making sequential API calls?
A: Because independent API calls are I/O-bound, not CPU-bound — while waiting for one model response, Python isn't doing any useful work, so asyncio.gather can fire multiple calls concurrently and let them all wait on the network at once, instead of one after another. For 10 independent prompts, that's roughly the latency of the single slowest call instead of 10x one call's latency. The distinction to be precise about: this helps because the bottleneck is waiting on a network response, not because Python is doing heavy computation in parallel — asyncio doesn't give you true CPU parallelism the way multiprocessing does.
Q: How do you handle a conversation that's grown too long for the model's context window?
A: Naively truncating (just dropping the oldest turns) loses information silently. A better approach: summarize older turns into a compact representation before dropping the originals, keeping the system prompt and most recent turns intact — trading some fidelity for staying within the context limit deliberately, rather than losing context unpredictably. The honest answer includes acknowledging summarization has its own cost (an extra model call) and its own failure mode (a bad summary loses something that mattered).
Q: What's the difference between exact-match evaluation and the kind of evaluation you'd actually use for LLM output?
A: Exact-match assumes deterministic, identical output for a given input — true for traditional software, false for LLM output, which varies in phrasing even when "correct." Real evaluation for LLM systems uses structural or semantic checks instead: does the response contain a required entity, does it satisfy a schema, does a separate scoring function (sometimes another model call, an "LLM-as-judge" pattern) rate it as correct. Getting this distinction right — and being able to explain why exact-match is the wrong tool here — is a genuine signal of practical AI engineering experience versus theoretical knowledge.
Q: You're building an agent with tool access — what stops it from looping forever?
A: A hard iteration cap in the agent loop itself, checked every cycle, not an assumption that the model will naturally stop. This matters for two real reasons, not just theoretical safety: an unbounded loop on a stuck agent burns real API cost with every iteration, and a genuinely malformed tool response can put the model into a retry pattern that never resolves on its own without an external limit forcing it to stop.
Q: How would you estimate and control the cost of an AI feature before shipping it?
A: Start by logging token usage (input and output separately, since they're usually priced differently) per request from the very first version, even in development — retrofitting usage visibility after a surprise bill is much harder than having the data already flowing. For control: caching repeated context where the provider supports it, capping max_tokens deliberately rather than leaving it unbounded, and — for anything with meaningful volume — running a cost estimate against expected usage patterns before shipping, not after.
Q: What's a real bug you'd expect to see in AI-calling Python code that wouldn't show up in a typical code review?
A: A missing Optional/null-check on an API response field that's usually present but occasionally absent (a citations field, an error field only present on failure responses) — this passes review because the happy-path code looks fine, and only fails at runtime on the specific response shape that wasn't tested. This is a good answer specifically because it shows you understand that AI API responses are less uniformly shaped than typical internal APIs, and that type hints plus explicit handling of the "field might be missing" case is a real, non-obvious defense against it.
Q: Streaming a response versus waiting for the full response — when would you NOT want to stream?
A: When you need the complete response before you can act on any of it — structured output you're going to json.loads() and use programmatically (not display to a user) generally needs to be complete and valid before it's useful; a partial JSON string mid-stream isn't parseable. Streaming's value is specifically for human-facing text output where partial content is still useful to see as it arrives — for a backend process consuming structured data, waiting for the complete response is usually simpler and correct.