面試任務要求候選人打造可故障回退的 AI 工單服務

At 11:40 p.m., a candidate's repo lands in your inbox. The screen recording is smooth: a log line goes in, a tidy ticket comes out, and the agent never stumbles. Then you open the code and find one prompt, no tests, and a .env file with a live key. The demo is impressive. The evidence is missing.

AI demos are cheap now. A model can make almost any take-home look finished for ninety seconds. What you need to see is whether the candidate can make an AI feature survive a bad response, a missing key, and a reviewer who runs the tests on a train.

The current AI conversation is full of agents that can do everything. The interview should be about what the agent does when it cannot.

The brief

Build a small HTTP service that turns one log line into a support ticket. Return schema-valid JSON with service, severity, first_seen, evidence, confidence, and needs_human. If the model is unavailable or returns junk, the service must still return a valid ticket and mark it for human review. Tests must run with no network and no API key. Include a runbook with one command to start, one to test, and one to replay a fixture. You may use MonkeyCode's free model access for live extraction and its free server option for the demo; do not commit secrets. Record model calls, cache hits, and fallbacks in a ledger.

That last sentence is the interview. Anyone can call a model. The ledger proves the candidate knows when the model was called, when it was not, and what happened when it failed.

Why the free tier belongs in the brief

MonkeyCode is an open-source AI coding project with free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The offer matters because it lowers the cost of the exercise. It does not make the exercise easy.

The task is shaped so a single small HTTP service is all you need, and the free server option is one place to put it. The brief I received describes a 10-million-token free allowance. I would treat that as a snapshot, not a spec. Quotas move, and a take-home should test whether the candidate can design around a limit, not whether they memorized today's number.

The rubric

A rubric for this task should reward boring reliability. The model's prose quality is almost irrelevant.

The weights matter less than the order. Reliability before cleverness. A candidate who writes a strong prompt but no fallback has built a demo, not a service.

A sample solution

The sample below is a skeleton you can show after the exercise, not before. It is deliberately small: one endpoint, one cache, one rule fallback. It proves the shape without doing the candidate's thinking.

# app.py
import hashlib
import json
import os
import sqlite3
import urllib.request

from fastapi import FastAPI
from pydantic import BaseModel, Field

class Ticket(BaseModel):
    service: str
    severity: str = Field(pattern='^(low|medium|high|critical)$')
    first_seen: str
    evidence: str
    confidence: float = Field(ge=0, le=1)
    needs_human: bool

class Request(BaseModel):
    line: str

app = FastAPI()

def db_path():
    return os.getenv('CACHE_DB', 'cache.sqlite3')

def cache_get(key: str):
    with sqlite3.connect(db_path()) as con:
        con.execute('create table if not exists cache (key text primary key, value text)')
        row = con.execute('select value from cache where key=?', (key,)).fetchone()
    return json.loads(row[0]) if row else None

def cache_put(key: str, value: dict):
    with sqlite3.connect(db_path()) as con:
        con.execute('insert or replace into cache values (?, ?)', (key, json.dumps(value)))

def call_model(prompt: str) -> str:
    # Adapt the request body to the endpoint in the current docs.
    req = urllib.request.Request(
        os.environ['MODEL_URL'],
        data=json.dumps({'prompt': prompt}).encode(),
        headers={
            'Content-Type': 'application/json',
            'Authorization': 'Bearer ' + os.environ['MODEL_KEY'],
        },
    )
    with urllib.request.urlopen(req, timeout=20) as res:
        return json.loads(res.read())['text']

def rule_fallback(line: str) -> dict:
    low = line.lower()
    severity = 'critical' if 'panic' in low else 'high' if 'error' in low else 'low'
    service = line.split()[0] if line.split() else 'unknown'
    return {
        'service': service,
        'severity': severity,
        'first_seen': 'unknown',
        'evidence': line[:160],
        'confidence': 0.2,
        'needs_human': True,
    }

@app.post('/extract')
def extract(req: Request):
    key = hashlib.sha256(req.line.encode()).hexdigest()
    if (cached := cache_get(key)) is not None:
        return {**cached, 'source': 'cache'}

    prompt = (
        'Return JSON only. Extract service, severity, first_seen, evidence, '
        'confidence, and needs_human from this log line: ' + req.line
    )
    try:
        ticket = Ticket(**json.loads(call_model(prompt))).model_dump()
        source = 'model'
    except Exception:
        ticket = Ticket(**rule_fallback(req.line)).model_dump()
        source = 'fallback'

    cache_put(key, ticket)
    return {**ticket, 'source': source}

The important line is not call_model. It is except Exception. The fallback returns a ticket with low confidence and needs_human: true, so a bad model day becomes a queue, not an outage. The cache key is the raw line, so repeated demos do not spend twice.

# tests/test_app.py
import pytest
from fastapi.testclient import TestClient
from app import app

@pytest.fixture(autouse=True)
def sandbox(tmp_path, monkeypatch):
    monkeypatch.setenv('CACHE_DB', str(tmp_path / 'cache.sqlite3'))
    monkeypatch.delenv('MODEL_URL', raising=False)
    monkeypatch.delenv('MODEL_KEY', raising=False)

client = TestClient(app)

def test_panic_falls_back_and_caches():
    line = 'billing-api panic: ledger write failed'
    first = client.post('/extract', json={'line': line}).json()
    second = client.post('/extract', json={'line': line}).json()
    assert first['severity'] == 'critical'
    assert first['needs_human'] is True
    assert first['source'] == 'fallback'
    assert second['source'] == 'cache'

Run it in this order. The test should pass before any key exists. If it does not, the candidate has coupled the test to the model.

python -m venv .venv && . .venv/bin/activate
pip install fastapi uvicorn pydantic pytest httpx
pytest -q
uvicorn app:app --reload
curl -s localhost:8000/extract \
  -H 'content-type: application/json' \
  -d '{"line":"billing-api panic: ledger write failed"}' | jq

Failure modes you will see

The prompt-only service. A candidate strings together a prompt and trusts the output. The first malformed JSON from the model becomes a 500. Full marks require validation and a fallback that still returns a ticket.

The hidden key. The tests pass only on the candidate's laptop because they call a live endpoint. A reviewer cannot reproduce the result. The fix is a fixture and a fake client, not a longer README.

The retry storm. On timeout, the service retries five times in a loop. That turns a model hiccup into a spend spike and a slow demo. Better: one bounded retry, then fallback, then a ledger entry.

Confidence theater. The model returns 0.97 and the candidate displays it as truth. Confidence should be calibrated or omitted. needs_human is more useful than a fake probability.

The overfit fixture. The sample has one obvious panic line, and the rule fallback hard-codes it. Change one token in the input and watch the service degrade honestly. A good take-home should make that change easy.

Limits and who should skip this

This task is for backend or product engineers who must ship a bounded AI feature. It is not for a research role where model quality is the product, and it is not for a team that cannot legally send logs to a third party. Use synthetic fixtures and the exercise still works.

Do not treat the free allowance as a permanent architecture constraint. It is a budget for the interview, not a production plan. If the current server option has limits, the candidate should document them instead of designing around a number they cannot verify.

Skip this task if you need to evaluate latency at scale or multi-tenant isolation. Those are different interviews. This one tests whether the candidate can make a small AI feature honest under failure.

If you want to try the same constraints, start with MonkeyCode's current README, run the offline test first, and only then point the client at a live endpoint. A take-home that survives that order is worth reading.