LLM human evaluation with verified people

Once live answering opens, a small human evaluation LLM check could reveal confusing wording, tone problems or a preference between two reply styles before a release decision. Ask which reply is more useful, then read why. That would give a team a concrete issue to inspect, especially when a model score misses the reader’s experience. The example card shows a fictional five-person comparison; it is not a measured product result.

This is a quick judgement format, not a training dataset at scale. It is also a poor fit for harmful outputs, specialist advice or a release gate that needs a representative sample. Do not send sensitive material to members without the agreed controls.

Only quick takes are in scope. A pick-one question can compare two or three options of up to 80 characters each; a longer reply can be uploaded as a screenshot. Pools are 5 to 50 UK adults. The sandbox uses synthetic members and carries no evidence of what real people prefer. Training rights for answers are not settled, so do not treat responses as licensed model-training material.

Write the prompt without naming the model or signalling which answer you favour. For example: “Which reply is more useful to someone asking how to return this item?” Add two short options, a pool size and an idempotency key. POST /v1/questions returns an ID and cursor; GET /v1/questions/{id} returns the split, reasons and status. For an image, upload PNG, JPEG or WebP first and pass its asset ID. The API reference gives current field names and limits.

An answer receipt is a signed ES256 JWS. Its claims bind the question and asset hashes, the answer, status and checks passed, without naming a member. Verify it against the public JWKS for live answers. Synthetic sandbox receipts use a separate key and say so. A valid signature proves the record came from this service; it does not make a five-person sample representative or the judgement correct. See how receipts work.

Sandbox requests and synthetic answers are free. During early access, asking is free; paid questions have no start date. The quick-take asker price is planned from 30p an answer plus 50p a question (launch example). Twenty quick takes would cost about £6.50. A sandbox key is available by invitation. There is no separate RLHF human feedback product price, and answer-use rights for model training are unsettled. Check the pricing page for the current launch example.