Extractive or generated summaries: preserving words is not enough
Copying a sentence can preserve its words while losing a warning from the next line. Rewriting can cover more topics yet turn what a note says into a direct assertion. We tested both routes on one synthetic source, not to choose a universal winner but to show what to check before using a summary.
Waiting list; no immediate access or free plan guaranteed. The destination is in Spanish.
Situations to review
You need a literal extract and know which lines to preserve.
The baseline selected L1 and L2 without rewriting. It is a deterministic selection of the first two lines, not an optimized extractor. It preserved the subtotal and proposed date but left out subsequent conditions.
Literal wording does not establish contextual coverage. An extract can be faithful to selected lines while omitting warnings necessary for a decision.
You need a synthesis of several points with references.
An assistant received the source and an instruction to write at most 60 words with citations. Its actual output used 53 words excluding citation tags and covered the subtotal/undetermined total, proposed/unconfirmed dispatch and absence of recorded authorization.
For cancellation, it lost attribution to a note. References L5–L6 are relevant but do not restore the omitted qualifier. We did not correct the original output to present a perfect success.
You intend to use a summary to decide whether to dispatch an order.
Separately check subtotal, proposal versus confirmation, cancellation and authorization. The sample records no authorization; it does not establish an order's status outside the document.
Neither text is a dispatch instruction. Obtain missing evidence and applicable authorization before any action; writing quality and a citation do not grant permission.
Waiting list; no immediate access or free plan guaranteed. The destination is in Spanish.
How to evaluate the work
- Before testing, fix the purpose, source/versions, instruction and rubric. Here C1 requires subtotal and undetermined final total; C2 a proposal and unconfirmed address/date; C3 attribution of the request to a note and no recorded acceptance; C4 no recorded authorization in this source. The downloadable protocol was saved before requesting the generated output.
Exact source and outputs in English, the test language. These are preserved verbatim for checking and do not describe a real order. [L1] The draft order lists two items at USD 90 each, a known subtotal of USD 180. [L2] Operations proposed dispatch on 20 November 2026. [L3] The customer has not confirmed the delivery address or dispatch date. [L4] Taxes and shipping costs are missing, so the final total is undetermined. [L5] A note says cancellation was requested, but no accepted cancellation is recorded. [L6] No dispatch authorization is recorded in this source.
EXTRACTIVE The draft order lists two items at USD 90 each, a known subtotal of USD 180. [L1] Operations proposed dispatch on 20 November 2026. [L2] GENERATED The draft lists two items at USD 90 each, with a USD 180 subtotal [L1]. Operations proposed dispatch on 20 November 2026, but the customer has not confirmed the address or date [L2–L3]. Taxes and shipping are missing, so the final total is undetermined [L4].
Cancellation was requested, but acceptance and dispatch authorization are not recorded [L5–L6].
- Our manual review of four criteria: extractive baseline 0 complete,2 partial and2 omitted; generated output 3 complete and1 partial. We detected no direct contradiction under this rubric, but generated criterion C3 strengthens the source's claim: ‘a note says’ is not a directly established request. Citations are relevant, not proof of external truth. We do not combine the results into a score or present them as system accuracy. One source, a 23-word baseline against a 53-word output, one generation and no retries: budgets are unequal and the baseline was not optimized. Word counts exclude citation tags and standalone punctuation.
- Review record: claim; source line/version; quotation or paraphrase; context/qualifier; complete/partial/omitted/contradicted criterion; missing evidence; decision/owner. Think² can organize the assignment with collaborators, groups/context, compatible models and authorized tools; a computer for each collaborator is optional. This test does not establish a Think² summarization feature or connectors. If your existing workflow already preserves sources and adequate review, do not add another layer by default. A corrected version would be a new reviewed proposal, not the original measured output.
Download the evidence
- Test outputs and metrics · JSON · SHA256: d7e81b09e9128bc2a80479eab971962618e7b82ed7b697fe6adaf7d34ad4ccb5
- Original protocol and source · JSON · SHA256: d790a78f77e3402a1d5bce84b4b7f0978b2f92e92b27a3ae27c1fba3db84e1b4
Frequently asked questions
Is an extractive summary error-free because it copies?
It can preserve selected lines literally while omitting decisive context. Copying L1–L2 here omits taxes/shipping, missing confirmation and authorization. This does not establish that every extractive method fails; it identifies what this baseline did not cover.
Is a summary with citations necessarily faithful?
No. Check whether each reference supports the claim and its qualifiers, and whether necessary points are missing. Citation L5 accompanies a sentence that lost ‘a note says’; a relevant reference is not full faithfulness or truth outside the source.
Does this example establish which model or format is always appropriate?
No. One synthetic English source, one generated output and one unoptimized literal selection cannot rank engines or generalize accuracy. We did not measure time, cost, other languages or customer results, or execute a Think² product workflow.
Sources and scope
- On Faithfulness and Factuality in Abstractive Summarization — Abstract and bibliography consulted; not the full methodology, and no results extrapolated to this test.
Keep exploring
Waiting list; no immediate access or free plan guaranteed. The destination is in Spanish.