A result that repeats your words may not answer your question
You look for a way to avoid a repeated charge and find an image draft titled ‘avoid repeated charges’. The words match, but there is no payment procedure. Before choosing keyword or meaning-based search, define which document would actually resolve the task and inspect what each route returned.
Waiting list; no immediate access or free plan guaranteed. The destination is in Spanish.
Situations to review
Q1: ‘evitar cobros repetidos’ — avoid repeated charges. The need is a procedure for avoiding a duplicate payment when retrying.
We labelled D1 relevant before the test. The word-overlap baseline returns D2, whose title matches but whose text explicitly describes an image draft, not a payment procedure. The assistant selects D1, which says to check whether a charge has already occurred before retrying.
Three matching words in D2 do not answer the request. This example reveals a false match; it does not show that all lexical search fails or that Think² prevents duplicate payments.
Q2: ‘no entro después de cambiar la contraseña’ — I cannot get in after changing the password. The need concerns access and sessions after a credential change.
D3 and D4 were labelled relevant. The baseline returns D1 and D2 because tied scores are resolved by document ID. D4 also scores 2 but falls outside the first two positions. The assistant selects D3 and D4.
Keep the tie-breaking rules and full scores: the final list alone would hide why D4 was missed. We did not test a commercial index, synonym expansion or access permissions.
Q3: ‘preparar informe con monedas distintas y datos faltantes’ — prepare a report with different currencies and missing data.
Only D5 was labelled relevant. The baseline returns D5 and D6; the assistant returns only D5. Both retrieve the single relevant document, but the first set also includes a notice without amounts or useful instructions.
Retrieving every relevant item does not mean every retrieved item is useful. Under this test’s definition, precision@2 is 0.5 for both routes: we always divide by two, even when only one item is returned. Stating the denominator prevents a misleading interpretation.
Waiting list; no immediate access or free plan guaranteed. The destination is in Spanish.
How to evaluate the work
- Define the need and documents that could resolve it before inspecting outputs. Distinguish a relevant document from one that merely mentions the words. Here, relevance is our own judgement, not an independent assessment or universal ground truth. The six synthetic texts deliberately include misleading matches; they are not a representative business sample. All test inputs are Spanish. This English article is an editorial translation, not an English-language retrieval test.
Literal Spanish corpus; English retrieval was not tested. D1 | Antes de reintentar un cargo incierto, consulta en el destino si la operación ya ocurrió. D2 | El título ‘evitar cobros repetidos’ identifica un borrador de imagen, no un procedimiento de pagos. D3 | Acceso deshabilitado tras renovar credenciales: revisa si se invalidó la sesión, sin compartir secretos.
D4 | Un cambio de contraseña requiere revisar qué sesiones siguen válidas y pedir entrada por el canal autorizado. D5 | Para preparar un informe financiero, separa USD y COP y conserva las celdas vacías como datos faltantes. D6 | El aviso ‘informe de gastos listo’ no contiene importes ni instrucciones para comprobar monedas. Literal queries and prior labels: Q1 | evitar cobros repetidos | D1 Q2 | no entro después de cambiar la contraseña | D3,D4 Q3 | preparar informe con monedas distintas y datos faltantes | D5
- Lexical method actually executed in Python: Unicode NFKC normalisation, lowercasing and sets of alphanumeric tokens. The score is the number of distinct tokens shared by query and document. No IDF, stop-word removal, frequencies or stemming. Sort by descending score and ascending document ID for ties; exclude zero scores and return up to two items. This is a deliberately simple baseline, not a definition or performance estimate for all lexical search.
- Observed selection by need: an assistant received the six texts, three queries and each query’s explicit purpose. It did not receive the relevance labels and produced one response without tools or retries. The baseline used only the query text, without that additional purpose: conditions were not equivalent. The protocol used the phrase ‘same information’ while also stating this difference; that phrase does not make the inputs equivalent. We used no embeddings and ran no semantic search engine. We do not identify a model version from which to generalise behaviour.
- Observed results; P@2 and R@2 under our own rubric. query | lexical IDs | P@2 | R@2 | assistant IDs | P@2 | R@2 Q1 | D2 | 0.0 | 0.0 | D1 | 0.5 | 1.0 Q2 | D1,D2 | 0.0 | 0.0 | D3,D4 | 1.0 | 1.0 Q3 | D5,D6 | 0.5 | 1.0 | D5 | 0.5 | 1.0
- For a fairer operational comparison, prepare representative queries and give the routes you want to compare the same query text and authorised corpus. Fix relevance rules, tie-breaking and the number of results to inspect beforehand. Keep unanswered cases, wrong documents and access restrictions. Independent assessment and a sample separate from the one used for tuning would help reduce bias; neither was performed here.
- Elastic’s official documentation describes semantic search as retrieval by meaning and discusses approaches such as semantic_text with inference configuration. This helps frame an evaluation; it does not establish quality on your corpus or an integration with Think². If you need an exact identifier, inspect whether the result preserves it. If users describe the same issue in different words, include those paraphrases in the evaluation. We choose no universal winner and measure no latency, cost or scale.
- Evaluation record: literal query; need; corpus and version; prior labels; each route’s configuration and inputs; ranked results and reasons; relevant items retrieved, missed or falsely matched; denominators; limitations; next review. Think² lets you organise collaborators, context and review using compatible models and authorised tools, with an optional computer per collaborator. We did not test its search, a document index, automatic file access or execution of this workflow inside the product.
Download the evidence
- Synthetic corpus, protocol and saved-output metrics — Spanish test inputs · JSON · SHA256: ad71eef0ace5fcac9fd1661bdf01cd01376843f6fc2e139ec5da8c221044fa6d
Frequently asked questions
Does this test prove semantic search is better?
No. It contrasts a basic lexical baseline with an assistant selection that received additional purpose information. There was no semantic engine, equal-input design, representative sample or product comparison. The results illustrate how to inspect relevance and errors.
Why does one correct result have precision@2 of 0.5?
We fixed two positions as the denominator, even if one remains empty. For Q1 and Q3 the assistant retrieves one relevant item in up to two positions: 1/2. Recall@2 is 1/1. If you prefer precision over returned items, define it as another metric rather than changing the definition after seeing outputs.
Can I repeat the assistant generation with the download?
Not with the included check: it recomputes the baseline and metrics using the corpus and saved response. The prompt is included for inspection, but the script does not call a model or promise to reproduce a new generation. It does not configure commercial search either.
Sources and scope
- Elastic — Semantic search — Documentation of meaning-based retrieval and configuration; not a test of Think² or measured quality on this corpus.
Keep exploring
Waiting list; no immediate access or free plan guaranteed. The destination is in Spanish.