AIToolworxThe TOPS Trap

Method

How we tested

What the hardware was, which configurations ran, how each test was structured and why one candidate was left out.

The setup

The hardware was a Raspberry Pi 5 with the Raspberry Pi AI HAT+ 2, whose Hailo-10H accelerator is listed at 40 TOPS (INT4). Some models ran on the accelerator and some on the CPU alone. Findings apply first to the exact configuration tested, because model revision, quantization, runtime, chat template, system instruction and packaging can each change behavior.

Method

How every test worked

Same structure for every model and both topics.

  1. 1

    Fixed evidence packet

    Identified by SHA-256 hash. The only sources the model may use.

  2. 2

    Preserved prompt

    Answer only from the packet. Cite entries as [S1], [S2].

  3. 3

    Model run

    Request, response, runtime log and model hash all kept.

  4. 4

    Check the answer

    Answers compared with the packet. The Ministral record was scored claim by claim.

Closed-book
Evidence-grounded
Without the study’s system message
A
B
With the study’s system message
C
D

Four conditions per model, on each of two matched topic packets, one China-related and one United States-related.

Key takeaway Every step leaves a file, so anyone can re-run it or re-score it.

Source: the method and results sections of the full paper and the reproducibility package; see the evidence page. The Ministral record was run at temperature 0 with three trials per condition.
  • Fixed evidence packets, identified by SHA-256 hash, held the only sources the model was allowed to use.
  • Prompts told the model to answer only from the packet and to cite packet entries as [S1], [S2], and so on.
  • Two matched topic packets, one China-related and one United States-related, used the same test structure.
  • Each model was tested in four conditions: with and without the study’s system message, each closed-book and evidence-grounded.
  • Requests, responses, runtime logs and model hashes were preserved for every run. The Ministral record was run at temperature 0 with three trials per condition.
  • Exploratory observations are reported separately from the controlled tests.

Seven configurations, six scored

ConfigurationWhere it ranWhat was observed
Qwen 2.5 Instruct 1.5BHailo acceleratorRepeated source blending and missing citations; response mode differed between the China and United States conditions.
Qwen 2.5 Coder 1.5BHailo acceleratorSaid the evidence was absent when it was present (China grounded); unsupported expansion in the United States condition.
Qwen 2 1.5BHailo acceleratorSource blending and unsupported additions; repeated factual fabrication and contradiction.
Llama 3.2 3BHailo acceleratorPartly grounded when it could be evaluated; runtime failures impaired key comparisons.
Gemma (gemma-4-e2b)CPUStrongest source discipline observed; repeatedly identified the limits of the evidence.
Ministral 3 3B Instruct 2512 (Q4_K_M)CPU (llama.cpp)Fluent answers with significant source blending; citation behavior differed by topic.
DeepSeek R1 Distill Qwen 1.5BHailo acceleratorFailed a basic three-sentence instruction check and was excluded before controlled testing. Not scored.

Summarized from Section 15 of the full paper.

Why DeepSeek was excluded

DeepSeek R1 Distill Qwen 1.5B ran on the Hailo accelerator and was a candidate for the study. It never reached the controlled tests. Before any evidence-grounding began, it was given a deliberately simple task: explain in three concise sentences why a local AI assistant is useful without Internet access. That was a minimal instruction-following check, not a test of politics or sources.

It failed on several counts at once. It did not stay within three sentences. It exposed a long stream of unrequested step-by-step reasoning text, including a visible </think> marker. It then gave a second answer, and it used far more output than the task needed.

A configuration that cannot meet that minimum cannot support a meaningful controlled comparison, so it was excluded before confirmatory testing. The exclusion is kept in the record instead of the model being quietly dropped. A separate exploratory prompt on a historical topic also produced a reasoning trace with factual confusion; that is not a controlled result.

What this does not show. Why it happened. The cause could lie in the underlying model, the distilled architecture, the accelerator packaging, the chat template or the runtime. DeepSeek receives no grounding, topic or reliability score, and nothing here ranks it against the six controlled configurations.

Plain-language glossary

TOPS
Trillion operations per second, a hardware throughput rating. The Raspberry Pi AI HAT+ 2’s Hailo-10H accelerator is listed at 40 TOPS (INT4).
Grounding
How closely an answer stays inside the evidence the model was given.
Provenance
Where a claim actually comes from, and whether the source it names really supports it.
False provenance
A citation that points to a source which does not support the claim. This is the label used in the claim-by-claim scoring.
Evidence packet
The fixed set of sources, identified by a SHA-256 hash, that a model is allowed to use in a test.
Closed-book / grounded
Closed-book: the model gets only the question. Grounded: the model also gets the evidence packet.
System message
An instruction given to the model ahead of the user’s question. The study ran each model with and without its own system message.
Chat template
The formatting rules that wrap messages before they reach the model. A template can add default instructions the user never typed.
Quantization
Storing a model’s numbers at lower precision so it fits and runs on compact hardware (for example Q4_K_M).
Tokens per second
How fast a model produces text.

The files behind every step are listed on the evidence page.