Benchmarks · Adil Eco V1.1 · Engineering report
Adil Eco V1.1 engineering benchmark.
Internally conducted comparative engineering benchmark. Not independent third-party certification.
A progress snapshot of the Adil Eco V1.1 release candidate, measured by our engineering team and published on . It compares the candidate with Qwen3-0.6B Base and with three other compact open-weight models, using standard evaluation tools and a custom structured-output suite.
Research release candidate Not publicly downloadable
Read this first.
- Internally conducted comparative engineering benchmark. Not independent third-party certification. It is not a leaderboard entry or an award.
- Adil Eco V1.1 is not released: it is not publicly downloadable, production-approved or offered as a service.
- No overall ranking is calculated, and no claim is made that Adil Eco is better than Liquid or Granite models overall: on most quality measures here, LFM2-700M scores higher.
- Every CPU figure comes from a shared RunPod cloud server (Intel Xeon Gold 6342, 4 threads). None is a laptop measurement, and the ThinkPad T480 hardware gate remains unverified.
- Chat-template results for Adil Eco and Qwen3-0.6B Base are marked unreliable: the chat protocol returned empty outputs for many of their responses, and the cause has not been established.
- Training-data overlap with the public benchmark sets has not been independently audited.
Summary
Where the candidate stands.
Promising
- Zero-shot answering: ARC-Easy 62.8 against 54.5 for Qwen3-0.6B Base, and MMLU zero-shot 41.1 against 32.4.
- High strict-JSON validity under raw completion (99.2), level with Qwen3-0.6B Base and Granite 4.0-H-350M.
- Context extraction held at 100 up to about 8K tokens in a small exploratory suite.
Needs improvement
- Mathematical reasoning regressed: GSM8K 22.0 against 31.0 for Qwen3-0.6B Base.
- Exact tool arguments under the chat template (46.7) and 22 of 120 chat responses with invalid argument keys.
- Long-context speed and memory on the RunPod server: about 150 seconds to the first token at the experimental 16K context.
- Extraction fell to 50 at about 15K tokens, the experimental 16K level.
Still unverified
- Performance on the low-end laptop acceptance target; the ThinkPad T480 gate is unverified.
- Independent audit of training-data overlap with the benchmark sets.
- End-to-end tool execution through the SDK.
- Readiness for a stable public release.
Against Qwen3-0.6B Base, five-shot MMLU (45.1 against 45.0) and HumanEval (22.0 against 18.9, 36 against 31 of 164 problems) are within noise. The zero-shot gains may reflect answer-format adaptation rather than new knowledge; this has not been tested.
Models and conditions
What was tested, and how.
| Model | Developer | Licence | File bytes | Notes |
|---|---|---|---|---|
| Adil Eco V1.1 | THYNKR SYSTEMS LTD | Not released | 484,219,776 | Release candidate A3b, 4-bit Q4_K_M file; frozen for this benchmark. |
| Qwen3-0.6B Base | Qwen team (Alibaba Cloud) | Apache 2.0 | 484,220,384 | Matched comparison model: the Qwen3-0.6B Base checkpoint, quantised to Q4_K_M by the same pipeline as the Adil Eco file. A base model, not Qwen's instruction-tuned release. |
| Liquid LFM2-700M | Liquid AI | LFM Open License v1.0 | 468,624,320 | Vendor-made Q4_K_M file. |
| IBM Granite 4.0-H-350M | IBM | Apache 2.0 | 222,662,560 | Vendor-made Q4_K_M file. |
| IBM Granite 4.0-350M | IBM | Apache 2.0 | 236,985,760 | Vendor-made Q4_K_M file. Open scorer-parity discrepancy: all of its scores carry a caveat. |
Quality tests
- Run on a RunPod NVIDIA A40 GPU server; quality scores do not depend on the processor.
- EleutherAI LM Evaluation Harness v0.4.13 task definitions, with a separately implemented log-probability scorer for multiple-choice tasks, checked against reference implementations.
- Greedy decoding, one run per model, same dataset revisions, shot counts and seeds for every model.
- Raw completion (no chat template) is the primary protocol; chat-template results use each model's own template and are shown separately.
CPU and memory
- RunPod Intel Xeon Gold 6342 cloud server, 4 threads, CPU-only build of llama.cpp commit 620d2225c; no GPU backend loaded.
- Memory is the process resident set size (RSS) after the model is loaded with the stated context size.
- Adil Eco, Qwen3-0.6B Base and LFM2-700M were measured on an idle host. Granite 4.0-H-350M was measured while other jobs ran, with shorter prompts. Granite 4.0-350M was not measured.
- Exploratory measurements on one cloud server: not consumer hardware and not system requirements.
How to read the charts
- Measured
- Completed measurement under the stated conditions, one greedy run.
- Parity caveat
- Open scorer-parity discrepancy for this model; not a validated result.
- Unreliable
- Chat protocol returned empty outputs for many responses from Adil Eco and Qwen3-0.6B Base; cause not established.
- Contended run
- Measured while other benchmark jobs used the host; not comparable with idle runs.
- Exploratory
- 12 trials per cell; exploratory, not conclusive.
- Not measured
- Not completed before the benchmark was closed; no figure is shown.
1. Knowledge and reasoning
Knowledge and reasoning.
| Model | Result |
|---|---|
| ARC-Easy, accuracy | |
| Adil Eco V1.1 | 62.8 |
| Qwen3-0.6B Base | 54.5 |
| LFM2-700M | 74.7 |
| Granite 4.0-H-350M | 59.0 |
| Granite 4.0-350M | 63.3Parity caveat |
| ARC-Challenge, normalised accuracy | |
| Adil Eco V1.1 | 33.3 |
| Qwen3-0.6B Base | 30.7 |
| LFM2-700M | 46.5 |
| Granite 4.0-H-350M | 34.3 |
| Granite 4.0-350M | 32.5Parity caveat |
| HellaSwag, normalised accuracy | |
| Adil Eco V1.1 | 46.6 |
| Qwen3-0.6B Base | 44.8 |
| LFM2-700M | 57.1 |
| Granite 4.0-H-350M | 48.6 |
| Granite 4.0-350M | 48.2Parity caveat |
| MMLU, zero-shot | |
| Adil Eco V1.1 | 41.1 |
| Qwen3-0.6B Base | 32.4 |
| LFM2-700M | 48.7 |
| Granite 4.0-H-350M | 30.8 |
| Granite 4.0-350M | 30.0Parity caveat |
| MMLU, five-shot | |
| Adil Eco V1.1 | 45.1 |
| Qwen3-0.6B Base | 45.0 |
| LFM2-700M | 48.6 |
| Granite 4.0-H-350M | 26.7 |
| Granite 4.0-350M | 29.8Parity caveat |
Adil Eco scores above Qwen3-0.6B Base on ARC-Easy and zero-shot MMLU, is level on five-shot MMLU and slightly above on HellaSwag. LFM2-700M scores highest on every measure in this group. The Granite 4.0-H-350M five-shot MMLU result (26.7) is near chance and below its own zero-shot result; the cause has not been determined.
2. Mathematics
Mathematics: a regression.
| Model | Result |
|---|---|
| GSM8K, five-shot, strict match (raw completion) | |
| Adil Eco V1.1 | 22.0 |
| Qwen3-0.6B Base | 31.0 |
| LFM2-700M | 43.9 |
| Granite 4.0-H-350M | 18.0 |
| Granite 4.0-350M | 20.2Parity caveat |
Adil Eco regressed against Qwen3-0.6B Base by 9.0 percentage points. In the saved samples, 75 in every 100 of its answers had a correctly formatted final number that was wrong, so the loss is in the reasoning, not the formatting. Chat-template maths results are not shown: they are format artefacts for models that answer briefly, and unreliable for Adil Eco and Qwen3-0.6B Base.
3. Coding
Coding.
| Model | Result |
|---|---|
| HumanEval, pass@1 (164 problems) | |
| Adil Eco V1.1 | 22.0 |
| Qwen3-0.6B Base | 18.9 |
| LFM2-700M | 22.6 |
| Granite 4.0-H-350M | 20.7 |
| Granite 4.0-350M | 20.7Parity caveat |
The differences in this group are small: one problem is worth about 0.6 percentage points. Two runs of LFM2-700M on the same problems solved 35 and 37 (the second, reported run used the network-blocking sandbox filter); batched decoding is not bit-for-bit deterministic.
4. Instruction following
Instruction following.
| Model | Result |
|---|---|
| IFEval, prompt-level strict (raw completion, primary) | |
| Adil Eco V1.1 | 22.6 |
| Qwen3-0.6B Base | 13.5 |
| LFM2-700M | 45.3 |
| Granite 4.0-H-350M | 24.0 |
| Granite 4.0-350M | 26.2Parity caveat |
| IFEval, prompt-level strict (chat template) | |
| Adil Eco V1.1 | 47.5Unreliable |
| Qwen3-0.6B Base | 53.0Unreliable |
| LFM2-700M | 62.1 |
| Granite 4.0-H-350M | 46.0 |
| Granite 4.0-350M | 42.5Parity caveat |
Raw completion is the primary result. Adil Eco is above Qwen3-0.6B Base there (22.6 against 13.5) and below LFM2-700M (45.3). Under the chat template, Adil Eco and Qwen3-0.6B Base are marked unreliable because of empty outputs.
5. Structured JSON output
Structured output: strong raw formatting, weaker chat reliability.
| Model | Result |
|---|---|
| Strict JSON, raw completion | |
| Adil Eco V1.1 | 99.2 |
| Qwen3-0.6B Base | 100.0 |
| LFM2-700M | 61.7 |
| Granite 4.0-H-350M | 99.2 |
| Granite 4.0-350M | 80.8Parity caveat |
| Exact tool arguments, raw completion | |
| Adil Eco V1.1 | 82.5 |
| Qwen3-0.6B Base | 87.5 |
| LFM2-700M | 52.5 |
| Granite 4.0-H-350M | 88.3 |
| Granite 4.0-350M | 60.8Parity caveat |
| Strict JSON, chat template | |
| Adil Eco V1.1 | 70.8Unreliable |
| Qwen3-0.6B Base | 60.0Unreliable |
| LFM2-700M | 16.7 |
| Granite 4.0-H-350M | 77.5 |
| Granite 4.0-350M | 51.7Parity caveat |
| Exact tool arguments, chat template | |
| Adil Eco V1.1 | 46.7Unreliable |
| Qwen3-0.6B Base | 64.2Unreliable |
| LFM2-700M | 6.7 |
| Granite 4.0-H-350M | 30.8 |
| Granite 4.0-350M | 13.3Parity caveat |
This is our own 120-case synthetic suite, not a recognised external leaderboard. It measures formatting and argument selection, not completed tool execution, and results depend strongly on the prompting protocol.
Under the chat template, 22 of the 120 Adil Eco responses contained argument keys that are not valid for the tool the model named, against 1 for Qwen3-0.6B Base. Most were structural, such as wrapping the payload in an extra key or using another tool's field names, rather than invented values. This is a material reliability limitation, and valid JSON alone must not be read as safe or accurate tool execution.
LFM2-700M answers in its own native tool-call format, which strict JSON scoring penalises; a native-format score was not computed. Tool calling through the Adil Edge SDK is not yet available and is tested separately.
6. CPU memory consumption
Memory grows with context.
| Model | Result |
|---|---|
| 4K context | |
| Adil Eco V1.1 | 1,184 |
| Qwen3-0.6B Base | 1,184 |
| LFM2-700M | 833 |
| Granite 4.0-H-350M | 435Contended run |
| Granite 4.0-350M | Not measured (benchmark closed) |
| 8K context | |
| Adil Eco V1.1 | 1,631 |
| Qwen3-0.6B Base | 1,632 |
| LFM2-700M | 881 |
| Granite 4.0-H-350M | 451Contended run |
| Granite 4.0-350M | Not measured (benchmark closed) |
| 16K context (experimental for Adil Eco) | |
| Adil Eco V1.1 | 2,528 |
| Qwen3-0.6B Base | 2,528 |
| LFM2-700M | 977 |
| Granite 4.0-H-350M | 485Contended run |
| Granite 4.0-350M | Not measured (benchmark closed) |
| Model | Result |
|---|---|
| 4K context | |
| Adil Eco V1.1 | 448 |
| Qwen3-0.6B Base | 448 |
| LFM2-700M | 48 |
| Granite 4.0-H-350M | 16Contended run |
| Granite 4.0-350M | Not measured (benchmark closed) |
| 8K context | |
| Adil Eco V1.1 | 896 |
| Qwen3-0.6B Base | 896 |
| LFM2-700M | 96 |
| Granite 4.0-H-350M | 32Contended run |
| Granite 4.0-350M | Not measured (benchmark closed) |
| 16K context (experimental for Adil Eco) | |
| Adil Eco V1.1 | 1,792 |
| Qwen3-0.6B Base | 1,792 |
| LFM2-700M | 192 |
| Granite 4.0-H-350M | 64Contended run |
| Granite 4.0-350M | Not measured (benchmark closed) |
On the RunPod server, the Adil Eco process used about 1,184 MiB at a 4K context, 1,631 MiB at 8K and 2,528 MiB at the experimental 16K context, against a weights file of 462 MiB. On that RunPod server, its context cache took about 112 KiB per token (1,792 MiB at the experimental 16K context), the same as Qwen3-0.6B Base, because both use full attention; LFM2-700M needs about 12 KiB per token and Granite 4.0-H-350M about 4 KiB.
These RunPod RSS figures are a different metric on a different host from the Apple M1 process-footprint figures on the benchmarks index, so the two sets should not be compared directly.
7. Context length and latency
Context length and latency.
| Model | Result |
|---|---|
| 4K context, 3,968-token prompt | |
| Adil Eco V1.1 | 13.7 |
| Qwen3-0.6B Base | 15.5 |
| LFM2-700M | 11.0 |
| 8K context, 8,064-token prompt | |
| Adil Eco V1.1 | 42.0 |
| Qwen3-0.6B Base | 47.3 |
| LFM2-700M | 25.6 |
| 16K context (experimental for Adil Eco), 16,256-token prompt | |
| Adil Eco V1.1 | 150.1 |
| Qwen3-0.6B Base | 152.9 |
| LFM2-700M | 64.1 |
| Context and prompt | Model | Time to first token (seconds) | Decoding there (tokens per second) |
|---|---|---|---|
| 4K context, 3,968-token prompt | Adil Eco V1.1 | 13.7 | 14.1 |
| Qwen3-0.6B Base | 15.5 | 13.9 | |
| LFM2-700M | 11.0 | 43.3 | |
| 8K context, 8,064-token prompt | Adil Eco V1.1 | 42.0 | 7.4 |
| Qwen3-0.6B Base | 47.3 | 7.6 | |
| LFM2-700M | 25.6 | 34.6 | |
| 16K context (experimental for Adil Eco), 16,256-token prompt | Adil Eco V1.1 | 150.1 | 3.4 |
| Qwen3-0.6B Base | 152.9 | 3.4 | |
| LFM2-700M | 64.1 | 21.1 |
| Model | Result |
|---|---|
| About 4K tokens | |
| Adil Eco V1.1 | 100.0Exploratory |
| Qwen3-0.6B Base | 41.7Exploratory |
| LFM2-700M | 100.0Exploratory |
| Granite 4.0-H-350M | 91.7Exploratory |
| Granite 4.0-350M | 100.0Parity caveat |
| About 8K tokens | |
| Adil Eco V1.1 | 100.0Exploratory |
| Qwen3-0.6B Base | 16.7Exploratory |
| LFM2-700M | 100.0Exploratory |
| Granite 4.0-H-350M | 91.7Exploratory |
| Granite 4.0-350M | 100.0Parity caveat |
| About 15K tokens (the experimental 16K level) | |
| Adil Eco V1.1 | 50.0Exploratory |
| Qwen3-0.6B Base | 8.3Exploratory |
| LFM2-700M | 100.0Exploratory |
| Granite 4.0-H-350M | 100.0Exploratory |
| Granite 4.0-350M | 100.0Parity caveat |
With near-full-context prompts on the RunPod server, Adil Eco took about 150 seconds to the first token at the experimental 16K context and then decoded at about 3.4 tokens per second; with short prompts it decoded at about 65 tokens per second on the same RunPod server. LFM2-700M took about 64 seconds at the same length on that server.
Granite 4.0-H-350M latency is not shown: it was measured with shorter prompts while other jobs ran, so it is not comparable. In the context suite, needle and multi-turn recall were 100 for every model at every length; the experimental 16K level was about 15K effective tokens.
Limitations
Known limitations of this benchmark.
- Internally conducted by our engineering team; not independent third-party certification, a formal leaderboard result or an award.
- One greedy run per model; small suites for context (12 trials per cell) and structured output (120 cases); no repeated-seed variance.
- Structured-output and tool-argument results are protocol-sensitive: raw completion and chat-template results differ widely, and the suite does not test tool execution.
- Chat-template tool-argument reliability is a known weakness of Adil Eco V1.1, and its chat-template results, with those of Qwen3-0.6B Base, are unreliable until the empty-output cause is found.
- Mathematical reasoning regressed against Qwen3-0.6B Base.
- Every CPU figure was measured on a shared RunPod cloud server and does not satisfy the T480 hardware gate, which remains unverified; Granite CPU measurements are incomplete or contended.
- Unresolved scorer parity: Granite 4.0-350M shows a discrepancy between the runtime and its reference implementation, so none of its results are validated.
- Incomplete or non-comparable competitor measurements: no native-format tool-call score for LFM2-700M, no idle CPU run for either Granite model.
- Dataset contamination: overlap between the Adil Eco training data and ARC, HellaSwag, MMLU, GSM8K or HumanEval has not been independently audited.
- Some evaluation choices were made after early results were visible; each is recorded with both figures where applicable.
- Competitor files are the vendors' own 4-bit files, while Adil Eco and Qwen3-0.6B Base share one quantisation pipeline.
Reproducibility
Versions and evidence.
- Runtime
- llama.cpp commit 620d2225c (CPU-only build for CPU measurements).
- Harness
- EleutherAI LM Evaluation Harness v0.4.13; transformers 5.21 for scorer parity checks.
- Datasets
- Public Hugging Face snapshots: ai2_arc 210d026, Rowan/hellaswag 218ec52, cais/mmlu c30699e, openai/gsm8k 740312a, google/IFEval 966cd89, openai_humaneval 7dce605.
- Adil Eco file
- SHA-256
ccce35dc13460804a53e15a6d3e7fc6be97714d7065717a9864703db9ab24b60, 484,219,776 bytes. The file is not published. - Third-party files
- LFM2-700M 684e8406…, Granite 4.0-H-350M 0a8d6a73…, Granite 4.0-350M 771c588a… (SHA-256 prefixes of the vendors' Q4_K_M files).
- Evidence
- Model hashes, scripts, configurations and raw results are retained in an encrypted engineering archive. A public reproduction package is not offered; it would need a separate release and licence review.
Attribution
Third-party models and datasets.
LFM2-700M is a model by Liquid AI, used for evaluation under the LFM Open License v1.0. Granite 4.0-H-350M and Granite 4.0-350M are models by IBM, evaluated under the Apache 2.0 licence. Model names identify the models tested; they are the trademarks of their owners, and no endorsement by Liquid AI or IBM is implied.
Benchmarks used: ARC (Allen Institute for AI, CC BY-SA 4.0), HellaSwag (MIT), MMLU (MIT), GSM8K (OpenAI, MIT), IFEval (Google, Apache 2.0) and HumanEval (OpenAI, MIT), through the EleutherAI LM Evaluation Harness (MIT) and llama.cpp (MIT). Scores reflect our configuration and may differ from figures published by the model developers.
Release status
Not released.
Adil Eco V1.1 remains a research release candidate: it is not publicly downloadable or production-approved, and its sealed acceptance gate failed. The SDK, hardware qualification, legal and licensing checks, independent backups and owner approval remain separate release conditions, not yet met.
We will publish revised measurements after later configurations have been tested and accepted.