Fine-Tuning a Tiny Language Model for Production Tool Calling

Can a model small enough for one CPU learn one narrow enterprise job well enough to ship?

Most enterprise "AI features" are not essays. They are small, repetitive translations. A person types:

Open a high priority Jira ticket in OPS: checkout returns 502 errors in production.

and the system needs exactly this, nothing more:

{
  "name": "create_jira_issue",
  "arguments": {
    "project": "OPS",
    "summary": "checkout returns 502 errors",
    "priority": "high",
    "environment": "production"
  }
}

A frontier model does this trivially. It also costs a network round trip, an API bill, a rate limit and a copy of your data leaving the building, per request. For a task this narrow, that is a lot of machinery.

The model we picked for the job is Needle 2 from Cactus Compute: 45 million parameters, quantised mostly to 2 bits (4-bit embeddings), shipped as a single 14 MB binary. It was built for tool calling on phones and embedded devices, which makes it an odd candidate for a server-side job that people usually hand to a model a thousand times its size.

So the question we set out to answer:

Can a tiny language model, fine-tuned on a purpose-built dataset, do enterprise tool routing and argument extraction reliably enough for production, and run on a single CPU with about a gigabyte of memory?

We did not benchmark a frontier model or a rules baseline on this set. The comparison in this article is Needle zero-shot against Needle fine-tuned; the large-LLM number is assumed, not measured.

This article stays at the level of what we fed the model, what knobs we set, what the loss curve did, and what the held-out set and a one-CPU container measured. Dataset, adapter, tuned model, evaluation harness, Dockerfile and benchmark script all exist alongside this write-up; every number here was produced by them.

The use case

The model is an internal operations router. It sits between a chat box (or a Slack command, or a ticket form) and five tools our ops team already has:

The model has three jobs, and the third is the one people forget: which tool, which arguments (taken from the request), and whether to call anything at all. The output is a structured call or an empty list.

So this:

Rerun the Salesforce connector for Acme in production.

becomes:

{
  "name": "rerun_connector",
  "arguments": {
    "connector": "Salesforce",
    "customer": "Acme",
    "environment": "production"
  }
}

while this:

The checkout-api recovered in production.

becomes:

[]

The empty list is a first-class answer. A router that fires an action when it should have stayed quiet is worse than one that occasionally asks you to repeat yourself. In a system that can send invoices and rerun production syncs, the false positive is the expensive mistake.

The data

Everything the model knows about this job comes from a synthetic dataset we built for it. All companies, invoice numbers and email addresses are made up (emails use the reserved .example domain).

So 12.7% of training and 16.7% of evaluation is "do nothing".

Each record carries the query, all five tool schemas (abbreviated here), the expected answer, and a one-line reasoning string. This is a training record:

{
  "query": "Create a high Jira issue in project CRM for customer login is failing in production.",
  "tools": [
    {
      "name": "create_jira_issue",
      "description": "Create a Jira issue for an operational or software problem.",
      "parameters": { "project", "summary", "priority", "environment", all required }
    },
    { "name": "send_invoice", ... },
    { "name": "rerun_connector", ... },
    { "name": "run_dbt_job", ... },
    { "name": "check_service_status", ... }
  ],
  "answers": [
    {
      "name": "create_jira_issue",
      "arguments": {
        "project": "CRM",
        "summary": "customer login is failing",
        "priority": "high",
        "environment": "production"
      }
    }
  ],
  "reasoning": "Use create_jira_issue; arguments are explicitly present: project=CRM, priority=high, environment=production, summary=customer login is failing."
}

Three design decisions did most of the work.

Every argument value is grounded in the query. The model is not learning that "HubSpot" goes with rerun_connector. It is learning that the thing after "Retry" and before "connector" is the connector, and the possessive before it is the customer. Nothing in an answer appears that was not in the request. One limit of that: the entity vocabulary (companies, services, environments) is shared between training and evaluation, and 93% of evaluation argument values also appear somewhere in training. Extraction of names the model has never seen is not tested here.

The same operation is phrased many ways. Per tool, the 550 training examples contain 508 to 550 distinct queries for Jira, invoices and connectors, but only about 240 for dbt jobs and status checks, the two tools that turn out to have the most argument errors. "Add an issue to OPS…", "Open a high issue in DATA because…", "Raise a low Jira ticket under CORE:…" all map to the same call. The evaluation set goes further: its phrasings are held out from the training templates entirely, so "Re-execute the Stripe connector for Globex in staging" and "Give the SAP sync another attempt for Tyrell Corporation in development" are shapes the model never saw during training. It uses three new templates per tool, fifteen in all, so each per-tool score below is a verdict on three phrasings.

"No" comes in three flavours. The 400 negatives are not just off-topic chatter. They split into:

Off-topic            "Who founded Microsoft?"  "Write a poem about databases."
Already resolved     "The payment-service looks healthy now."  "I already sent the invoice."
Missing arguments    "Send the invoice."  "Rerun Salesforce."  "Create a Jira issue in PAY."

The third category is the interesting one. "Rerun Salesforce." is clearly a rerun_connector request, but it names no customer and no environment. The correct output is [], not a call with invented values. That is the behaviour that keeps a router from guessing its way into a production incident.

One caveat about the negatives: there are only 38 distinct negative phrasings, repeated, and 36 of them also appear verbatim in the evaluation set. So the no-tool score below measures only recall of sentences the model trained on, and the 59% figure it produces is an upper bound. The positive side of the evaluation is properly held out; the negative side needs a harder, unseen set before we would trust the number in production.

The fine-tune

We used LoRA (Low-Rank Adaptation) rather than full fine-tuning. The base weights stay frozen; the adapter learns a small low-rank correction on top of each targeted weight matrix:

With rank 16 and alpha 32 the scale is 2, and the two learned matrices for a 512×512 weight are 16×512 and 512×16 instead of another 512×512. Rank 16 is the bottleneck: it caps how complicated the update can be, and cuts trainable parameters by more than 20× (about 2M of 45M).

The run, as Needle reported it:

One line in that table deserves a sentence. Needle trains with the same quantised numerics it will export with, so the adapter is learning against the model it will actually run as, not a higher-precision cousin that gets rounded afterwards. That is what lets the CPU-side numbers later in this article be about the same model that was trained.

Training loss

Loss every 17 steps, with the validation loss Needle computed at each epoch boundary:

Three things to read off the curve.

The first epoch does almost all of the learning. Loss starts at 0.72 and is at 0.05 by step 170. Everything after that is refinement.

The steps are noisy on the way down (0.72, 0.86, 0.48, 0.81, 0.47). Minibatches differ in difficulty, so a single step going up means nothing. The trend is what counts.

Validation tracks training closely, 0.0012 against 0.0011 at the end, with no sign of the gap widening. On a dataset this regular, that is expected, and it is also why loss is not the metric we care about.

Results: accuracy, base against fine-tuned

A loss of 0.0011 says the model reproduces the training format; the real experiment starts after training.

We merged the adapter into the base and exported a tuned .cact (13.7 MB, in line with the 14 MB base engine). Then both models, base Needle 2 and the tuned one, answered the same 480 held-out requests on a MacBook through the same harness, with the library's default decoding, one request at a time. Single training run, single evaluation pass, no seeds varied. And because the 400 positive requests are 15 held-out templates with 20 to 31 fillings each, the score is closer to "12 of 15 phrasings work" than to 400 independent trials; one template flipping moves exact match by 4 to 7 points.

The metrics:

Exact match is the production number. Tool accuracy alone lets environment: "staging" pass when the user said production.

The no-tool row is n=80, on sentences seen in training, and the container run (below) put the base at 52.5%; treat the +10 as within noise.

Exact match nearly triples. Per tool, the picture is uneven, and that unevenness is the useful part:

Where each model fails

Invented values under a required key count as "wrong argument value", not as the last row.

The base model is not wrong so much as reluctant. 231 of its 480 answers are false negatives: it returns [] for a request that plainly asks for a tool. It has never seen these tools, and its built-in confidence gate declines them at Needle's default threshold (the confidence it reports on those requests has a median of 0.17). We did not sweep that threshold or disable the gate, so the base row is a zero-shot floor, not a tuned competitor; a fairer comparison would sweep the base threshold and add a threshold to the tuned model, which currently has no gate at all. When the base does call, arguments are the problem (70 wrong values). Wrong tool is rare (4). A general tool-caller, in other words, understands the schemas but does not trust itself with them.

The fine-tuned model has 107 misses, in three clusters.

All 31 wrong-tool errors are one sentence shape. Every held-out request phrased "Report the health of service in environment" was routed to create_jira_issue. The training set has no "report" verb for status checks, and "report" is a Jira word. One template, one systematic miss: 31 of the 45 status-check misses.

The argument errors cluster in five held-out templates. 14 from "Verify the operational state of service…", where the model returns service: "operation"; 11 from "Can you open CORE issue for duplicate orders are being created, priority critical, on uat?", which came back with project: "duplicate" and summary: "CORE issue"; 7 from "Execute transformation job with dbt…", where it invents job names; and 11 spread over two more. The Jira case shows the mechanism. The model learned the field order from training phrasings where the project code follows a preposition ("in OPS", "under CORE"); a bare "CORE issue for…" shifts the boundaries and it slices the sentence in the wrong place. Argument accuracy is 89.5% overall and above 95% on three of the five tools; the misses are span-boundary errors on unseen phrasings.

The negatives are where it is weakest. 33 false positives. Off-topic chit-chat ("Tell me a joke", "Write a poem about databases") is refused. Off-topic questions that mention a product name are not: "How does OAuth work?" and "What is PostgreSQL?" became check_service_status calls with service: "OAuth" and "PostgreSQL", and 8 of the 33 are this kind. Three more are our mistake, not the model's: "Check payment-service in production." is labelled no-tool in the dataset but names a service and an environment, so the model's status call is the right answer and the label will be fixed. The remaining 22 are under-specified operational requests and status reports:

Send invoice INV-10001.                 → send_invoice, customer "INV-10002", email "INV-10001"
Rerun HubSpot for Acme.                 → rerun_connector, environment "HubSpot"
The payment-service looks healthy now.  → check_service_status, environment "payment-service"

These are exactly the cases the 400 training negatives were meant to teach. With 38 distinct phrasings, 36 of them also in the evaluation set, the model is failing on sentences it has effectively seen; on genuinely new refusals the number would be lower still. The invented values are mostly a nearby token from the sentence ("HubSpot" as an environment, an invoice number as an email) or the field name itself (environment: "environment"): a byte-level grammar forcing a required field to be filled from a sentence that does not contain it. A few are genuine hallucinations: job_name: "expecute" and "convert" for "Execute transformation…", project: "Curiosity", and priority: "medium" for "Create a Jira issue in PAY.", which is exactly the plausible-looking guess a router must not make. The fix is more, and more varied, "missing argument" negatives, and it is the first thing we would change in the dataset.

One more thing the fine-tune costs you. Needle's base model ships with a calibrated confidence head; LoRA does not train that head, so the tuned model reports no confidence at all. Every "no" now has to be learned from data, which is one more reason the negatives matter.

Loss said 0.0012. Exact match says 78%.

Validation loss was measured on a 10% slice of the training distribution, phrasings the model had seen variants of. Exact match was measured on held-out phrasings. The model learned the training templates almost perfectly and generalises to new ones about four times out of five. Which number you report decides whether you ship.

Results: cost, on one CPU in a small box

Training needs a GPU. Serving should not. The pipeline is:

We ran the exported model inside a Docker container on the same MacBook, limited to 1 vCPU and 1 GB of RAM via --cpus and --memory, replayed the same 480 requests, and recorded latency, throughput, tokens per second and memory, for both models through the same harness. This is a Docker Desktop VM on an Apple Silicon core, not a cloud vCPU; the same run on the bare Mac was about 4× faster, so treat the absolute latencies as one data point, not a spec.

The fine-tuned model costs the same per token as the base: prefill and decode speeds are within a few percent, and the model's peak RAM is the same 50-odd megabytes (the Python interpreter and runtime are not counted in that figure). Its median request is 22% slower, 484 ms against 397 ms, because it produces more tokens: the base answers [] for over half the requests and stops, while the tuned model writes out a full call. Same speed, more useful output.

The container run reproduced the tuned scores to within one example (372 vs 373 correct, 7 rows changing bucket) but the base drifted more: 129 vs 134 correct, with 24 rows changing bucket. Same weights, same inputs; the base's confidence gate sits near its threshold on many rows and CPU numerics decide them. The accuracy tables above report the Mac run.

Method: N independent copies of the harness in one container, each loading its own model and answering the same 120 requests; wall time includes the N model loads. It measures process-per-client, not a shared server, and we stopped at four because the trend was clear. (The single-client figure differs from the 2.1 above by run-to-run noise; it is a separate, shorter run.)

One CPU is one CPU, and a second process does not just halve throughput, it does worse: 240 requests took 215 s where perfect time-slicing would take 118 s, and four processes were four times worse than that again. We did not profile why; --cpus=1 is a CFS quota rather than a pinned core, and four interpreters in 1 GB may be paging. Whatever the cause, the serving shape on this box is one process and a queue in front of it. Scaling is horizontal, and cheap: another 1 CPU / 1 GB box, not a bigger one.

What the numbers say

A large jump in task accuracy at no change in cost per token (the median request is 22% slower because it now writes a full call instead of []), on a model that fits in the memory budget of a browser tab. That argues for splitting the work:

The large model stays where it earns its cost. The narrow, high-volume, well-specified operations move to a model that behaves less like a chatbot and more like a learned software component: local, fast, cheap, predictable, and small enough to ship inside the service that needs it.

Conclusion

This started as a fine-tuning exercise and ended as an argument about architecture.

A 45M-parameter model, about an hour of LoRA on a Colab T4 and 3,150 synthetic examples took exact-match accuracy on unseen phrasings from 28% to 78%, at no change in cost per token: the same 14 MB file, the same 50 MB of model RAM, the same tokens per second, inside a container capped at one CPU and a gigabyte.

78% exact match is not a production number for a router that can send invoices. But most of the misses have a name and a plausible fix in the data: one verb the positives never used, one sentence shape that moves the project code, a set of "missing argument" negatives that was too small and too repetitive to teach refusal, and three labels that were simply wrong. Whether more data closes those gaps, or whether a larger model would, is the next experiment, and the whole loop (regenerate data, an hour on a T4, re-export, re-run this harness) costs less than a day. The question is not "can a tiny model compete with a frontier model?" It is "what is the smallest model that reliably solves this problem?", and for this problem the answer is one you can iterate on before lunch.

The tuned model's no-tool accuracy of 59%, measured on refusals it had already seen in training, is the one figure that would stop us shipping this router as-is, because in a system that can send invoices, a confident wrong call costs more than a missed one. Small models learn to act quickly; teaching them when not to is where the dataset effort belongs.

Further reading

• Cactus Compute, Needle 2: the model, the Python package and its fine-tuning workflow
• Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (2021)
• TurboQuant, on shrinking a model's memory without retraining, the other half of the small-footprint story

About J33.AI

At J33.AI we size models to the job. Most of that work is this article: building the dataset that defines the task, measuring the thing that matters rather than the loss, and finding the smallest model that clears the bar. Our AI foundations practice runs from model selection through to deployment, across every industry we serve.