Building J33-Jev: Is Your Own AI Decision Model Worth It?

What we tested, at a glance

Size is how many learned values the model has. The number in brackets is the work each tool saved per 100 decisions, explained below. We also built four other versions of J33-Jev; they are in the results table.

Think of it as training your own new hire

Buying an off-the-shelf model is like hiring someone experienced from another company. Building your own is like taking a bright graduate and training them on your own past cases. Either way, the skill that matters is knowing when they might be wrong, and asking. The question for a business is whether the training pays off.

What we did

We used the same public case set as last time, typed-decisions. It covers four kinds of work: customer service, invoices, security alerts and checking on AI assistants. Each case, such as one support ticket, comes with five questions. The test has 400 cases, which makes 2,000 decisions.

The cases are made up for testing. The "right answers" were set by another AI model, not by people, so "right" means "agreed with that model". That model agrees with itself 73.5% of the time. A score well above that partly means a tool has learned the labeller's habits.

We kept the same score, agreed before looking at any results.

• The AI decides alone and gets it right: +1, a piece of work a person no longer does.
• The AI decides alone and gets it wrong: −5, because someone has to find and fix the mistake.
• The AI hands the case to a person: 0, the same as today.

One mistake cancels out five right answers, so the AI should only decide alone when it is at least 83.3% sure.

We built J33-Jev in two parts. The base is EmbeddingGemma 2, a free model from Google DeepMind that turns text into numbers a computer can compare. On top, we added a small scoring layer that reads each case and question and scores every possible answer. Some versions also put an extra layer between the base and the scoring layer. We followed the design of Laya, whose makers publish their code. The name J33-Jev refers to the Jev-style format: typed questions in, a confidence for each answer out. J33-Jev is not affiliated with or endorsed by TypeSafe, the maker of Jev.

We fine-tuned the whole model on the test's practice cases. Every part of it learned from them, EmbeddingGemma 2 included, not just the scoring layer. The test comes with 1,200 practice cases, kept separate from the 400 test cases. For the four text-only versions, we trained on 1,080 of them and kept 120 aside. We used those 120 to fit the model's confidence setting (this is called calibration), so its "how sure I am" matches how often it is actually right. We never tuned anything on the test cases.

We tried five versions, to see what made a difference:

• First version, with an extra layer.
• First version, refitted: the same model and answers, with only its confidence setting fitted again, this time to the single best answer for each case. The first fit had left it underselling itself.
• No extra layer: the simplest version, with the scoring layer reading the base model directly.
• Bigger extra layer: a larger version of that extra layer.
• Also trained on images: trained on the business cases plus questions about photos from the public COCO photo collection.

Every version had one training run. Small gaps between versions, of a few percentage points right or a few pieces of work per 100, could change if we ran them again.

What we found

One unit is one decision a person no longer has to make. So +22.5 means that for every 100 decisions, the tool took 22.5 off your team's desk after paying for its mistakes. "Right overall" counts every decision, including the ones the tool would hand to a person. "Right when it decided alone" is the number that matters for trust.

Not every tool here had the same head start. Every J33-Jev version and both Laya rows studied this test's practice cases. Jev was used exactly as delivered, with no training on them. Julia-1's makers don't publish its training data, and their files mention the same four kinds of work. The test's own notes say trained and untrained scores should not be compared directly. The fair comparison for J33-Jev is Laya.

We also adjusted the confidence setting only for our own model and for Laya (explained below). Jev's and Julia-1's scores were used exactly as delivered.

Confidence that matched accuracy saved more work

Laya was right more often than J33-Jev: 76.9% against 75.2%. When it decided alone, it was right 99.4% of the time and made only 2 mistakes in 2,000 decisions. J33-Jev made 58.

J33-Jev still saved more work, because it decided alone more than twice as often. After we fitted it on held-out practice cases, its confidence was the closest match to its accuracy of any tool here. On average, how sure it said it was came within 1.9 percentage points of how often it was right. Jev came within 3.3 points, with no training or tuning on this test at all. Re-tuned Laya was 13.7 points out.

Laya's most confident 39.7% of decisions were 95% right taken together, yet it decided alone on only 17.7%. The difference, about one decision in five, went to the team.

Part of this is down to how we ran each model. We fitted J33-Jev's confidence setting ourselves, on 120 practice cases it had not seen. For Laya we used the setting stored in its checkpoint. Its documentation says to fit that setting on your own data, and we have not done that for Laya.

Which you prefer depends on your business. If a mistake costs you far more than five right answers save, Laya's careful profile may suit you better. On your real cases, with answers your own staff agree on, the numbers will be different.

Refitting confidence alone more than doubled the work saved

Our first version and its refitted copy gave exactly the same answers. The only difference was the confidence setting. Refitting it raised the share of decisions handled alone from 10.2% to 34.1%, and work saved from +3.3 to +8.3.

A model that undersells itself sends easy work back to your team, as we saw with Laya last time.

Simpler beat bigger, in our one run of each

In our single run of each version, the simplest, with no extra layer, did best. The bigger extra layer was less accurate (72.2%) and saved little more than half as much work (+12.9). Add the bigger layer only if your own tests show it helps.

One version was also trained on photo questions. In our own training-time check it was right 96.1% of the time on those, but on the business cases it scored 65.1% and saved +5.9. It was trained and tuned differently from the others, so we draw no conclusion from it.

Where this pays off

The test covered four kinds of everyday work:

• Customer support: sending each ticket to the right team and spotting the urgent ones.
• Accounts payable: checking whether an invoice is a duplicate or needs a closer look.
• Security: sorting alerts into "needs someone tonight" and "can wait".
• AI assistants: checking whether an automated assistant did its job or needs a person.

J33-Jev was strongest on invoices, right 80.2% of the time. Against Laya, the fair comparison, it trailed on every kind of work: by 0.4 points on invoices and by up to 3.2 points on customer service. A model trained on your own work can still be uneven, so check each kind of work separately.

Take a team handling 10,000 of these decisions a month. This is an example volume, scaled from the rates we measured:

• J33-Jev would handle about 3,985 alone, with around 290 mistakes to fix. Net, about 2,245 decisions off the team's desk.
• Laya, re-tuned would handle about 1,765 alone, with around 10 mistakes. Net, about 1,705.
• Jev would handle about 3,755 alone, with around 410 mistakes. Net, about 1,295, with no training on this test's practice cases.
• Julia-1 would handle about 9,030 alone, but around 2,230 would be wrong. That creates more work than it saves.

Everything not handled alone goes to the team as it does today.

On this test, building our own took about 950 more decisions a month off the desk than Jev. But ours had studied the test's practice cases and Jev had not.

What does it need to run?

Training

We trained our best version on one NVIDIA RTX 4090, a 24 GB graphics card sold for gaming PCs, not a data-centre server. Training took about 10 minutes: four passes over the practice cases at roughly two and a half minutes each. Our other versions were trained on an NVIDIA A40, a 48 GB data-centre card.

Inference (using the trained model)

We ran our best version on the same RTX 4090. It answered each case, five decisions, in about 33 thousandths of a second. That is a different machine from the one in our last test, so the speeds are not directly comparable with the other tools.

It also runs without a GPU. We ran the same model on the processor of an Apple M3 Max laptop, with the graphics chip unused, over all 400 test cases. The answers were identical: 75.2% right and +22.5 work saved per 100 decisions. It took just under a second per case, about 28 times slower than the RTX 4090. For the 10,000 decisions a month in our example, about 2,000 cases, that is roughly half an hour of computer time. A processor is enough for work that can wait a second or two. Anything that must answer instantly, such as a live chat, needs a GPU.

The model file is about 1.09 GB, and you can download it from Hugging Face. The model card shows how to load it in Python and run it on a processor. Google designed EmbeddingGemma 2, the base we built on, to run on phones and laptops.

When is it worth building your own AI decision model?

Here is how we read these results for a business deciding whether to build.

1. Test an off-the-shelf tool on your own past cases first. Jev saved +13.0 per 100 decisions with no training on this test's practice cases. If your data may leave your company and that is enough, you may not need to build anything.
2. Build when you have the past decisions to train on. Training on your own kind of cases gave the biggest single gain in accuracy we have seen across both articles. Laya's general version was right 36.1% of the time; the version trained on practice cases, 76.9%.
3. Work out what a mistake costs you first. It sets how sure the model must be, and it decides whether J33-Jev's profile or Laya's suits you.
4. Fit the confidence setting (calibration) on cases kept aside. Refitting it alone took our first version from +3.3 to +8.3, with no change to its answers.
5. Start small. In our single run of each version, the simplest beat the bigger one. Add complexity only when your own tests show it helps.
6. Keep people in the loop and keep checking. Even the best version here handed six decisions in ten to a person. Plan how those reach your team, and re-test on recent cases every few months.

Frequently asked questions

Working with J33

Our automation and AI agents work starts by working out what a mistake costs your business and testing off-the-shelf tools on your own past cases. If none is good enough, we train a small model on your past decisions and fit its confidence on cases it has never seen. Then we set it up so your people only see the decisions that need them.