Laya scored 47–48% in Korean, Qwen-based 0.8B decision models 71–81% — six Ollaya models tested on matched Korean and English sentences

Summary

Key terms

We write percentage-point differences as “pp”: 56.9% to 47.0% is a 9.9 pp drop.

How we measured

Data — Korean and English sentences with the same meaning

We used Amazon's MASSIVE 1.1 (CC BY 4.0), a dataset of short requests to a voice assistant in 51 languages. Across languages, the sentence with the same ID has the same meaning and the same label. We checked that the Korean and English labels agree for all 2,974 pairs in the held-out test split. The Korean sentences are localized rather than translated word for word: English “can you book an uber for me” becomes “카카오택시 예약 좀 해줄래” (“could you book a KakaoTaxi”, KakaoTaxi being a Korean ride-hailing service).

The task is to pick one of 18 domains (MASSIVE's “scenarios”) for each request: alarm, audio (volume), calendar, cooking, datetime, email, general (small talk), iot (smart home), lists, music, news, play, qa (general questions), recommendation, social, takeaway (food delivery), transport and weather. We drew 30 sentences per domain with a fixed random seed to build 540 pairs, so guessing at random scores about 5.6% (1 in 18). The slower decoder-family models ran only the first 10 sentences per domain — 180 pairs — and when comparing models we re-scored Laya on the same 180 pairs.

Three conditions

Each request carried one choice question. For the Korean-input, Korean-label condition, the request body looked like this (two of the 18 options shown). The question reads “Which smart-assistant domain does this user request belong to?”, the alarm description “set, check or cancel an alarm”, and the iot description “control smart-home devices: lights, plugs, coffee machine, robot vacuum”.

{"model": "laya:multilingual",
 "state": "오전 다섯 시 에 깨워줘",
 "questions": {"domain": {"type": "choice",
   "instructions": "이 사용자 요청은 스마트 비서의 어느 영역에 해당합니까?",
   "criteria": {"alarm": "알람 설정, 확인, 취소",
                "iot": "스마트홈 기기 제어: 조명, 플러그, 커피 머신, 로봇 청소기", …}}}}

We wrote the 18 domain descriptions ourselves, based on the MASSIVE intent names within each domain (for alarm, for example: set, query and remove). How the descriptions are worded changes the results, which we come back to in sections 7 and 11.

Models and setup

We ran the macOS binary of Ollaya 0.7.2 (released 2026-09-26) from a temporary folder after verifying its checksum, without installing it, and pulled model weights from the Hugging Face commits Ollaya pins for each developer. The hardware was an Apple M1 Max with 32 GB of memory. Requests were sent one at a time, without batching (grouping several requests into one call). Of these models only Laya runs on the Mac's GPU; the decoder family runs on the CPU in 32-bit floating point (fp32). Ollaya's status endpoint (/api/ps) reported metal for both Laya models and cpu for the decoder family.

Same sentences, different language — Laya on 540 pairs

540 pairs, 18 classes, random baseline 5.6%. The 95% confidence interval is the range accuracy could move if the sentences were drawn again; we computed it with the Wilson method for proportions. Through Ollaya's laya router, all 540 English sentences went to laya:en (first row); of the Korean sentences, 536 went to laya:multilingual and 4 to laya:en. Korean accuracy through the router was 47.0% (English labels), the same as the third row.

• Changing only the input language reduced accuracy by about 10 percentage points on the same model. laya:multilingual got 112 sentences right in English and wrong in Korean, and 59 the other way round. The McNemar p-value is about 0.00006, so the difference is hard to attribute to chance.
• The router widens the gap in practice. English goes to the more accurate laya:en (63.1%) and Korean to laya:multilingual (47.0–48.1%), so English and Korean users of the same service get accuracy 15–16 pp apart.
• Korean mixed with English words can be routed to the English model. The 4 Korean sentences the router sent to laya:en all contained English words, such as “beatles 노래 재생해줘” (“play a beatles song”), “pizza hut 배달 가능하나요” (“does pizza hut deliver?”) and “locality 뜻” (“meaning of locality”). The router reported that the text was in Latin script and that it could not identify the language, so it fell back to the default English model.
• Korean fed to the English model is almost always wrong. 12.4%. Yet 5.2% of its answers came with confidence ≥ 0.9, and 61% of those were wrong. The Laya developer also notes that the English checkpoint is “confidently wrong” on non-Latin scripts such as Korean hangul. That is why you should be careful when bypassing the router and naming a model directly.
• Korean label descriptions barely changed accuracy (47.0% → 48.1%, p=0.66) but changed how much the confidence values can be trusted. Section 6 covers this.

Some of MASSIVE's Korean sentences have a space before a particle (the short grammatical suffixes Korean attaches directly to a noun), as in “오전 다섯 시 에 깨워줘” (“wake me at 5 a.m.”, with a stray space before 에, “at”). This affects 81 of the 540 sentences. Re-running with those spaces removed gave the same accuracy, 47.0% and 48.1%.

Six models compared — 180 pairs

With only 180 pairs (10 sentences per domain), each value could move by roughly ±6–7 pp if the sentences were drawn again (95% confidence interval). Significance is a McNemar test pairing each sentence's English and Korean outcome. The two Laya rows differ slightly from section 4 because they count only these 180 of the 540 pairs.

• The three Qwen3.5-based ~0.8B models scored 70.6–80.6% in Korean, 21–31 pp above Laya (45.6–49.4%) on the same sentences and conditions. decider and kev, whose model cards list English only, handled Korean well. That is plausibly because the Qwen3.5 base is a multilingual LLM, but this measurement cannot establish the cause.
• The language gap was also smaller. For kev and decision the English–Korean difference was not statistically significant. decider:0.8b dropped 7 pp with English labels (p=0.015), and the gap shrank to 2.8 pp with Korean labels.
• The effect of the label language went in different directions. decider:0.8b and kev gained more than 4 pp with Korean labels, while decision lost 2.7 pp. None of these differences is significant, so you cannot generalize that Korean labels raise accuracy.
• decider (2B) scored highest: 88.9% in English and 83.9% with Korean input and Korean labels. The 5 pp difference between languages has a p-value of 0.06, not significant at the usual 0.05 level. Each request took several seconds on this Mac (section 9), so we did not run the Korean + English labels condition.

Filtering by confidence

In practice decision models are usually deployed like this: answers above a confidence threshold are handled automatically, the rest go to a person or a larger LLM. So what matters is less the overall accuracy than how accurate the answers above the threshold are and what share of all answers clears it.

• With Laya, the language of the label descriptions changed how much the confidence values could be trusted. Given Korean input with English descriptions, one in four answers at “0.9 or above” was wrong; with Korean descriptions, fewer than one in ten. The share clearing the threshold fell slightly, from 28.7% to 25.2%. In counts, 117 of 155 were right with English descriptions and 125 of 136 with Korean descriptions. The two groups contain different sentences, so this is an exploratory rather than a paired comparison; an unpaired z-test of the two proportions gives p below 0.001.
• A threshold of 0.9 means something different for each model. Only 8–12% of kev's answers reached 0.9, but all of them were right; more than half of decider:0.8b's answers passed 0.9 and still over 95% were right. At 0.9, kev would send over 90% of requests to a person, while decider:0.8b would handle more than half automatically. Ollaya's documentation also says not to reuse a threshold set for one model on another, but to set it per model.
• Measured by how much can be automated, the gap is wider. With Korean labels, answers that were both at confidence ≥ 0.9 and correct made up about 61% of all requests for decider:0.8b and about 23% for Laya (our multiplication of the two table values).

Where the wrong answers pile up

Laya pushed uncertain sentences into one label. Of the 540 sentences, 30 are iot (smart home), yet laya:multilingual answered iot 177 times in English, 192 times for Korean with English labels and 158 times for Korean with Korean labels — about 30% of all predictions in one label. Counting the same 180 sentences, Laya answered iot for 55–73 (31–41%) in Korean, while the three decoder-family models did so for 12–38 (7–21%).

Taken from the laya:multilingual results for English input + English labels and Korean input + English labels.

Part of this pile-up may come from our iot description (“control lights, plugs, coffee machine, robot vacuum”) being broad; adjusting speaker volume can be read as device control. But the decoder-family models, given the same descriptions, mostly got these right. The usable signal for practice is this: if one label is predicted several times more often than it occurs in the labelled data, it is time to narrow that label's description or change models.

By domain, with Korean input and Korean labels, Laya was weakest on general (0%), calendar (10%), recommendation (10%), play (20%) and social (20%) (180 pairs, 10 sentences per domain). On four of those domains, excluding play, the three decoder-family models got 50–80% right. Play was weak for every model (20–40%), and its wrong answers went mostly to iot and music.

Against the developers' published numbers

A critical English-language write-up of Laya appeared first. The OrcaRouter blog, citing the model card's typed-decisions figures, concluded that Laya out of the box is closer to random than to the majority-class baseline. On our task, by contrast, Laya cleared the majority-class baseline (the accuracy of always picking the most common label — 5.6% in this balanced sample) by a wide margin. Results depend heavily on the task.

Speed on a Mac

After all accuracy runs had finished and nothing else was running, we sent each model 30 Korean requests one at a time (English requests for laya:en). “Processing time” is the in-model time Ollaya reports in each response (tokenization, forward pass, calibration); “first request” includes loading the model into memory.

• On a Mac only Laya uses the GPU. Ollaya's home page says “Laya and NLI on MLX” for macOS, and /api/ps reported metal only for Laya. The decoder family runs on the CPU in fp32 through ONNX Runtime, so the ~0.8B decoder models were 45–79 times slower than laya:multilingual. In Ollaya's published RTX 4090 figures the gap is 16–23 times (Laya 8.1–9.6 ms vs decider 155–190 ms).
• Running several models at once on one Mac made things slower. With three decoder models running concurrently, requests took 6–9 seconds each and the 2B model up to 65 seconds, for lower throughput than running them one after another. We ended up re-running the accuracy measurements sequentially.
• For real-time classification at many requests per second, the decoder family on a Mac CPU will struggle. For a nightly batch job classifying a few thousand tickets, 1–2 seconds is perfectly usable. The first request takes 6–36 seconds to load the model, so warm it up once when the server starts.

What to check before adopting one

1. Check that non-English input reaches the multilingual model. Ollaya's /api/decide response includes the model that actually answered (model) and the routing reason (routing). When Korean text is sent to the English model, accuracy drops to 12.4%, and some answers are wrong at high confidence. If your text often mixes in English brand names, naming laya:multilingual directly is safer than the router.
2. Write the label descriptions in the input's language. With Laya, accuracy stayed the same but wrong answers at confidence ≥ 0.9 fell from 38 to 11. For the decoder family the effect varied by model, so measure both.
3. Label 200–500 of your own examples and measure first. Drawing the same number per class, as here, makes any pile-up into one label obvious. Numbers on model cards do not transfer to a different task.
4. Count the predicted labels. If one label is predicted several times more often than it occurs, its description is too broad or the model is dumping sentences it cannot place into it.
5. Set a new confidence threshold for each model. At the same 0.9, 8–12% of kev's answers passed, versus 49–63% of decider:0.8b's. Decide the accuracy you need first, then find the lowest threshold that delivers it on your own data.
6. Set a latency budget first. If every request must be answered within 100 ms, on a Mac only Laya qualifies, and you have to accept that roughly half of Korean requests will be misclassified. Ollaya's published RTX 4090 figures put the decoder family at 155–190 ms per request.
7. Check the training data on the model card. If the public dataset you evaluate on was used in training, the score is inflated. Some cards, like decider's, contradict themselves.
8. Pin versions. Ollaya reached 0.7.2 three days after its repository was created, and models get new versions every few days. Ollaya pins weights to a specific commit in the developer's repository, so record that commit in production.

What remains uncertain

• One task only. We measured a single problem: sorting short voice-assistant commands into 18 domains. Long support tickets, yes/no judgments (noul), ordinal score questions, or questions with dozens of options may behave differently. The Laya developer says the model weakens above 20 options at default settings (we used 18).
• We wrote the label descriptions. Different descriptions give different results. The pile-up into iot in particular may reflect the wording, and we did not experiment with refining the descriptions (tuning them against the evaluation data would inflate the scores).
• The decoder family ran on 180 pairs. Each value carries about ±6–7 pp of uncertainty, so this measurement cannot rank the decoder-family models against each other. What it does show is that all three scored above Laya on the same sentences.
• The evaluation sentences may have been in some models' training data. decider's card is inconsistent about MASSIVE, and we could not confirm for every model whether MASSIVE was used in training. If the English MASSIVE sentences were, the English scores may be inflated. Whether the Korean sentences were included, we could not determine.
• The Korean sentences are translated and localized. Their style and length may differ from what Korean users actually write. The stray spaces before particles did not change Laya's results, but we did not check the other models.
• Speed is from this one Mac. M1 Max, one request at a time, no batching. NVIDIA GPUs or other CPUs will differ substantially. Accuracy should not depend much on the device, but Ollaya's documentation notes that a different precision (such as fp16 on CUDA) can flip the answer when the top probabilities are nearly tied.
• We did not measure Jev. We had no TypeSafe API key and measured local models only. Larger models such as decider 4B, kev 4B and 9B, and Winnow were left out because they would take too long on this Mac.
• The sentence sample was drawn once. Decision models answer without random sampling, so the same input gives the same answer (the 536 sentences we re-sent through the router all returned the same answers and confidences). Drawing a different sample could still move the values by a few pp.

References

Primary sources and model cards were checked on 2026-09-27. The measurement script draws 30 sentences per domain from the MASSIVE test split with a fixed seed (20260927) and sends each one to Ollaya's /api/decide endpoint one at a time.

Ollaya and these models get new versions every few days. The figures here are for Ollaya 0.7.2 and the model commits current on 2026-09-27.