Passing the Swedish Medical Licensing Exam by Post-Training Open-Weight LLMs

In this post: SFT and RLVR on Swedish MedQA

(10 min read)

To practice medicine in Sweden, foreign medical doctors who want to validate their license are first required to pass a multiple-choice test. Given around 140 questions with 5 answer options each, they must reach at least 60% accuracy to pass.

This theory test is challenging, with a pass rate of just about one in four and only 54% of participants ultimately succeeding after one or more attempts.

I was curious how well newer open-weight LLMs could do on this task by now. This post documents what turned into a little summer project, pushing the accuracy of smaller open-weight LLMs on Swedish MedQA with post-training techniques like supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR).

Related Work

For English language, similar benchmark datasets with medical multiple-choice questions like MedQA have already been mostly saturated, with frontier models scoring above 95%.

Swedish, however, accounts for less than 1% of pre-training data for LLMs, making it a ‘medium resource language’. Its medical vocabulary, local treatment standards and medication also do not cleanly translate from English MedQA.

Fortunately for me, MedQA-SWE released by Hertzberg & Lokrantz in 2024 is composed of exactly the medical doctors’ licensing exam data I was interested in. LLMs were evaluated on this data both in the original paper and also by Moëll et al. in the 2025 Swedish Medical LLM Benchmark (SMLB).

When published a year ago, it already reported several open-weight models as capable of passing the test on data of MedQA-SWE, with Gemma2-9B as the smallest (with 61.3%) and DeepSeek R1 Distill Llama-70B as the most accurate (with 77.8%).

Data, Prompt and Evaluation

The relevant dataset used in the SMLB paper overlaps with MedQA-SWE. To avoid contamination through shared samples, I therefore isolated the MedQA-SWE samples with the most recent exam date as a validation set (158 samples) and kept the rest of it in reserve for training (3022 samples). All results reported here refer to greedy decoding on that validation set, which has no overlap with the training set. The prompt and evaluation harness are based on those used by Moëll et al., expecting the correct answer to be stated verbatim.

MedGemma-1.5-4B

Just six months ago, researchers at Google Health released MedGemma-1.5-4B for medical tasks. It has multimodal capabilities for both medical image analysis and clinical reasoning, so it seemed like a perfect candidate for this task.

When deployed out-of-the-box for inference, it only reached a somewhat disappointing 14.6% accuracy, however. Many of its outputs actually give the correct answer eventually, but fail the formatting requirement of stating the correct answer option verbatim. Instead, the model tends to digress with lengthy explanations that the evaluation harness scores with zero.

This failure in instruction-following appears to be a known failure mode of MedGemma-1.5-4B, even for English. In fact, the authors provide their own post-training example for resolving this, as well as one for SFT.

While working on this, I also found another paper, PeruMedQA by Carrillo-Larco et al., that successfully used the SFT strategy to adapt an older variant of MedGemma to Peruvian medical licensing exam questions in Spanish language.

So I decided to set up a similar approach here, using the MedQA-SWE samples from prior years as training data for SFT with LoRA adapters on MedGemma-1.5-4B, to try and get the model to respond with the correct answer option verbatim. To reduce the risk of memorization, all answer options were randomly shuffled with re-assigned letters during training.

This post-training strategy raised the model accuracy on the validation set from 14.6% to 60.8% on the first attempt, sufficient for passing the exam. Find the implementation on GitHub.

Rather than outright instilling new medical knowledge or improving Swedish language comprehension of the model, the predominant impact of SFT from such a small training set is improved format compliance. By learning to state the correct answer verbatim and nothing else, the model learns to generate outputs that never get truncated and are correctly scored by the evaluation harness. The underlying medical knowledge was likely already baked in the model weights from the start.

Overall, this is a rather interesting result for a 4B parameter model, considering that a minimum of 9B was reported for this just a year ago. But turns out that we can do even better.

Gemma4-E4B

After Google Health released MedGemma-1.5-4B in January, which built on the Gemma3 series, Google DeepMind more recently released the newer Gemma4 series in April.

When applying its Gemma4-E4B variant to the chosen validation set, it reached 77.2% accuracy out-of-the-box. With no post-training, it reaches almost perfect format compliance, robustly handles the prompt, questions and answers in Swedish language and exceeds the 60.8% accuracy of the fine-tuned MedGemma-1.5-4B reported above. It would appear as if, within a span of three months, Google has been left behind by Google.

Even its smaller variant Gemma4-E2B reached 52.5%. However, neither seemed to respond well to SFT. The recipe from above failed to improve their accuracy, perhaps due to their already excellent format compliance. Out of curiosity, I decided to venture outside of the Google ecosystem from here.

Qwen3.5-4B

Released by Chinese Alibaba in March this year, Qwen3.5-4B (with 77.2% accuracy here) and Qwen3.5-2B (with 46.8%) turned out to perform virtually on par with their similarly sized counterparts from Google. Again, the SFT recipe did not yield any improvement for these.

However, I was curious to see how far the performance could be pushed by enabling reasoning for the 4B model. In the original SMLB paper, o3 in particular was noted for its good performance due to its reasoning capability. In the given project here, Qwen3.5-4B could also generate more or less extensive reasoning traces enclosed in blocks that could be stripped before scoring the verbatim answer.

With reasoning enabled, Qwen3.5-4B improved from 77.2% (with ‘no thinking’) to 88.0% accuracy. These gains, however, come at a cost of substantially longer inference times. From the verbatim responses with up to just ~50 tokens, these reasoning traces can grow to 7,000 tokens and beyond. Three of them never concluded despite the reasoning traces showing a strong preference for the correct option, exhausting the entire context window limit of 32,768 tokens with repetitive reasoning loops about formatting formalities. Interestingly, all reasoning by this model is exclusively in English, in spite of the Swedish prompt, question, answer options and final response.

This uncapped reasoning can take up to five minutes for a single answer even on an A100 GPU. So from here, I looked into several approaches for getting the best of both worlds: Condensing as much of the accuracy gains from reasoning as possible into shorter and thus faster reasoning traces.

Reasoning Compression

Truncation with max_tokens

A simple way to cut down on inference time is a hard limit, stopping all outputs at a maximum count of 1024 tokens per sequence. Doing this here yields an answer within about 10s, but often truncates the model responses mid-reasoning, so that only 29.1% are completed with the correct answer.

Thinking Intervention

A more sophisticated type of thinking intervention is proposed in a 2025 paper on early-exit techniques for prevention of overthinking in LLMs.

It halts the decoding at a pre-determined token count and appends the phrase Time is limited, stop thinking and start answering. \n\n\n to conclude the reasoning block with a soft transition. Decoding then proceeds, making the model provide its actual response immediately.

For my experiments on MedQA-SWE here, I applied this technique at a token limit of 896…