New model

State-of-the-art in 24 hours

We are introducing Llama-3-8B-Meditron v1.0, a new medical Large Language Model (LLM) with 8 billion parameters, fine-tuned within 24-hours of the release of Meta's Llama-3. This new model outperforms all state-of-the-art open models within its parameter class on standard benchmarks such as MedQA and MedMCQA. It also outperforms Llama-2-70B and is within 10% of the current leading open model for medicine in the 70B range: Llama-2-70B-Meditron.

This work shows the innovative potential of open foundation models with widely available weights and is part of a larger initiative to ensure equitable participatory access to this technology in low-resource settings.

Results

Using the LM-Evaluation Harness benchmarking tool, we compared Llama-3-8B (base) vs. Meditron-7B vs. Llama-3-8B-Meditron v1.0 on 3 standard medical question-answering benchmarks (MedQA, PubMedQA, MedMCQA).