Extending Existing LLMs with Ollama’s Modelfile: A Practical Guide

Large Language Models (LLMs) are incredibly powerful, but out-of-the-box, they may not always fit your exact use case. Maybe you want your model to answer in a certain style, be restricted to a domain (like healthcare or finance), or load in extra context. Instead of training from scratch — which is expensive — Ollama provides a lightweight way to extend existing models using its Modelfile.

What is Modelfile?

A Modelfile in Ollama is like a Docker file for LLMs. It defines how your custom model should be built, based on an existing base model.

You can extend any available model (e.g., llama2 , mistral , gemma , etc.).
Add system prompts, defaults, or extra parameters.
Keep the efficiency of the base model while customizing behavior.

1. Create a repo/folder structure

In this example, we are creating two AI Assistants.

med-assistant
sarcastic-bot

Each folder has a separate Modelfile. Please note that the name of the Model file must be Modelfile without any extension.

  
    FROM gemma3:1b

SYSTEM """
You are a medical advisor who provides clear, professional guidance.
If asked about non-medical topics, politely decline.
"""

PARAMETER temperature 0.2
PARAMETER num_ctx 4096

That's it — you've just created a medical-focused AI assistant, without training a single parameter.

2. Create AI Assistance and Run it

Go inside the med-assistant folder from the command line and execute the following command.

  
    ollama create med-assistant -f Modelfile
ollama run med-assistant

Best Practices

Start with a solid base model (Mistral, LLaMA2, or Gemma are good choices).
Keep your system prompts short — long ones eat into your context window.
Use PARAMETER num_ctx wisely if your model needs to handle long documents.
Version your Modelfiles ( med-assistant-v1 , med-assistant-v2 ) for reproducibility.

Conclusion

Ollama's Modelfiles give you the power to extend LLMs without the cost of training .
They're fast, flexible, and developer-friendly — making it possible to spin up specialized assistants or fun chatbots in minutes.

If fine-tuning is like teaching your model new skills, Modelfiles are like giving it a job description and set of rules.