Benchmarking GitHub Copilot Local Models with Ollama in JetBrains

Introduction

AI coding assistants are becoming a regular part of the developer workflow, but not every team wants every coding task to depend on a cloud-hosted model.

There are several reasons for considering local models. Developers may want to experiment with local inference, reduce dependence on network connectivity, keep certain source-code workflows closer to the development machine, or compare different models before choosing a standard approach.

The combination of GitHub Copilot, JetBrains IDEs, and Ollama creates an interesting setup for this kind of evaluation.

Instead of asking only whether a local model can generate code, a more useful question is:

How does a local model compare with a cloud model when performing the same development tasks inside a JetBrains workflow?

The comparison should include more than response speed. Developers should evaluate latency, code correctness, test success, context handling, resource usage, privacy considerations, and the amount of manual correction required.

What Is a Local Coding Model?

A local coding model is an AI model that runs on the developer's own machine or within infrastructure controlled by the organization.

A simplified architecture looks like this:

JetBrains IDE
      |
      v
AI Coding Assistant
      |
      v
Local Model Runtime
      |
      v
Coding Model
      |
      v
Generated Response

With a cloud-based model, the architecture is different:

JetBrains IDE
      |
      v
AI Coding Assistant
      |
      v
Remote Model Service
      |
      v
Generated Response

Neither architecture is automatically better.

They have different trade-offs.

Why Ollama Is Interesting

Ollama provides a local runtime for running AI models on a developer machine.

The important part for this benchmark is not Ollama itself.

The interesting question is whether a locally hosted model can participate effectively in an AI-assisted development workflow.

For example:

Developer
   |
   v
JetBrains IDE
   |
   v
Copilot Workflow
   |
   v
Local Model
   |
   v
Code Generation

This makes it possible to compare local and cloud model behavior under controlled conditions.

Local vs Cloud Model

At a high level:

Area	Local Model	Cloud Model
Inference location	Developer machine or controlled infrastructure	Remote infrastructure
Network dependency	Lower for inference	Required
Hardware dependency	High	Low on developer machine
Model selection	Depends on local runtime	Depends on available service
Latency	Depends heavily on hardware/model	Depends on network and service
Data movement	Can remain local depending on setup	May leave local environment
Scaling	Limited by local resources	Provider infrastructure
Setup	Requires local model/runtime setup	Usually simpler
Maintenance	Developer/team responsibility	Provider-managed
Cost model	Hardware/resource cost	Service or subscription model

These differences make benchmarking worthwhile.

What Should Be Measured?

A common mistake is measuring only response time.

For example:

Model A: 4 seconds
Model B: 8 seconds

That tells us very little.

If Model A produces code that requires ten manual corrections while Model B produces working code on the first attempt, the slower model may actually be more useful.

A better benchmark measures:

Latency
+
Correctness
+
Test Success
+
Manual Rework
+
Context Handling
+
Resource Usage

Core Benchmark Metrics

Metric	Description
Time to first response	How quickly the model begins responding
Total response time	Time required to complete the response
Tokens generated	Amount of generated content
Compilation success	Whether generated code builds
Test success	Whether generated changes pass tests
Manual corrections	Developer changes after generation
Task completion time	End-to-end time to acceptable solution
Context accuracy	Whether the model follows repository context
CPU usage	Local processing load
Memory usage	RAM consumption
GPU usage	GPU utilization where applicable
Network dependency	Whether inference requires network access

Choose Real Development Tasks

A useful benchmark should use realistic developer tasks.

Avoid testing only:

"Write a C# class."

That is too simple.

Instead, use tasks such as:

Add pagination to an existing API.

Create unit tests for a service.

Refactor a legacy method.

Add validation to a command.

Implement an EF Core query.

Fix a failing test.

Explain and improve an existing method.

Add error handling to an API endpoint.

These tasks require the model to understand existing code rather than generate isolated snippets.

Example .NET Benchmark Task

Consider an existing service:

public async Task<Order?> GetOrderAsync(
    int orderId,
    CancellationToken cancellationToken)
{
    return await dbContext.Orders
        .Include(x => x.Items)
        .FirstOrDefaultAsync(
            x => x.Id == orderId,
            cancellationToken);
}

Now give the same task to two models:

Add pagination to the order search endpoint.
Keep the existing architecture and add tests.

The benchmark should record:

Model A
- Response time
- Files changed
- Build result
- Test result
- Manual fixes

Model B
- Response time
- Files changed
- Build result
- Test result
- Manual fixes

Now the comparison becomes meaningful.

Hardware Matters

Local model performance is heavily influenced by the hardware on which the model runs.

A local benchmark should record at least:

CPU
RAM
GPU
GPU Memory
Operating System
Model
Model Quantization
Runtime Configuration

For example:

Machine A
CPU: 8 cores
RAM: 32 GB
GPU: 12 GB

Machine B
CPU: 16 cores
RAM: 64 GB
GPU: 24 GB

The same model may behave differently on these machines.

Therefore, do not publish local-model latency without describing the test environment.

Model Size Matters

Local models are available in different sizes.

Conceptually:

Small Model
   |
   +--> Lower resource requirement
   +--> Faster inference
   +--> Potentially weaker reasoning

Large Model
   |
   +--> Higher resource requirement
   +--> Potentially slower inference
   +--> Potentially stronger reasoning

This is not a universal rule for every model, but it illustrates why model selection must be part of the benchmark.

A fair benchmark should compare clearly identified model variants.

Quantization Matters Too

Local models are often available in quantized forms.

Quantization can reduce resource requirements and make models practical on consumer hardware.

The trade-off can involve:

Model Size
Memory
Speed
Output Quality

Therefore, record the exact model variant used.

Do not compare:

Cloud Model

against:

Local Model

without identifying the local model's size and configuration.

Benchmarking Latency

Latency should be measured consistently.

For each task:

Start Timer
    |
    v
Send Prompt
    |
    v
First Response
    |
    v
Complete Response
    |
    v
Stop Timer

Record:

Time to first token or visible response
Total generation time
Number of tokens, where available

Run the same task multiple times.

One measurement is not enough because local system load and model behavior can vary.

Warm vs Cold Performance

This is particularly important for local models.

A cold run may include:

Load Model
    |
    v
Allocate Memory
    |
    v
Initialize Runtime
    |
    v
Generate Response

A warm run may already have the model loaded:

Model Loaded
    |
    v
Generate Response

These are different scenarios.

Record them separately:

Test	Meaning
Cold start	Model must initialize
Warm inference	Model already loaded
Repeated inference	Multiple tasks in sequence
Long context	Large repository context
Short context	Small task

This makes the results much easier to interpret.

Benchmarking Code Correctness

Speed does not matter if the generated code is incorrect.

For .NET tasks, use automated validation.

A basic pipeline is:

AI Response
    |
    v
Apply Changes
    |
    v
dotnet build
    |
    v
dotnet test
    |
    v
Review

For example:

dotnet build

dotnet test --no-build

If the build fails:

Build Failure
     |
     v
Record Failure
     |
     v
Count Manual Corrections

Do not silently fix the generated code before recording the original result.

Otherwise, the benchmark will overstate model quality.

Measuring Manual Rework

Manual rework is one of the most useful metrics.

Suppose a model generates:

100 lines of changes

but the developer changes:

45 lines

afterward.

That tells us something about practical usefulness.

Track:

Added Lines
Modified Lines
Deleted Lines
Developer Corrections

More importantly, categorize the corrections.

For example:

Syntax
Architecture
Logic
Tests
Naming
Performance
Security

A model that generates syntactically correct code but repeatedly violates architecture rules may require significant review effort.

Repository Context Testing

AI coding becomes more difficult when the task spans multiple files.

Consider:

OrdersController
      |
      v
OrderService
      |
      v
OrderRepository
      |
      v
AppDbContext

Ask the model to modify the complete workflow.

Now evaluate:

Did it find the correct service?
Did it follow the existing pattern?
Did it modify the correct repository?
Did it update tests?
Did it preserve existing behavior?

This is more representative of real development than single-file generation.

Testing Large Contexts

Create several test levels.

Small Context

One class
One method

Medium Context

Three to five related files

Large Context

Multiple projects
Shared services
Database layer
Tests
Configuration

Then measure how performance and correctness change.

A useful result might look like:

Context	Local Model	Cloud Model
Small	Measure	Measure
Medium	Measure	Measure
Large	Measure	Measure

The numbers should come from controlled experiments rather than assumptions.

Privacy Considerations

One reason organizations investigate local models is data control.

With local inference, depending on the exact setup:

Source Code
    |
    v
Local Runtime
    |
    v
Local Model

The code may remain within the developer's environment.

However, do not automatically assume that every local setup is completely isolated.

The complete system may still include:

IDE
Extensions
Authentication
Package Downloads
Telemetry
Remote APIs
Model Downloads

Therefore, a privacy assessment should consider the entire workflow rather than only the model runtime.

Offline Development

Local models can be useful when developers need reduced network dependency.

A simple test is:

1. Download required tools and model.
2. Disconnect from the network.
3. Start the IDE.
4. Load the repository.
5. Run the AI task.
6. Build the project.
7. Run tests.

Then record what still works.

This identifies hidden network dependencies.

The result might be:

Model inference: Works
Build: Works
Tests: Works
Package restore: Fails
External API: Fails

That is much more useful than simply claiming the workflow is "offline."

Comparing Code Quality

Quality should be evaluated using objective criteria.

For example:

Category	Evaluation
Compilation	Builds successfully
Tests	Existing tests pass
New tests	Appropriate tests added
Architecture	Existing boundaries respected
Security	No obvious unsafe patterns
Maintainability	Code remains understandable
Consistency	Matches repository conventions
Performance	No obvious unnecessary work

A simple scoring system can be created for internal experiments.

For example:

Compilation       20
Tests             20
Correctness       20
Architecture      15
Security          15
Maintainability   10
Total            100

The weights should reflect the team's priorities.

Example Benchmark Workflow

A practical experiment could look like:

Repository
    |
    +--------------------+
    |                    |
    v                    v
Cloud Model          Local Model
    |                    |
    v                    v
Same Task            Same Task
    |                    |
    v                    v
Build                Build
    |                    |
    v                    v
Tests                Tests
    |                    |
    +---------+----------+
              |
              v
        Compare Results

Keep everything else constant.

Avoiding an Unfair Benchmark

Several factors can make a comparison invalid.

Different Prompts

Use the same prompt.

Different Repository State

Use the same commit.

Different Tasks

Use equivalent tasks.

Different Context

Provide comparable context.

Different Hardware

Document the hardware.

Different Model Settings

Record the model and relevant configuration.

Different Evaluation Criteria

Use the same tests and scoring process.

A benchmark is useful only when the comparison is controlled.

Common Mistakes

Mistake 1: Measuring Only Speed

Fast incorrect code is not useful.

Mistake 2: Comparing Different Models Without Recording Versions

Model variants can behave significantly differently.

Mistake 3: Ignoring Hardware

Local inference depends heavily on available resources.

Mistake 4: Testing Only Simple Prompts

Single-file generation does not represent enterprise development.

Mistake 5: Fixing Code Before Recording Results

This hides model errors.

Mistake 6: Using Only One Run

Inference performance can vary.

Mistake 7: Ignoring Cold Starts

A model that performs well when already loaded may behave differently when launched for the first time.

Mistake 8: Treating Privacy as Automatic

Local inference reduces some data-transfer concerns but does not automatically guarantee complete isolation.

Troubleshooting

Problem	What to Check
Model responds slowly	Hardware, model size, quantization, context size
High RAM usage	Model size and runtime configuration
GPU memory exhausted	Model size and available VRAM
Model cannot follow repository conventions	Context retrieval and project instructions
Generated code does not compile	Model capability and task complexity
Tests frequently fail	Prompt quality, context, model capability
Cold startup is slow	Model loading and hardware
Offline workflow fails	Hidden network dependencies
Results vary significantly	Run multiple trials and control system load

Best Practices

Record the Complete Environment

Document:

IDE Version
Runtime Version
Model
Model Variant
Hardware
Operating System
Prompt
Repository Commit

Use the Same Repository State

Do not compare models against different code versions.

Automate Evaluation

Use:

dotnet build
dotnet test

where appropriate.

Run Multiple Trials

Avoid drawing conclusions from a single response.

Separate Cold and Warm Tests

They represent different real-world scenarios.

Measure Rework

Developer correction effort is often more useful than raw generation speed.

Include Large-Context Tasks

Enterprise development frequently involves multiple files and projects.

Test Failure Cases

A useful benchmark should include tasks where the model might reasonably struggle.

Keep Security in the Evaluation

Generated code should be reviewed for insecure patterns just like human-written code.

Advantages

Reduced Network Dependence

Local inference can continue without requiring every AI request to travel to a remote model service, depending on the complete setup.

Greater Infrastructure Control

Organizations can control the hardware and runtime used for local inference.

Useful for Experimentation

Developers can compare different local models and configurations.

Potential Data-Control Benefits

Source code can remain within a controlled environment during inference when the complete setup is configured accordingly.

Flexible Hardware Optimization

Teams can select hardware appropriate for their chosen models.

Disadvantages and Limitations

Hardware Requirements

Larger models can require substantial memory and compute resources.

Model Management

Teams must download, update, configure, and maintain local models.

Performance Can Vary

Local performance depends heavily on hardware and model configuration.

Quality Differences

A smaller local model may not perform as well on complex coding tasks as a more capable model.

Developer Setup

Local AI environments introduce additional tooling and configuration.

Offline Does Not Mean Fully Isolated

Other parts of the development workflow may still require network connectivity.

A Practical Benchmark Plan

For an enterprise .NET team using JetBrains IDEs, a useful first experiment can be organized into five stages.

Stage 1: Select the Environment

Record:

JetBrains IDE
.NET SDK
CPU
RAM
GPU
Operating System

Stage 2: Select the Models

Record:

Model Name
Model Size
Quantization
Runtime Configuration

Stage 3: Select Tasks

Choose:

Code Generation
Refactoring
Unit Tests
Bug Fixing
EF Core Query
API Development
Cross-File Change

Stage 4: Automate Validation

Run:

dotnet build

dotnet test

Also perform code review against predefined criteria.

Stage 5: Compare Results

Record:

Latency
Build Success
Test Success
Manual Corrections
Task Completion Time
Resource Usage

This produces an engineering benchmark rather than a subjective product comparison.

Example Evaluation Sheet

Test	Cloud Model	Local Model
Simple C# generation	Measure	Measure
Refactoring	Measure	Measure
Unit test generation	Measure	Measure
EF Core query	Measure	Measure
API change	Measure	Measure
Multi-file change	Measure	Measure
Large context	Measure	Measure
Cold start	Measure	Measure
Warm response	Measure	Measure
Offline operation	N/A or measure	Measure

The results should be interpreted based on the organization's priorities.

For one team, privacy may be the most important factor.

For another, task completion quality may matter more.

For another, local hardware cost may make cloud inference more practical.

How to Interpret the Results

Avoid declaring a universal winner.

Instead, ask:

Which model is better for which task?

You may discover:

Local Model
Good for:
- Simple refactoring
- Documentation
- Basic tests
- Offline experimentation

while:

Cloud Model
Good for:
- Complex reasoning
- Large-context tasks
- Difficult debugging
- Architecture-heavy changes

These are examples of the type of conclusion a benchmark might reveal, not predetermined results.

The actual outcome should come from testing.

Conclusion

Local AI models are becoming an interesting option for developers who want more control over their coding-assistant environment. Combining a JetBrains development workflow with GitHub Copilot and Ollama provides a practical setup for testing local inference against cloud-based alternatives, but the comparison should be based on real engineering work rather than response speed alone. A useful benchmark should measure latency, compilation, test success, repository-context handling, manual rework, resource consumption, cold-start behavior, and offline operation. Hardware and model configuration must also be recorded because they can significantly influence local performance. For enterprise .NET teams, the biggest value of this kind of benchmark is not finding one universal winner. It is understanding which model and deployment approach works best for specific development tasks while balancing code quality, developer experience, resource requirements, and data-control considerations.