In my software development work, I use tools like GitHub Copilot and ChatGPT daily. However, I’m interested in exploring self-hosted alternatives that don’t depend on external services.

Disclaimer

It’s important to understand that these tools, while helpful, don’t replace the developer’s role. A common misconception is that AI tools can do the job for you, which is simply not true.

Developers are still crucial for problem-solving, understanding requirements, and applying generated code effectively. Automating code snippets from sources like Google or Stack Overflow doesn’t change this; the developer remains responsible for the final result.

The experiment

Following my recent exploration of running LLMs locally with Ollama previous post, I wondered if these models could replace GitHub Copilot.

My goal isn’t to cut costs (my employer covers these tools) or prioritize privacy (keeping code “secret” is moot when using GitHub). Instead, I’m curious about the capabilities of these alternatives – and doing this for the fun of it!

Setup and available resources

My main machine is a MacBook Pro with an M2 Pro CPU and 32 GB RAM. This setup comfortably handles my core requirements:

• Slack
• Docker (for work containers)
• VSCode
• Safari / Chrome

The question is: can it also handle a Copilot substitute?

Tested models and extensions

To replicate Copilot’s functionality, you need an LLM and a compatible VSCode extension (similar extensions exist for other IDEs).

I’m running the models locally using Ollama. For details on its usage, refer to my previous post.

The tested models are:

• stable-code:3b-code-q4_0
• codellama:7b-code-q4_K_M
• codellama:13b-code-q4_K_M

The number preceding “b” signifies the parameter count (billions), with each model requiring roughly 4 GB, 8 GB, and 16 GB of RAM, respectively.

Meta offers even more potent models with 34 billion and 70 billion parameters (found at https://ollama.com/library/codellama), but these demand 64-128 GB RAM.

My tested extensions include:

• LLama Coder https://marketplace.visualstudio.com/items?itemName=ex3ndr.llama-coder
• Code GPT https://marketplace.visualstudio.com/items?itemName=DanielSanMedium.dscodegpt
• twinny https://marketplace.visualstudio.com/items?itemName=rjmacarthy.twinny

Results

My findings were inconsistent, primarily influenced by the LLM model rather than the extension.

While stable-code:3b-code-q4_0 provided rapid code completion, its output was frequently incorrect or nonsensical. It struggled with basic Python method structuring and indentation.

Models like codellama:7b-code-q4_K_M and codellama:13b-code-q4_K_M demonstrated improved accuracy. However, despite my system’s resources, their 3-4 second completion times rendered them impractical for my use.

None of the tested models came close to matching GitHub Copilot’s speed and accuracy.

Conclusion

Although the concept of a personal, private code assistant is appealing, replicating GitHub Copilot’s performance locally proves difficult with current technology.

However, I’m optimistic about the future. As models and extensions evolve, we can expect enhancements in quality and potentially reduced resource requirements.

I welcome any suggestions for superior models or extensions; feel free to share in the comments. I’m always eager to experiment and update this post or create new content.

For now, GitHub Copilot remains my preferred choice.