GitHub Copilot (Gen-AI) coding tool, as of June 2025, looks much more capable than it did 3 months ago.
Abstract: After the appearance of the GitHub Copilot Agent, I decided to try it on my real-life ASP.NET8 project. I tried assisted coding and produced 5.300 lines of code, of which 95% was generated. There is a very, very big quality advancement in usability of this version compared to the version I tried 3 months ago.
1. Real-life project in VS2022
I am working on the development of C#/.NET8/ASP.NET application, which now has around 123.000 lines of code (SLOC), in Visual Studio 2022.
The environment is,
- Visual Studio 2022, 17.14.5
- GitHub Copilot (GHC). License Copilot Pro+
- Agent mode, GPT-4o, GPT-4.1, Claude 3.7 Sonnet
- C#/.NET8/ASP.NET/JavaScript/jQuery/HTML/CSS/Bootstrap/SQL-Server
2. Trying GitHub Copilot
2.1. March 2025
I first seriously tried to use GitHub Copilot (GHC) on my project in March 20025, and I detailed the results in the article [1]. I found it practically useless, except for short snippets.
2.2. June 2025
After the appearance of the GitHub Copilot Agent, I decided to try it again. It is described in [2]. [3], [4], [5]. It showed so far as a good tool for template code cloning and generation
2.3. Comparison
I was selfish; I was just interested in how useful this Gen-AI tool could be to my project work. I did not care if they called it “chat mode” or “agent mode”; I just wanted to see what it could do for me and my project.
| |
GitHub Copilot (GHC) March 2025 |
GitHub Copilot (GHC) June 2025 |
| Main functionality
|
Ghost Text
GHC Chat
|
Ghost Text
GHC Chat
Agent Mode
|
|
Prompt engineering
|
Requires strict use of # for files, etc.
|
Natural language accepted
|
|
Ghost Text
|
Very good predictions
|
Very good predictions
|
|
Syntax Errors in C#
|
- Numerous
- not sure of the nullability of variables, mixes string with string?, leave all that mess to the user to fix it
- Has a problem with inserting snippets of code into the app; the user needs to add/remove brackets
- Generated code can not be compiled.
|
No syntax errors at all. All compiled.
|
|
Coding style
|
Does not follow user instructions strictly, sometimes enforces his own ( from who knows which textbook) style
|
Does not follow user instructions strictly, sometimes enforces his own ( from who knows which textbook) style
|
|
Hallucinated methods and properties
|
- A huge number of hallucinated methods and properties
- The code does not compile at all
- Leaves it to the user to manually resolve the mess and refer to proper methods and references
- Generated code can not be compiled.
|
- No hallucinated methods or properties at all. All methods and properties refer to real classes. All compiled.
- Only unresolved stay some new strings from .resx file, it looks like it can find properly existing strings in .resx but can not add new ones.
|
|
GHC chat
|
- Serious focus problem, just babbles sometimes
- Tend to waste time with verbose answers off-topic
|
Not sure yet
|
|
Follow user instructions closely
|
Does not follow user instructions strictly, improvises what he (GHC) thinks is better.
|
Does not follow user instructions strictly, improvises what he (GHC) thinks is better.
|
| |
GitHub Copilot (GHC) March 2025 |
GitHub Copilot (GHC) June 2025 |
| Small, limited-scope tasks
|
Excels
|
Excels
|
|
a little project of 4 C# files
|
Does 1 file properly. The task needs to be divided into subtasks and given explicit instructions for each file.
|
Does three files properly. The task needs to be divided into subtasks and given explicit instructions for each file.
|
|
Clone with modifications a 200-line C# method
|
It has some syntax errors and hallucinated properties and methods. Can not be compiled.
|
- Does perfectly well. All compiled.
- Really good work, changed all variable names, comments, figured out proper EF LINQ, method parameters.
|
|
Leaves unfinished work to the user
|
- Leaves partly usable code, full of syntax errors, and hallucinated properties/methods
- Requires a lot of user work to manually fix all the errors
|
- All compiles
- No or very little work for the user to fix the generated code
|
|
Template-based task
|
- Poor results even for a task carefully chosen for GHC
- The final generated code is of such poor quality and requires so much manual rework that it is not worth using.
|
- Excels in template-based tasks
- Does perfectly well. All compiled.
- Really good work, changed all variable names, comments, figured out proper EF LINQ, method parameters.
- Even found in a project, which icons should be used in singular vs plural for Account/Accounts.
- Still, small logical errors if it can not figure out why some parameters are passed in JS
|
|
Usability for a practical ASP.NET project
|
- The number of syntax errors and hallucinated code was so big that it required huge manual rework.
- Practically useless, except for short snippets
|
- All compiled. Practically no syntax errors or hallucinated properties/methods
- Good tool for template code cloning and generation
|
3. Project work done
3.1. Work with GHC assistance
- In my project, I used GHC assistance to add around 5.300 lines of code, out of which 95% is generated by GHC. (measured by Visual Studio Code Metrics tool).
- All that was done in the new Agent mode
- This was still a task carefully chosen for GHC, in the sense that it is template-based. It only needed to clone the code and adapt it from the Account entity to the Contract entity.
- Since this was a template-based task, carefully chosen for GHC, I am still not sure about usability for general code generation for brand-new methods/functions. Needs to be verified first.
3.2. Things I did manually
- It looked like it had problems with locating and creating Razor .cshtml files. So I created empty files with proper names in the proper location and let GHC fill the content
- It looked like it could not add new strings to .resx files. But it created string names in the code. I added proper 4 language strings to .resx
- For simple HTML buttons here and there, it is easier to just add them than to write a command prompt
- Moved code/methods, reorganized a bit to my liking
- Some sophisticated EF-LINQ expressions I modified manually. Easier to explain it in the command prompt
- GHC tends to omit C# comments, and I like my comments, so I added/copied them manually
3.3. Perspective of a programmer
- I was selfish; I was just interested in how useful this Gin-AI tool could be to my project work. I did not care if they call it “chat mode” or “agent mode”, just what it can do for me
- In this version of GHC, syntax errors are very rare, but there were a few logical errors.
- It is a bit of a limitation of us humans to review all the changes if changes are spread across too many places. I refused all that, even if that may be OK. I make GHC do the work again
- I do not see the programmer role disappearing, it is going more be on a higher level, leading and verifying the work of Gen-AI
- I GO SLOW AND REVIEW every line of generated code. Slow in review, but altogether saves time later because of fewer defects.
- So, basically, it is a Generate-Review-Drop-Generate-Review-Accept cycle as a work process. Reviewing is a burden on humans and requires significant effort, because you are reading someone else’s code, but I just do not want to accept anything into my repository that I haven’t read and understood.
- It is a bit of a limitation of us humans to review changes. That GHC thing can generate 200 lines of valid code every 10 minutes, but I can not read and understand that code that fast. There is also human fatigue appearing, you get tired from reading and reviewing the generated code whole day.
- It actually feels (on this smaller task, based on produced results) a bit like human coworker, does not figure it right away, so you need to explain details more, it does quality job, but it does not do exactly what you asked and does some things his own way, and sometimes is more clever that you think it will be.
- Based on modifications GHC made to the code of my original template C#/cshtml 4 files, I was able to follow “how GHC is thinking”, and it does think like a good programmer.
- It does feel as if you have a competent coworker who gets instructions from you, and you can delegate work to him/her.
- Commands to GHC are in practically natural language, no “pseudo-science” prompt engineering needed, I spontaneously stopped using # for marking file names, and it all worked. I just reasonably explain like I would to some other developer in an email.
4. Weird behaviors of GHC – probably bugs
It seems there are still bugs in the product. Plain VS2022/GHC restarts from time to time are beneficial.
I was working on some plain HTML and was giving a task to GHC to add some JavaScript, and the GHC thing was suddenly so stupid. For an hour, I was changing different LLMs, it was telling me it added JS to the code, and I did not see anything or just unfinished fragments. I was thinking, how come it can do so complicated tasks, and can’t add just 30 lines of JS? It took me an hour to figure out. It all disappeared when I restarted VS2022. It seems that automation in Agent mode was stuck somewhere; it was not reading or writing files. It looks like GHC keeps some tmp files where it remembers its work. Some file corruption somewhere. It started to work well after a restart.
5. Intellectual Property and Copyright
I noticed that in point of time,the GHC Agent loaded 750 source files of my project. It was looking for something, trying to find something, or learning about the project. Is he learning from my code? So, if I work with GHC, all my code is Open Source?
For example, I am working on a specific Bank protocol ABCD. After working 2 weeks with GHC, it has practically loaded all my source files. Has he learned and memorized all that?
What if a month later, some other user, somewhere, asks GHC to implement a similar app with the ABCD Banking protocol? That GHC thing can, in several minutes, create a whole app based on what it learned from my source code. And, it can, of course, easily make it unrecognizable/different from my source code.
Where are my legal “Intellectual Property” and “Copyright” rights? You get the idea.
Who can guarantee that those AI companies are not behaving like pirates? They already claim that used all the text on the Internet ([7]) and their systems are still “text hungry”.
6. Conclusion: Seeing is Believing
I was a great sceptic, with a good reason. Marketing on AI is very strong, and Rock-star CEOs tend to overpromise in order to sell their company's products and shares. An example is Tesla’s Autonomous drive promised in December 2015 ([6]).
I was very disappointed in the version of GHC from March 2025. It did not look practically usable. The number of syntax errors and hallucinated code was so big that it required huge manual rework. Also, there were many hallucinated properties and methods, meaning that GHC did not look into other files of the project, just working on one.
In contrast, the version of GHC from June 2025 is practically without syntax errors. Hallucinated methods and properties are all gone, meaning GHC now reads (a number of) project source files to resolve dependencies. Delivered generated code all compiles, but it can have logical errors. GHC is now quite usable for practical projects.
Progress in the quality of GitHub Copilot in 3 months from March 2025 to June 2025 is much more than we usually see for software products. If it continues to progress in quality like this…well, by the end of the year, we will have a very smart code assistant.
So, the version of GHC from June 2025 starts to look like what all those media personalities were talking about, a replacement for Junior/Intermediate level programmers ([8]). I am starting to see the basis for that prediction.
7. References
[1] GitHub Copilot (Gen-AI) is Helpful, But Not Great (March 2025)
https://markpelf.com/2717/github-copilot-gen-ai-is-helpful-but-not-great-march-2025/
[2] GitHub Copilot Agent looks promising - Part 1 (June 2025)
https://markpelf.com/2744/github-copilot-agent-looks-promising-june-2025/
[3] GitHub Copilot Agent looks promising – Part 2 (June 2025)
https://markpelf.com/2746/github-copilot-agent-looks-promising-part2-june-2025/
[4] GitHub Copilot Agent looks promising – Part 3 (June 2025)
https://markpelf.com/2748/github-copilot-agent-looks-promising-part3-june-2025/
[5] GitHub Copilot Agent looks promising – Part 4 (June 2025)
https://markpelf.com/2750/github-copilot-agent-looks-promising-part4-june-2025/
[6] There’s a Very Simple Pattern to Elon Musk’s Broken Promises
https://www.wired.com/story/theres-a-very-simple-pattern-to-elon-musks-broken-promises/
[7] Microsoft AI CEO Says Almost All Content on the Internet Is Fair Game for AI Training
https://www.entrepreneur.com/business-news/microsoft-ai-ceo-anything-on-open-web-fair-use-for-training/477030
[8] Zuckerberg claims AI will replace mid-level developers, halting hiring in Korea
https://biz.chosun.com/en/en-it/2025/01/13/GLJEIP6PGRHIFC77IB6Q747DCQ/