Microsoft open-sourced code-testing-generator, a polyglot unit-test agent that understands repositories before generating tests; internal results
Microsoft's .NET team has recently open-sourced a cross-language unit-test agent called code-testing-generator, whose goal is to first help the Agent understand a project’s structure, test tools, and existing testing conventions, and then generate and validate unit tests.[1][3]
According to public information, this agent is located in the dotnet-test plugin inside the GitHub dotnet/skills repository, and can be used through GitHub Copilot CLI, Visual Studio Code, and VS Code Insiders. Microsoft also says Visual Studio support is being prepared.[2][3]
Microsoft’s internal evaluation shows that the agent completed 140 out of 152 tasks, for a completion rate of 92.1%, while native GitHub Copilot under the same conditions, using the same AI model and prompt, achieved a completion rate of 78.9%.[2][4]
This difference does not reflect model capability alone, but rather the workflow design difference between a “specialized Agent” and a “general coding assistant”: the former reviews the project step by step in a repository-aware way before deciding how to generate and validate tests, while the latter is more oriented toward one-shot generation.[2][4]
From a process perspective, the core value of this agent is not “writing tests” itself, but “understanding before generating.” It first scans the code repository, identifies the target code, test framework, test file locations, and execution method, then generates test code based on the project’s current state, and checks whether the new tests are actually integrated into the existing automation workflow.[1][3]
This design is especially important for unit tests, because the purpose of unit testing is to verify whether individual functions behave as expected under specific conditions; if tests can run independently but are not connected to the project’s build and test pipeline, their practical value is greatly reduced.[1][6]
Public materials indicate that the agent does not only generate tests; it also checks coverage for specific scenarios, the effectiveness of assertions, and the build and test results of the full project, and it verifies whether the tests can really detect behavior changes by making small modifications to the code under test.[1]
This is critical because a common problem with AI-generated tests is not that they fail to compile, but that they appear to pass while missing key logic. By changing the target code and observing whether the tests fail, the agent can assess whether the tests have genuine defect-detection ability.[1][3]
Microsoft’s results also show that the advantage of this type of Agent appears mainly when requirements are vague. Among 89 more ambiguous tasks, the specialized agent completed 79, for a completion rate of 88.8%, while native Copilot completed 59, or 66.3%; but among 63 tasks with detailed instructions, both completed 61.[4]
This means that when the prompt is already sufficiently clear, the gap between a general model and a specialized agent narrows; the real difference comes from understanding repository context, task decomposition ability, and the ability to land tests in the existing framework.[2][4]
There were also 15 tasks that required adding tests for specific code changes. The specialized agent completed all 15, while native Copilot completed none, showing a structured advantage in tasks such as filling in regression tests based on change content.[4]
In terms of supported languages, this agent supports .NET, Python, TypeScript, JavaScript, Java, Go, Ruby, Rust, Swift, Kotlin, PowerShell, and C++, and it generates tests according to each project’s existing practices.[1][4]
However, Microsoft also acknowledged that not all languages showed improvement. For example, in 10 PowerShell tasks, the specialized agent completed 7 while native Copilot completed 8, showing that results may fluctuate when sample sizes are small and language characteristics differ.[4]
Therefore, this evaluation is better interpreted as “repository-aware agents are more reliable in most scenarios,” rather than as a claim that any language and any project can consistently outperform a general AI assistant.[4]
For enterprise development teams, the direct value of this kind of tool is reducing the time needed to add tests, especially in cases involving scattered legacy code, modules with insufficient test coverage, and situations that require rapid creation of regression tests.[2][4]
If a team uses common .NET testing frameworks, or maintains consistent testing conventions in a multilingual repository, this kind of Agent can first identify the framework and execution method, then integrate the tests into the real CI process, reducing the risk of “tests exist, but they do not run.”[1][6][14]
For security and quality governance, the impact is not only about development speed, but also about observability and maintainability. When tests can be automatically checked for build-pipeline integration and for whether they fail after code changes, teams can detect regressions earlier.[1][3]
Another important aspect is where data and code are retained. Public information indicates that this is not a hosted service, but a skill defined within an existing coding agent, so it can run in the local workflow and keep code within the user’s environment.[2]
This is especially important for organizations that care about source code control, because test generation often touches unpublished business logic, internal naming, and architectural details; local execution can reduce additional external exposure.[2][4]
That said, Microsoft clearly warns that the test sample sizes for some languages are small, and the test data cannot be directly taken to represent the real performance of all projects; therefore, the 92.1% figure should not be interpreted as a universal guarantee, but rather as an internal benchmark with reference value.[4]
If an enterprise is preparing to adopt this kind of unit-test agent, the first step should be to confirm that the tests it produces are actually integrated into CI/CD, rather than remaining only at the local or temporary execution level.[1][3]
The second step is to check whether the tests cover key branches, exception handling, and boundary conditions, and to validate test effectiveness using mutation thinking; if the tests still pass after the program behavior is changed, that usually means the test signal is too weak.[1]
The third step is to inventory the current state of each language and test framework before adoption, to avoid a situation where the Agent generates content but cannot correctly connect to the existing toolchain.[1][6][14]
Fourth, for PowerShell or languages with smaller sample sizes, a limited pilot should be conducted first, rather than applying a single benchmark result to the entire organization.[4]
Fifth, if the team places high importance on source code privacy and supply-chain risk, it should prioritize a mode that can run locally within the existing development environment, and inspect its permissions, plugin installation source, and repository access scope.[2][3]
5-step remediation checklist