Artikkeli
As AI agents accelerate code production, traditional end-to-end testing has become a major bottleneck due to brittle, high-maintenance frameworks that struggle with minor UI changes. This post explores how the industry is shifting toward AI-driven testing, featuring Google’s Artemis – an autonomous test runner that leverages multimodal large language models to interact with mobile apps and reason through test cases using natural language. By shifting from rigid, scripted steps to human-like visual comprehension and device interactions, AI testing tools are changing how quality assurance is done and transforming QA engineers into AI engineers.
Code is cheap, verification is not
Programmers have been using AI agents to help them write code for a couple of years now. The AI disruption is real and it’s here, changing how we create software every day. Now that producing code is no longer the problem, verifying the outcome is even more important. Programmers are struggling to keep up with code reviews for increasingly large changes.
Letting AI generate unit tests for the changes it makes is a good but only partial solution. We still have to verify that the unit tests cover all aspects of the change. And how can we be sure that unit tests cover all corner cases of a feature? This can be even more cognitively demanding because we have to focus on a larger logical-thinking space and account for complex inter-component interactions to ensure comprehensive test coverage.
We still need some kind of end-to-end validation of a feature or the application to verify that all the components and screens work together. We must validate that the actual user experience meets business requirements while maintaining expected quality and performance standards. The most obvious, and still valid, solution is to perform manual quality assurance. It will remain valid in the era of agentic coding because we must be able to catch all the corner cases and validate all the nuances of the user experience, which are things humans are still better at doing.
The fragility of mobile end-to-end testing
We have improved end-to-end testing by introducing testing frameworks that let us write automated end-to-end tests. Frameworks like Cucumber and later Calabash introduced Behaviour-Driven Development to the mobile context in the early 2010s. Later, those were replaced by platform-specific solutions like XCUITest for iOS and Espresso for Android. Recently we have seen frameworks like Appium and Maestro that can target both iOS and Android without the need to write platform-specific code.
The challenge with these testing frameworks is that they require specific steps and instructions to be able to execute the tests successfully. One unintentional popup dialog on the screen, one localization change, or an ID for a component that has changed will fail the test run. From the end user’s perspective the application would have still worked fine, but the problem is that these end-to-end tests require constant maintenance in order to stay relevant.
The AI paradigm shift in end-to-end testing
I think we are now witnessing a paradigm shift in testing driven by the AI revolution, similar to the one in programming. Large language models can understand the content of the screen and they can reason about the expected outcome of the test case. It is no longer necessary to write exact, brittle steps for test cases, just as it is not necessary to write every line of code ourselves.
What is Artemis?
Enter Artemis from Google. Artemis is an AI test runner that can interact with the phone like a human and reason about test cases using natural language. It is enough to tell Artemis what kind of steps it should take and what it should verify in the end, and Artemis will create a test plan, execute that test plan by interacting directly with the on-screen UI, and verify the final outcome via multimodal reasoning.
Artemis is still in its early phases, but the roadmap shows that Google has some bold goals for it: Artemis will be integrated directly into the Android Studio IDE and will target iOS in addition to Android, which it already works with today.
Using Artemis in your workflow
You can interact with Artemis in multiple ways. It supports MCP and ships with agent skills so that you can use Artemis right from your coding agent to, for example, verify during development that a feature or UI flow works. For a more holistic approach, the appropriate solution is to write test suites with the Python SDK or run test cases right from the CLI.
Artemis has extensive tracing support so that test cases can be verified afterward or, in case of a failure, the exact step with a screenshot is stored for later analysis.
Artemis supports Google’s own Gemini models out of the box, but it is fully multimodal. Providing an API key to xAI, Anthropic, or OpenAI also works more or less out of the box, but if you want, you can give a URL and a key to any OpenAI-compatible model provider and run Artemis with a variety of cheap open-weight models or even fully locally with LM Studio or Ollama.
Artemis is really impressive. In my—admittedly quite limited—testing, it has yet to fail a single test case I’ve given it. Interacting with it is trivial from the polished web UI, which allows for some impressive demos, or from the Python SDK to build more sophisticated test suites.
Artemis in action
Quality assurance engineers will become AI engineers in the same way that programmers have. With Artemis, Google is showing us the future of testing. An example illustrates this the best. Imagine writing the following test case in any of today’s end-to-end test frameworks: “Open Settings app, navigate to Display settings, ensure Dark theme is enabled, and check Wi-Fi connection status”. Traditional end-to-end test scripts break here easily when for example sub-menus differ between Android manufacturers. Artemis, however, navigates visually and semantically, finding ‘Display’ or ‘Network’ regardless of the layout position.
First, Artemis reasons about what is expected:
Then Artemis performs the necessary actions to complete the tasks by interacting directly with the native UI. Below are just two of the steps shown for reference, but you can see how Artemis reasons about the UI and how the before and after states are stored for tracing:
Finally, the desired outcome is reached and Artemis verifies it:
The future of quality assurance: from scripts to AI engineering
LLMs are non-deterministic by nature. Artemis reduces the risk of “hallucinated passes” with closed-loop visual observation, explicit verification checkpoints and full execution traces. The traces matter, because they let you check why a test passed or failed.
While manual testing remains invaluable, Artemis offers a seamless way to create far more robust end-to-end test suites that complement human verification.
End-to-end tests have been notoriously difficult to write and maintain. Rather than spending hours updating broken element IDs or fighting flaky selectors after every minor redesign, QA engineers can focus on defining meaningful user journeys, exploratory scenarios, and high-level quality criteria. In the age of agentic coding, verification is key, and tools like Artemis ensure that quality keeps pace with the speed of creation.
Artemis is open source and available on GitHub at https://github.com/google/artemis.
Kirjoittaja
Johan Paul on käytännönläheinen ohjelmistoarkkitehti, jolla on yli 20 vuoden kokemus ohjelmistotekniikasta. Johanin erikoisalaa on full-stack mobiilisovelluskehitys, ja hänellä on horjumaton intohimo rakentaa korkealaatuisia, kestäviä ja käyttäjäkeskeisiä tuotteita. Hän suosii koiria kissojen sijaan.
