Ask for evidence from your own stack
Give candidates the same anonymized workflow, such as checkout with an expired payment token. Ask them to identify assertions, test data, environment needs, and excluded dependencies. Their explanation should show how the proposed tests detect a wrong business outcome.
Request a sample change that runs through your normal review process. Inspect naming, setup, cleanup, and failure messages. A polished demonstration alone cannot show whether your engineers can maintain the suite when screens, endpoints, and product rules change later.
Compare the full operating cost
Separate implementation work from license fees, hosted execution, device access, and maintenance. Ask who fixes a failing test after a product change. Include the time your team spends supplying accounts, preparing environments, answering domain questions, and reviewing proposed assertions.
Check whether the deliverable can run outside the provider's account. Require a dependency list and clear instructions for local and continuous integration execution. Confirm ownership and export terms in the agreement rather than inferring them from a sales presentation.
Use a bounded pilot to make the decision
Choose one critical journey and one difficult edge case for the pilot. Define completion as reviewed tests, reproducible setup, useful failure artifacts, and a documented gap list. Keep the scope small enough that competing proposals remain directly comparable to each other.
Evaluate the pilot by the defects it could detect and the effort needed to diagnose failures. Ask a team member who did not write the tests to run them. Record their questions as handover gaps before expanding the provider's scope.
Working sheet
| Criterion | Ask for | Warning sign |
|---|---|---|
| Ownership | Repository changes and export terms | Account-only deliverables |
| Failure diagnosis | Trace or clear assertion output | Screenshots without context |
| Maintenance | Named owner and change process | Undefined ongoing work |
| Scope | Explicit included risks and exclusions | Coverage percentage alone |
Questions about how to compare test automation companies
Should we choose the lowest quote?
Compare equivalent scope and operating costs first. A lower quote may omit data setup, handover, or maintenance.
Does a large test count prove quality?
No. Inspect assertions and failure behavior. Many shallow checks can miss a critical business rule.
How should I use the working sheet?
Download the CSV and replace the sample rows with evidence from your repository. Record a clear acceptance check and an owner for each item. Keep unresolved assumptions visible rather than treating the example as a completed assessment.
Sources and further reading
Examples are illustrative. They are not customer results, security guarantees, or a substitute for checking the documentation for your installed version.
