Ask for evidence from your own stack

Give candidates the same anonymized workflow, such as checkout with an expired payment token. Ask them to identify assertions, test data, environment needs, and excluded dependencies. Their explanation should show how the proposed tests detect a wrong business outcome.

Request a sample change that runs through your normal review process. Inspect naming, setup, cleanup, and failure messages. A polished demonstration alone cannot show whether your engineers can maintain the suite when screens, endpoints, and product rules change later.

Compare the full operating cost

Separate implementation work from license fees, hosted execution, device access, and maintenance. Ask who fixes a failing test after a product change. Include the time your team spends supplying accounts, preparing environments, answering domain questions, and reviewing proposed assertions.

Check whether the deliverable can run outside the provider's account. Require a dependency list and clear instructions for local and continuous integration execution. Confirm ownership and export terms in the agreement rather than inferring them from a sales presentation.

Use a bounded pilot to make the decision

Choose one critical journey and one difficult edge case for the pilot. Define completion as reviewed tests, reproducible setup, useful failure artifacts, and a documented gap list. Keep the scope small enough that competing proposals remain directly comparable to each other.

Evaluate the pilot by the defects it could detect and the effort needed to diagnose failures. Ask a team member who did not write the tests to run them. Record their questions as handover gaps before expanding the provider's scope.

Working sheet

Provider evaluation evidence
CriterionAsk forWarning sign
OwnershipRepository changes and export termsAccount-only deliverables
Failure diagnosisTrace or clear assertion outputScreenshots without context
MaintenanceNamed owner and change processUndefined ongoing work
ScopeExplicit included risks and exclusionsCoverage percentage alone
Download this worksheet (CSV)

Questions about how to compare test automation companies

Should we choose the lowest quote?

Compare equivalent scope and operating costs first. A lower quote may omit data setup, handover, or maintenance.

Does a large test count prove quality?

No. Inspect assertions and failure behavior. Many shallow checks can miss a critical business rule.

How should I use the working sheet?

Download the CSV and replace the sample rows with evidence from your repository. Record a clear acceptance check and an owner for each item. Keep unresolved assumptions visible rather than treating the example as a completed assessment.

Sources and further reading

Examples are illustrative. They are not customer results, security guarantees, or a substitute for checking the documentation for your installed version.

Make the next change verifiable.

Getzilla is an open source workflow for changes to existing repositories. Inspect the project, define acceptance checks, and keep independent review in the loop.

See the repository workflow See current access and pricing