Choose a narrow review job
Specify a small set of review targets, such as missing error handling around changed network calls or inconsistencies with a documented API contract. Keep deterministic formatting, type checking, and dependency policies in their existing tools. An assistant is easier to evaluate when its expected contribution is explicit.
Select representative historical changes with known defects and acceptable changes without those defects. Use the same evaluation set when adjusting prompts or repository instructions. Record actionable findings, missed known issues, and unnecessary comments rather than treating total comment count as a measure of value.
Design the CI integration boundary
Limit repository and token permissions to what the review job needs. Decide how it handles external contributions, sensitive paths, large diffs, and generated files. Review the provider's documented data handling and execution model before sending private source code or issue context outside your environment.
Illustrative scenario: a repository receives public pull requests. Run untrusted application code separately from any review job holding privileged credentials. Validate event triggers and repository permissions in the actual CI platform; a convenient default integration should not silently gain write access beyond its task.
Tune feedback and monitor the result
Group related findings, link them to changed lines, and request a concrete explanation of the triggering condition. Give developers a way to dismiss incorrect findings with a reason. Keep the reviewer responsible for checking suggestions before applying patches, especially changes to authorization and persistence logic.
Review pilot results after a defined sample of changes. Compare useful findings, reviewer time spent validating noise, and delays added to the pull-request workflow. Adjust scope or stop the integration if it generates more investigation work than it saves. Reevaluate after meaningful model or configuration changes.
Working sheet
| Decision | Initial setting to consider | Evidence before expansion |
|---|---|---|
| Merge blocking | Advisory findings | Stable precision on important issues |
| Repository scope | Selected paths | Useful results across representative changes |
| Permissions | Minimum required access | Documented need for each permission |
| Output volume | Deduplicated focused comments | Low dismissal and repeat-comment burden |
Questions about how to add AI code review automation to CI
Should AI approval count as human approval?
Keep ownership explicit. Automated feedback should not impersonate an independent domain reviewer or satisfy a human approval policy by accident.
What should happen when the review service fails?
Define the behavior in advance. An advisory pilot can report an unavailable check while existing required checks continue to govern merging.
How should I use the working sheet?
Download the CSV and replace the sample rows with evidence from your repository. Record a clear acceptance check and an owner for each item. Keep unresolved assumptions visible rather than treating the example as a completed assessment.
Sources and further reading
Examples are illustrative. They are not customer results, security guarantees, or a substitute for checking the documentation for your installed version.
