Thiago Julio

Software development

PT EN

Code review in a small team: one standard, a bot for the first pass, a human for the merge

code-review ai gitlab leadership

TL;DR - Our reviews were either a rubber stamp or a week of comments about naming. I wrote a one-page standard, put PR-Agent on our self-hosted GitLab to do the first pass, and kept the merge decision with a person. The bot catches the obvious things, so the human review can focus on business rules and risk. It costs almost nothing.

When I started leading the team, code review depended on who reviewed. Some merge requests were approved in two minutes. Others sat for days with comments about style. Neither caught the bugs that reached production. On top of that, more and more code was written with AI, which looks clean and fails at the edges.

The standard: one page

For the author

  • Keep the MR small. A big diff gets a shallow review.
  • Fill in the template: what changed, why, how to test it, and what could go wrong. The template lives in the repo (.gitlab/merge_request_templates/), so GitLab fills it in automatically.
  • Link the ticket.

For the reviewer

  • Style is the linter’s job. If the linter didn’t complain, I don’t either.
  • Behavior: does it do what the ticket asked, and what happens with bad input?
  • Tests: is there at least one test for the unhappy path, with an assertion about a business rule?
  • Risk: migrations, new dependencies, changes to shared code.

Every comment starts with its kind: blocker:, suggestion:, question: or nit:. Only blockers stop the merge. That alone ended a lot of arguments.

AI-written code gets the same review, with three extra checks

AI code usually looks good. Formatting and names are fine. What fails is elsewhere:

  • Edges. It covers the happy path and forgets null, empty lists and out-of-range values.
  • Tests that mirror the code. Coverage goes up and nothing is guaranteed.
  • Dependencies nobody asked for. Every new library is something the team will have to update and debug.

PR-Agent as the first reviewer

Half of the human comments were things a careful first pass would catch. I set up PR-Agent on our self-hosted GitLab, using an OpenAI model. It runs when an MR is opened or updated and posts back on the MR.

sequenceDiagram
  participant Dev as Developer
  participant GL as GitLab
  participant PA as PR-Agent
  participant Rev as Reviewer
  Dev->>GL: Open merge request
  GL->>PA: MR opened or updated
  PA->>GL: Description and first-pass review
  Dev->>GL: Fix or reply
  Rev->>GL: Review behavior, tests, risk
  Rev->>GL: Approve and merge
  • /describe drafts the MR description from the diff. The author edits it, but starts from something.
  • /review lists possible bugs, security concerns and missing tests.
  • /improve suggests code changes. It’s the noisiest, so it runs only on request.

The bill was a surprise in the good way: with the size of our MRs, the model cost is close to zero.

Rules I kept

  • The bot never approves. Approval is a person’s job and a person’s responsibility.
  • No secrets or customer data in the diff, bot or no bot.
  • If the bot is wrong, the author answers “not applicable” and moves on. Nobody argues with it.

What changed

MRs got smaller, reviews came back faster, and the human comments moved from naming to edge cases and business rules. The bot didn’t replace the reviewer. It moved the reviewer’s attention to the part only a person can judge.