AI-Driven PR Review

You:

Yes, I would like to include the AI to actually perform a PR review for each diff and also automate with the git commands to persist the comment for every diff as well as the overview comment for the entire PR. Is this possible?

ChatGPT:

"Absolutely, it's possible—and it's the kind of automation that makes you look like a wizard in the code review game. Let’s have the AI review each diff, post comments directly on the files in the PR, and include an overall review summary. All fully automated."

How to Automate AI-Powered PR Reviews

Here’s how we’ll structure it:

Iterate Through Each File Diff:
- Analyze the file diff using the AI.
- Generate a review comment for the diff.
- Post the comment directly to the specific line in the PR using GitHub’s API.
Generate an Overview Review:
- Summarize all file-level analyses.
- Post an overall comment on the PR summarizing risks, impacts, and recommendations.
Automate with Git Commands:
- Use git diff to extract changes for each file.
- Persist comments to GitHub with octokit (GitHub’s REST API library for Node.js).

Implementation Steps

Step 1: GitHub Workflow

This workflow triggers on a pull request and delegates tasks to a custom GitHub Action.

name: AI-Powered PR Review

on:
  pull_request:
    types: [opened, synchronize]

jobs:
  review-pr:
    runs-on: ubuntu-latest

    steps:
      # Step 1: Checkout the repository
      - name: Checkout code
        uses: actions/checkout@v3

      # Step 2: Install dependencies
      - name: Install dependencies
        run: npm install

      # Step 3: AI-Powered PR Review
      - name: AI PR Review
        uses: ./actions/ai-pr-review # Custom GitHub Action
        with:
          pr_number: ${{ github.event.pull_request.number }}
          repo_owner: ${{ github.repository_owner }}
          repo_name: ${{ github.event.repository.name }}
          base_branch: ${{ github.event.pull_request.base.ref }}
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Step 2: Custom GitHub Action

`action.yml`

Defines inputs and execution for the action.

name: AI-Powered PR Review
description: Analyze pull requests with AI and post comments on diffs.
inputs:
  pr_number:
    description: The PR number to review.
    required: true
  repo_owner:
    description: The repository owner.
    required: true
  repo_name:
    description: The repository name.
    required: true
  base_branch:
    description: The base branch of the PR.
    required: true
runs:
  using: "node16"
  main: "index.js"

`index.js`

Implements the logic to:

Extract file-level diffs.
Analyze diffs with AI.
Post file-specific comments and an overall PR review.

const { execSync } = require('child_process');
const { Octokit } = require('@octokit/rest');
const axios = require('axios');
const core = require('@actions/core');

async function analyzeDiff(diff, fileName, apiKey) {
  const response = await axios.post(
    'https://api.openai.com/v1/chat/completions',
    {
      model: 'gpt-4',
      messages: [
        {
          role: 'system',
          content: "You are an AI assistant reviewing code changes in a pull request. Provide a detailed review comment based on the provided diff.",
        },
        {
          role: 'user',
          content: `Review this diff for the file ${fileName}:\n\n${diff}`,
        },
      ],
      functions: [
        {
          name: "file_review",
          description: "Analyze a file diff and provide a review comment.",
          parameters: {
            type: "object",
            properties: {
              file_name: { type: "string", description: "The file being reviewed." },
              review_comment: { type: "string", description: "The AI's review comment." },
            },
            required: ["file_name", "review_comment"],
          },
        },
      ],
      function_call: { name: "file_review" },
    },
    {
      headers: {
        Authorization: `Bearer ${apiKey}`,
        'Content-Type': 'application/json',
      },
    }
  );

  return response.data.choices[0].message.function_call.arguments;
}

async function postComment(octokit, repoOwner, repoName, prNumber, fileName, lineNumber, comment) {
  await octokit.pulls.createReviewComment({
    owner: repoOwner,
    repo: repoName,
    pull_number: prNumber,
    path: fileName,
    line: lineNumber,
    body: comment,
  });
}

async function postOverallReview(octokit, repoOwner, repoName, prNumber, overallComment) {
  await octokit.pulls.createReview({
    owner: repoOwner,
    repo: repoName,
    pull_number: prNumber,
    body: overallComment,
    event: 'COMMENT',
  });
}

async function run() {
  const prNumber = core.getInput('pr_number');
  const repoOwner = core.getInput('repo_owner');
  const repoName = core.getInput('repo_name');
  const baseBranch = core.getInput('base_branch');
  const apiKey = process.env.OPENAI_API_KEY;
  const githubToken = process.env.GITHUB_TOKEN;

  const octokit = new Octokit({ auth: githubToken });

  // Step 1: Get list of changed files
  const changedFiles = execSync(`git diff --name-only origin/${baseBranch}`).toString().split('\n').filter(Boolean);

  const overallComments = [];
  for (const fileName of changedFiles) {
    // Step 2: Generate diff for each file
    const diff = execSync(`git diff origin/${baseBranch} -- ${fileName}`).toString();

    // Step 3: Analyze diff with AI
    const { review_comment } = await analyzeDiff(diff, fileName, apiKey);

    // Step 4: Post file-specific comments
    overallComments.push(`### ${fileName}\n\n${review_comment}`);
    await postComment(octokit, repoOwner, repoName, prNumber, fileName, 1, review_comment); // Default to line 1 for simplicity
  }

  // Step 5: Post overall PR review
  const overallComment = overallComments.join('\n\n');
  await postOverallReview(octokit, repoOwner, repoName, prNumber, overallComment);

  console.log('AI-powered PR review completed.');
}

run();

How It Works

File-Level Review:
- Extracts individual file diffs from the PR.
- Analyzes each diff with AI.
- Posts comments directly to the relevant files using GitHub’s API.
Overall PR Review:
- Aggregates all file-level comments.
- Posts a summary review comment on the PR itself.

Example Output

File-Specific Comment (on `file.ts`):

Review for file.ts

The changes improve functionality but introduce a new dependency. Please ensure tests cover edge cases for the new logic.

Overall PR Comment:

AI-Powered Review Summary

file.ts: The changes improve functionality but introduce a new dependency. Ensure edge cases are tested.

module.spec.ts: Test coverage is adequate, but consider additional tests for invalid inputs.

Why This Works

Full Coverage: Every file gets individual attention, and the PR gets a comprehensive summary.
Seamless Workflow: GitHub comments are posted directly to the PR, making it easy for reviewers to act.
Scalable: Can handle small or large PRs by iterating through diffs.

Next Steps

Test the GitHub Action with a real PR.
Refine the AI prompt for more targeted feedback.
Enhance file-comment linking for multi-line diffs.

Let me know if you’d like further enhancements or real-world testing!

You:

Yes, let's go to the next level with your suggestions.

ChatGPT:

"Alright, let’s crank it up a notch! We're taking this to the next level with advanced enhancements, smarter AI prompts, and cleaner GitHub integrations. Let’s make this the ultimate PR review automation system."

Next-Level Enhancements

Here’s what we’ll do to level up:

Smarter AI Prompts:
- Refine the prompts to include code context, repository metadata, and review objectives.
- Customize feedback based on file types (e.g., core logic vs. tests vs. docs).
Multi-Line Comments:
- Add support for posting comments across multiple lines of a diff.
- Automatically detect the range of changes in a file.
Customizable Scoring and Actions:
- Add a scoring system (e.g., overall PR complexity or risk score).
- Trigger specific actions based on the score (e.g., require more reviewers for high-risk PRs).
Enhanced Error Handling and Logging:
- Ensure robust handling of API failures or large PRs.
- Add logs for easy debugging.
Extensive Test Coverage:
- Simulate large PRs to validate token efficiency and scaling.

Step 1: Smarter AI Prompts

Enhanced Prompt

We’ll include:

File-specific context (e.g., test files vs. core logic).
Objectives for the AI (e.g., identify risks, evaluate test coverage, recommend changes).

{
  "role": "system",
  "content": "You are an AI assistant reviewing code changes in a pull request. Your role is to evaluate the quality, risks, and adequacy of test coverage for each file. Use the following guidelines:

  1. Identify potential risks in the code changes (e.g., security, performance, maintainability).
  2. Evaluate whether the test files adequately cover the changes.
  3. Recommend improvements if necessary.

  Respond with JSON in this format:
  {
    \"file_name\": \"The file being reviewed\",
    \"review_comment\": \"Detailed feedback for the file\",
    \"risk_score\": \"Numeric score for the level of risk (1-10)\",
    \"recommendations\": [\"List of actionable recommendations\"]
  }"
}

Example AI Output

{
  "file_name": "src/app/module/file.ts",
  "review_comment": "The changes introduce a new function that modifies global state. Consider encapsulating the logic to reduce side effects.",
  "risk_score": 7,
  "recommendations": [
    "Refactor the new function to avoid modifying global state directly.",
    "Add unit tests for edge cases involving null inputs."
  ]
}

Step 2: Multi-Line Comments

To support multi-line diffs:

Parse the diff to detect the range of changed lines.
Use GitHub’s multi-line comment API to add comments across those lines.

API Example for Multi-Line Comments

await octokit.pulls.createReviewComment({
  owner: repoOwner,
  repo: repoName,
  pull_number: prNumber,
  path: fileName,
  side: "RIGHT",
  start_line: 10, // Start of the change
  line: 15, // End of the change
  body: "The changes in this block introduce high cyclomatic complexity. Consider refactoring."
});

Step 3: Scoring and Actions

Add a scoring system for the overall PR based on AI outputs:

Formula Example:
- PR Complexity Score = Avg(risk_score) + LOC_factor
High-risk PRs trigger:
- Additional reviewer requirements.
- Automated notifications to team leads.

Updated Overall Review Example

{
  "summary": "This PR contains changes to critical files with moderate risks. Test coverage is adequate but requires improvements for edge cases.",
  "complexity_score": 8.5,
  "actions": [
    "Assign an additional reviewer.",
    "Run regression tests for impacted modules."
  ]
}

Step 4: Enhanced Error Handling

Add robust error handling for:

Large Diffs:
- Chunk diffs and process them iteratively to stay within token limits.
API Failures:
- Retry logic with exponential backoff for failed API requests.
Invalid Responses:
- Validate AI responses before posting comments.

Updated Implementation

Here’s the enhanced index.js for the custom GitHub Action:

const { execSync } = require('child_process');
const { Octokit } = require('@octokit/rest');
const axios = require('axios');
const core = require('@actions/core');

async function analyzeDiff(diff, fileName, apiKey) {
  try {
    const response = await axios.post(
      'https://api.openai.com/v1/chat/completions',
      {
        model: 'gpt-4',
        messages: [
          {
            role: 'system',
            content: "You are an AI assistant reviewing code changes in a pull request. Your role is to evaluate quality, risks, and adequacy of test coverage."
          },
          {
            role: 'user',
            content: `Review this diff for the file ${fileName}:\n\n${diff}`
          }
        ]
      },
      {
        headers: {
          Authorization: `Bearer ${apiKey}`,
          'Content-Type': 'application/json'
        }
      }
    );

    return JSON.parse(response.data.choices[0].message.content);
  } catch (error) {
    console.error(`Error analyzing diff for ${fileName}:`, error);
    return null;
  }
}

async function postComment(octokit, repoOwner, repoName, prNumber, fileName, startLine, endLine, comment) {
  await octokit.pulls.createReviewComment({
    owner: repoOwner,
    repo: repoName,
    pull_number: prNumber,
    path: fileName,
    side: "RIGHT",
    start_line: startLine,
    line: endLine,
    body: comment
  });
}

async function run() {
  const prNumber = core.getInput('pr_number');
  const repoOwner = core.getInput('repo_owner');
  const repoName = core.getInput('repo_name');
  const baseBranch = core.getInput('base_branch');
  const apiKey = process.env.OPENAI_API_KEY;
  const githubToken = process.env.GITHUB_TOKEN;

  const octokit = new Octokit({ auth: githubToken });

  // Get list of changed files
  const changedFiles = execSync(`git diff --name-only origin/${baseBranch}`).toString().split('\n').filter(Boolean);

  const overallComments = [];
  let totalRiskScore = 0;

  for (const fileName of changedFiles) {
    const diff = execSync(`git diff origin/${baseBranch} -- ${fileName}`).toString();
    const analysis = await analyzeDiff(diff, fileName, apiKey);

    if (analysis) {
      const { review_comment, risk_score, recommendations } = analysis;

      // Post multi-line comment (assuming single block for simplicity)
      await postComment(octokit, repoOwner, repoName, prNumber, fileName, 1, 10, review_comment);

      // Add to overall review
      overallComments.push(`### ${fileName}\n\n${review_comment}\n\n**Recommendations:**\n${recommendations.join('\n')}`);
      totalRiskScore += risk_score;
    }
  }

  // Post overall PR review
  const overallComment = `${overallComments.join('\n\n')}\n\n**Overall Complexity Score:** ${totalRiskScore / changedFiles.length}`;
  await octokit.pulls.createReview({
    owner: repoOwner,
    repo: repoName,
    pull_number: prNumber,
    body: overallComment,
    event: 'COMMENT'
  });

  console.log('AI-powered PR review completed.');
}

run();

Next Steps

Test with Real PRs:
- Ensure multi-line comments and scoring work as expected.
Fine-Tune Prompts:
- Adjust for different file types (e.g., core logic, tests, configs).
Add Regression Tests:
- Validate outputs for edge cases, large PRs, and token limitations.

Let me know where you’d like to dig deeper or refine further!

How to Automate AI-Powered PR Reviews​

Implementation Steps​

Step 1: GitHub Workflow​

Step 2: Custom GitHub Action​

action.yml​

index.js​

How It Works​

Example Output​

File-Specific Comment (on file.ts):​

Review for file.ts​

Overall PR Comment:​

AI-Powered Review Summary​

Why This Works​

Next Steps​

Next-Level Enhancements​

Step 1: Smarter AI Prompts​

Enhanced Prompt​

Example AI Output​

Step 2: Multi-Line Comments​

API Example for Multi-Line Comments​

Step 3: Scoring and Actions​

Updated Overall Review Example​

Step 4: Enhanced Error Handling​

Updated Implementation​

Next Steps​

How to Automate AI-Powered PR Reviews

Implementation Steps

Step 1: GitHub Workflow

Step 2: Custom GitHub Action

`action.yml`

`index.js`

How It Works

Example Output

File-Specific Comment (on `file.ts`):

Review for `file.ts`

Overall PR Comment:

AI-Powered Review Summary

Why This Works

Next Steps

Next-Level Enhancements

Step 1: Smarter AI Prompts

Enhanced Prompt

Example AI Output

Step 2: Multi-Line Comments

API Example for Multi-Line Comments

Step 3: Scoring and Actions

Updated Overall Review Example

Step 4: Enhanced Error Handling

Updated Implementation

Next Steps