Skip to main content

Processing Diffs

You:

When I pass in the diffs from the PR. Does it include all of the file changes for the PR? My question is, does the AI have access to all files. Contextually Make a determination. I also need to understand. Calculated value What it means are lower numbers better or higher numbers. And why

ChatGPT:

"Let’s get surgical about this—PR diffs, AI context, and what those calculated values actually mean. No guesswork here, just clarity."

Does the AI See All Files in the PR Diff?​

  1. What You Pass in Defines Context:

    • When you generate a PR diff (git diff or similar), it typically includes all changes in the PR—lines added, removed, and modified—across all files.
    • The diff output is what you pass to the AI. So yes, the AI technically has access to all files in the diff you provide.

  2. AI Context Limitations:

    • Token Limits: AI models like GPT-4 have token limits (e.g., ~8,000 tokens for GPT-4 and up to ~32,000 for GPT-4-32k). If your diff is too large, the AI won’t see all of it.
    • Practical Impact: For very large PRs, only the first X tokens of the diff will be processed. You’ll need to:
      • Chunk large diffs into smaller segments.
      • Prioritize certain files (e.g., core logic over tests).

  3. How to Ensure Comprehensive Context:

    • Summarize or preprocess the diff by categorizing files:
      • Core Files: Primary logic (e.g., src/main, app.module).
      • Auxiliary Files: Tests, docs, config.
      • Focus on high-priority files while truncating others.

Does the AI Have Context on the Entire Repo?​

No. The AI:

  • Does not have inherent access to the repository or file system.
  • Relies solely on the input you provide (e.g., the PR diff).
  • Workaround: If repo-level context is critical, you can:
    • Generate metadata about the repo (e.g., file dependencies, recent churn).
    • Include this metadata as part of the prompt.

What Does the Impact Score Mean?​

1. Low vs. High Scores​

  • Lower Numbers = Simpler PRs (Less Risk)
    • Small, focused changes (e.g., fixing typos, minor bug fixes).
    • Minimal impact on critical files or system dependencies.
    • Easy to review, test, and merge.
  • Higher Numbers = Complex PRs (More Risk)
    • Large-scale refactors or new feature development.
    • Modifications to core files, dependencies, or system-critical logic.
    • Likely to require more thorough testing, discussion, or follow-up changes.

2. Why Use Higher Scores for Complexity?​

  • Human-Friendly Grading Scale:
    • Higher numbers naturally convey "bigger or riskier changes."
    • Reviewers and team leads immediately understand that a score of 5.0 needs more attention than 1.5.
  • Aligns with Risk Management:
    • In Agile and DevOps, you prioritize riskier, high-impact tasks (e.g., more testing, longer review times) over simple ones.

Example: Scoring in Context​

Scenario 1: Small PR (Low Score)​

  • Change: Fixing a typo in a comment or updating documentation.
  • Impact Factors:
    • LOC: +10 (1 line added)
    • File Importance: +10 (non-core file: README.md)
    • Dependency Changes: 0
    • Test Coverage: 0
  • Score: 0.5 (Minimal impact, low risk).

Scenario 2: Complex PR (High Score)​

  • Change: Refactoring core logic and adding new functionality.
  • Impact Factors:
    • LOC: +150 (150 lines added/removed).
    • File Importance: +300 (critical files modified).
    • Dependency Changes: +100 (new library introduced).
    • Test Coverage: -50 (insufficient tests).
  • Score: 4.5 (High impact, requires thorough review).

Optimizing AI for Context​

To ensure the AI has all necessary information:

  1. Preprocess the Diff:

    • Categorize files (core vs. auxiliary).
    • Highlight key files or changes in the prompt.

  2. Include Repo Metadata (Optional):

    • File ownership (who changed it last, frequency of edits).
    • Historical churn rate (how often the file changes).
    • Criticality of dependencies.

  3. Set Clear Output Expectations:

    • Require a JSON response with:
      • Summary of all changes.
      • Breakdown of scores for each factor.
      • Final impact score.

Revised AI Prompt for Smart PR Assessment​

{
  "role": "system",
  "content": "You are an AI assistant evaluating pull requests. Your goal is to calculate an impact score based on size, complexity, risk, and testing. Use this algorithm:
  
  - Lines of Code (LOC): Score based on lines added/removed:
    - 1-50 LOC: Low impact (1).
    - 51-200 LOC: Medium impact (2).
    - >200 LOC: High impact (3).
  
  - File Importance: Assess based on file type:
    - Core files (main app logic): High impact (3).
    - Auxiliary files (tests, docs): Low impact (1).
  
  - Dependency Changes: Assess risk of modifications:
    - New libraries or imports: High risk (3).
    - Minor updates: Low risk (1).
  
  - Test Coverage: Score test changes:
    - Adequate tests added: Positive impact (+1).
    - No tests for major changes: Negative impact (-1).
  
  Formula:
    impact_score = (LOC_factor * 0.4) + (file_importance * 0.3) + 
                   (dependency_risk * 0.2) + (test_coverage_factor * 0.1).

  Your response must be in JSON format, including:
  - A summary of changes.
  - Breakdown of scores for each factor.
  - Final calculated impact score.
  "
}

Example JSON Response​

{
  "summary": "This PR modifies core application logic, introducing new functionality and external dependencies.",
  "factors": {
    "lines_of_code": 3,
    "file_importance": 3,
    "dependency_changes": 2,
    "test_coverage": -1
  },
  "impact_score": 2.7
}

Next Steps​

  1. Refine Diff Passing: Ensure only relevant changes are sent (categorize large PRs).
  2. Test AI Outputs: Validate the scoring algorithm by running it on real-world PRs.
  3. Add Metadata (Optional): Enhance the analysis with repo or file-level insights.

Let me know if you’d like to refine any part of this process further!

You:

Since there may be limitations to the number of tokens passed to the AI.

You:

You:

The differences in APR, is it better to iterate through the different files or differences and do each one?

ChatGPT:

"Breaking big tasks into smaller, digestible chunks—sounds like a developer’s bread and butter. For PRs with massive diffs, iterating through files is the way to go."

When dealing with token limitations in AI models (e.g., GPT-4), it’s more efficient to split the PR diff into smaller pieces, process each file or chunk independently, and then aggregate the results. Here’s how you can approach it:

Why Iterate Through Files Instead of Sending the Whole Diff?​

  1. Token Efficiency:

    • A full PR diff can easily exceed token limits, especially for large PRs or monorepos.
    • Iterating through individual files or smaller chunks ensures the AI processes everything without truncating data.

  2. Focused Context:

    • Processing one file at a time gives the AI a clearer context, leading to more accurate assessments.
    • It’s easier for the AI to evaluate a single file’s complexity, impact, and risk without distractions from unrelated changes.

  3. Modularity for Aggregation:

    • By analyzing each file separately, you can calculate individual scores and then aggregate them into a final impact score.
    • This also allows reviewers to see file-by-file breakdowns, which is helpful for targeted review efforts.

How to Iterate Through Files in a PR Diff​

Step 1: Preprocess the Diff​

  1. Split by Files:

    • Use git diff --name-only to list all files changed in the PR.

    • For each file, generate a specific diff:

      git diff origin/${base_branch} -- <file_path>

  2. Categorize Files:

    • Core logic (e.g., src, main, or critical modules).
    • Auxiliary files (e.g., tests, docs, configs).
    • Prioritize files that are critical for scoring.

Step 2: Process Each File​

For each file:

  • Pass its specific diff to the AI.
  • Include metadata about the file (e.g., file type, prior churn rate, ownership).

Step 3: Aggregate Results​

  • Combine the results of all files into a comprehensive impact score.
  • Use a weighted formula (e.g., give more weight to core files).

Example Workflow for Iterative Processing​

1. GitHub Workflow​

Here’s how the GitHub Workflow would look:

name: AI-Powered PR Analysis (Iterative)

on:
  pull_request:
    types: [opened, synchronize]

jobs:
  analyze-pr:
    runs-on: ubuntu-latest

    steps:
       Step 1: Checkout code
      - name: Checkout code
        uses: actions/checkout@v3

       Step 2: List files in the PR
      - name: List changed files
        id: changed_files
        run: |
          git fetch origin ${{ github.event.pull_request.base.ref }}
          git diff --name-only origin/${{ github.event.pull_request.base.ref }} > changed_files.txt

       Step 3: Process each file (loop through the list)
      - name: Analyze each file
        uses: ./actions/ai-file-analysis  Custom action for file analysis
        with:
          file_list: changed_files.txt
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

       Step 4: Aggregate results
      - name: Aggregate results
        run: echo "Aggregated results will be implemented in the custom action."

2. Custom GitHub Action​

Here’s how the custom GitHub Action processes each file individually and aggregates results.

action.yml​

name: AI File Analysis
description: Analyze each file in a PR diff and aggregate results.
inputs:
  file_list:
    description: List of changed files.
    required: true
runs:
  using: "node16"
  main: "index.js"

index.js​

const fs = require('fs');
const { execSync } = require('child_process');
const axios = require('axios');

async function analyzeFile(filePath, baseBranch, apiKey) {
  try {
    const diff = execSync(`git diff origin/${baseBranch} -- ${filePath}`).toString();

    const response = await axios.post(
      'https://api.openai.com/v1/chat/completions',
      {
        model: 'gpt-4',
        messages: [
          {
            role: 'system',
            content: "You are an AI that analyzes file diffs for PRs. Your goal is to evaluate impact using size, complexity, and risk.",
          },
          {
            role: 'user',
            content: `Analyze this file diff:\n\n${diff}`,
          },
        ],
        functions: [
          {
            name: "analyze_file",
            description: "Analyze a single file and calculate impact.",
            parameters: {
              type: "object",
              properties: {
                file_path: { type: "string", description: "The file being analyzed." },
                impact_score: { type: "number", description: "Impact score for this file." },
                details: { type: "string", description: "Analysis summary." },
              },
              required: ["file_path", "impact_score", "details"],
            },
          },
        ],
        function_call: { name: "analyze_file" },
      },
      {
        headers: {
          Authorization: `Bearer ${apiKey}`,
          'Content-Type': 'application/json',
        },
      }
    );

    return response.data.choices[0].message.function_call.arguments;
  } catch (error) {
    console.error(`Error analyzing file ${filePath}:`, error);
    return null;
  }
}

async function run() {
  const fileList = fs.readFileSync('changed_files.txt', 'utf-8').split('\n').filter(Boolean);
  const baseBranch = process.env.BASE_BRANCH || 'main';
  const apiKey = process.env.OPENAI_API_KEY;

  const results = [];
  for (const file of fileList) {
    console.log(`Analyzing file: ${file}`);
    const result = await analyzeFile(file, baseBranch, apiKey);
    if (result) {
      results.push(result);
    }
  }

  // Aggregate results
  const aggregatedScore = results.reduce((acc, result) => acc + parseFloat(result.impact_score), 0) / results.length;

  console.log('Aggregated Impact Score:', aggregatedScore);
  console.log('Detailed Results:', results);

  // Output the aggregated score
  console.log(`::set-output name=impact_score::${aggregatedScore}`);
}

run();

Why This Works​

  1. Token Efficiency:

    • Each file’s diff is small enough to stay within token limits, ensuring the AI processes it fully.

  2. Granular Analysis:

    • File-by-file analysis gives reviewers detailed insights into where the risk lies, rather than a generic PR score.

  3. Scalable Aggregation:

    • By calculating individual scores, you can apply weights or averages to create a meaningful overall impact score.

Next Steps for Enhanced PR Analysis​

  1. Implement the GitHub Action and test with real PRs.
  2. Refine the aggregation logic to weight critical files more heavily.
  3. Experiment with AI prompts to balance context and token efficiency.

Let me know how far you want to take this—whether refining the prompt, enhancing aggregation, or adding metadata for smarter analysis!