AI Prompt Optimization
AI Prompt Optimization
You:
I'm wondering if we can optimize the prompt for the Api, call? For example, I would like to have a better Algorithm or formula to determine the Value Of the Smart PR assessment. Period Let's focus on the prompt. To include the context. The algorithm or formula to calculate. Let's also include The steps If necessary And also an example of what we want back in terms of a response. I would like to have the response returned in JSON format using open Ai's new Jason Formatter
ChatGPT:
"Alright, let’s flex some brainpower and get that OpenAI prompt optimized for a Smart PR Assessment. We'll include everything—context, algorithm, steps, and an expected JSON response format using OpenAI’s JSON formatter."
Optimized Prompt Design
Here’s the structure for the OpenAI prompt:
Prompt Context
Provide context about the task and what the API needs to evaluate.
You are an intelligent assistant tasked with analyzing pull request (PR) changes for a software repository. Your role is to assess the PR based on size, complexity, risk, and impact using a pre-defined algorithm. The PR changes are provided as a diff, and your output should summarize the analysis and provide an impact score.
Algorithm for Smart PR Assessment
Here’s a sample algorithm/formula to evaluate the PR:
- Lines of Code (LOC):
- Score based on the number of lines added or removed.
- Large changes increase complexity and risk.
- File Importance:
- Core files (e.g., config, main modules) have higher impact than auxiliary files (e.g., tests, docs).
- Dependency Changes:
- Modifications to imports or external libraries increase risk.
- Test Coverage:
- Check if tests were added/modified. Missing tests for significant changes reduce the score.
- Churn Rate:
- Higher churn (frequent edits to the same files) indicates potential instability.
Steps for Analysis
-
Analyze the diff for:
- Total lines added/removed.
- File types affected.
- Dependency changes (e.g., new imports).
- Test files and changes.
-
Apply the formula to calculate an overall impact score:
impact_score = (LOC_factor * 0.4) + (file_importance * 0.3) +
(dependency_risk * 0.2) + (test_coverage_factor * 0.1) -
Return a JSON-formatted response that includes:
- Summary of findings.
- Breakdown of the factors.
- Final impact score.
Prompt Example
{
"role": "system",
"content": "You are an intelligent assistant that analyzes pull requests (PRs) for software repositories. Your goal is to calculate an impact score and provide a detailed assessment in JSON format. Use the following algorithm to assess the PR:
- **Lines of Code (LOC):** Assign a factor based on the total lines added/removed:
- 1-50 LOC: Low impact (score: 1).
- 51-200 LOC: Medium impact (score: 2).
- >200 LOC: High impact (score: 3).
- **File Importance:** Determine the importance of the files affected:
- Core files: High impact (score: 3).
- Auxiliary files (e.g., tests, docs): Low impact (score: 1).
- **Dependency Changes:** Assess modifications to dependencies:
- Added external libraries: High risk (score: 3).
- Minor dependency updates: Low risk (score: 1).
- **Test Coverage:** Evaluate changes to test files:
- Adequate tests added/updated: Positive impact (+1).
- No tests for major changes: Negative impact (-1).
- **Formula:** Calculate the impact score using:
impact_score = (LOC_factor * 0.4) + (file_importance * 0.3) + (dependency_risk * 0.2) + (test_coverage_factor * 0.1).
**Steps:**
1. Analyze the PR diff provided.
2. Extract lines of code, affected files, dependencies, and test coverage.
3. Apply the formula to calculate the overall impact score.
**Example Response:**
Return your results as a JSON object in this format:
{
\"summary\": \"Summary of the PR analysis\",
\"factors\": {
\"lines_of_code\": 3,
\"file_importance\": 2,
\"dependency_changes\": 1,
\"test_coverage\": -1
},
\"impact_score\": 2.3
}"
}
Passing the Diff
In the messages array of the OpenAI API call, add the actual PR diff as input:
{
"role": "user",
"content": "Here is the PR diff to analyze:\n\n<PR_DIFF_CONTENT>"
}
OpenAI API Request (JSON Formatter)
Here’s how you’d configure the API request to use OpenAI’s JSON response formatting:
API Call Example
const axios = require('axios');
async function analyzePR(diff, apiKey) {
const response = await axios.post(
'https://api.openai.com/v1/chat/completions',
{
model: 'gpt-4',
messages: [
{
role: 'system',
content: "You are an intelligent assistant that analyzes pull requests (PRs)...", // Use the prompt example above
},
{
role: 'user',
content: `Here is the PR diff to analyze:\n\n${diff}`,
},
],
functions: [
{
name: "analyze_pr",
description: "Analyze PR changes and calculate an impact score",
parameters: {
type: "object",
properties: {
summary: { type: "string", description: "Summary of the PR analysis" },
factors: {
type: "object",
properties: {
lines_of_code: { type: "integer", description: "Score for lines of code changes" },
file_importance: { type: "integer", description: "Score for file importance" },
dependency_changes: { type: "integer", description: "Score for dependency changes" },
test_coverage: { type: "integer", description: "Score for test coverage" },
},
required: ["lines_of_code", "file_importance", "dependency_changes", "test_coverage"],
},
impact_score: {
type: "number",
description: "Overall impact score calculated using the algorithm",
},
},
required: ["summary", "factors", "impact_score"],
},
},
],
function_call: { name: "analyze_pr" },
},
{
headers: {
Authorization: `Bearer ${apiKey}`,
'Content-Type': 'application/json',
},
}
);
return response.data.choices[0].message.function_call.arguments;
}
Next Steps for Prompt Optimization
- Integrate the Optimized Prompt: Use the example prompt in your Nx executor or GitHub Action.
- Test the API Response: Ensure the JSON formatting works as expected and that the impact score aligns with your algorithm.
- Refine the Algorithm: Adjust weights or add factors (e.g., cyclomatic complexity) as you test more PRs.
Let me know if you’d like to iterate further on the algorithm or integration!