Storing Custom Templates for GitHub PR Review Bot

You:

Yes, let's continue on the aspect of storing custom templates. I want to store also prompts that include placeholders for inputs and possibly use a custom action to manage version and store my prompts that are used by the system. I would also perhaps want to store and use the persona templates with input variables as well to change the name or different aspects, etc. Can you give me some suggestions or innovations there that might be aligned with this system?

ChatGPT:

"Now we’re really leveling up! Storing custom prompts and persona templates with placeholders and making them dynamic opens up next-level flexibility for your system. This design not only ensures reusability but also adds a layer of personalization that can adapt to different use cases or teams."

Here’s how to design and innovate around this:

Design Goals for Custom Prompt and Persona Management

Centralized Storage:
- Store prompts and persona templates in a shared repository or custom GitHub Action for consistency across projects.
Dynamic Placeholders:
- Use placeholders in prompts and personas for runtime customization (e.g., {userName}, {riskScore}).
Versioning:
- Version prompts and personas to ensure updates don’t break compatibility.
Dynamic Retrieval:
- Fetch prompts and personas dynamically during runtime, allowing the system to adapt based on configurations.
Ease of Updates:
- Allow teams to easily update or override prompts and personas without redeploying the bot.

What to Include in Prompts and Persona Templates

1. Prompts

Prompts define how the bot interacts with the AI system for various tasks.

Example Prompt Structure

{
  "diffReview": "You are reviewing a code diff for file `{fileName}`. Provide feedback on maintainability, performance, and readability. Focus on improving code quality.",
  "ownerQuestionResponse": "The pull request owner has asked: `{question}`. Respond in a concise and helpful way.",
  "riskAssessment": "Analyze this PR and assign a risk score based on complexity, test coverage, and potential bugs. Provide actionable recommendations."
}

Placeholders in Prompts

{fileName}: Name of the file being reviewed.
{question}: The owner’s question.
{riskScore}: The calculated risk score.

2. Persona Templates

Personas control the tone and behavior of the bot.

Example Persona Template

{
  "name": "{personaName}",
  "tone": "{tone}",
  "description": "I am {personaName}, a {tone} assistant focused on improving your codebase.",
  "commentTemplates": {
    "complexity": "The function exceeds the allowed complexity. Refactor for better maintainability.",
    "coverage": "Test coverage is below {minCoverage}%. Please add tests."
  }
}

Placeholders in Personas

{personaName}: Bot’s name.
{tone}: Tone of feedback (friendly, professional, etc.).
{minCoverage}: Minimum required test coverage.

How to Store Prompts and Persona Templates

1. Centralized Storage in a GitHub Action

Repository Structure

.
├── action.yml          # Metadata for the GitHub Action
├── templates/          # Directory to store templates
│   ├── prompts.json    # AI prompts
│   ├── persona.json    # Persona templates
├── src/                # Logic for the GitHub Action
│   └── index.js
├── package.json        # Dependencies for the action
└── README.md           # Documentation for the action

Example `prompts.json`

{
  "diffReview": "You are reviewing changes to `{fileName}`. Focus on readability and maintainability.",
  "riskAssessment": "Evaluate the PR and assign a risk score based on these criteria: {criteria}.",
  "commentFollowUp": "The author has replied: {reply}. Provide further feedback if necessary."
}

Example `persona.json`

{
  "name": "{personaName}",
  "tone": "{tone}",
  "description": "I am {personaName}, your {tone} assistant for improving code quality."
}

2. Versioning the Prompts and Personas

Use tags or semantic versioning in your action repository to manage changes.

Versioning Examples

Tag versions: v1.0.0, v1.1.0

Use action.yml to define the version:

name: "AIPR Prompt and Persona Manager"
description: "Manages custom prompts and personas for the AIPR ReviewBot."
runs:
  using: "node16"
  main: "src/index.js"
version: "v1.0.0"

Dynamic Retrieval During Runtime

1. Fetch Prompts and Personas

Action to Retrieve Templates

Write a custom GitHub Action that serves prompts and personas dynamically.

name: Retrieve Prompts and Personas

inputs:
  templateType:
    description: "Type of template to retrieve (prompts or persona)."
    required: true
outputs:
  templateContent:
    description: "The content of the retrieved template."
runs:
  using: "node16"
  main: "src/index.js"

`index.js` Implementation

const fs = require('fs');
const path = require('path');

async function run() {
  const templateType = process.env.INPUT_TEMPLATE_TYPE; // "prompts" or "persona"
  const templatePath = path.join(__dirname, '..', 'templates', `${templateType}.json`);

  if (!fs.existsSync(templatePath)) {
    throw new Error(`Template ${templateType} not found.`);
  }

  const templateContent = fs.readFileSync(templatePath, 'utf8');
  console.log(`::set-output name=templateContent::${templateContent}`);
}

run().catch((error) => {
  console.error(error.message);
  process.exit(1);
});

2. Use Prompts and Personas with Placeholders

Dynamic Placeholder Replacement

Replace placeholders at runtime with actual values.

function applyPlaceholders(template, variables) {
  return template.replace(/\{(\w+)\}/g, (_, key) => variables[key] || `{${key}}`);
}

// Example Usage
const promptTemplate = "You are reviewing `{fileName}`. Focus on `{criteria}`.";
const variables = { fileName: "src/app/file.ts", criteria: "readability and maintainability" };

const filledPrompt = applyPlaceholders(promptTemplate, variables);
console.log(filledPrompt);
// Output: "You are reviewing `src/app/file.ts`. Focus on `readability and maintainability`."

Fetch and Use Prompts in Runtime

async function fetchTemplate(templateType) {
  const { execSync } = require('child_process');
  const output = execSync(`npx your-org/aipr-template-action --templateType=${templateType}`);
  return JSON.parse(output.toString());
}

// Example Usage
(async () => {
  const prompts = await fetchTemplate('prompts');
  const filledPrompt = applyPlaceholders(prompts.diffReview, { fileName: 'src/app/file.ts' });
  console.log(filledPrompt);
})();

3. Automating Template Updates

Provide a CLI tool or GitHub Action to update prompts and personas.

CLI Command to Update Templates

npx ai-pr-review templates update prompts --file new-prompts.json

Implementation

const fs = require('fs');
const path = require('path');

function updateTemplate(templateType, newContentPath) {
  const templatePath = path.join(__dirname, '..', 'templates', `${templateType}.json`);
  const newContent = fs.readFileSync(newContentPath, 'utf8');
  fs.writeFileSync(templatePath, newContent);
  console.log(`${templateType} updated successfully.`);
}

// Example Usage
updateTemplate('prompts', './new-prompts.json');

Suggestions for Innovation

Template Overrides:
- Allow repository-specific overrides of prompts and personas to customize behavior for specific projects.
- Example: A repository-specific bot-config.json file that overrides default templates.
Dynamic Persona Switching:
- Switch personas dynamically based on context.
- Example: Use a friendly tone for internal repos and a formal tone for public repos.
Template Validation:
- Add a validation step to ensure prompts and personas use valid placeholders.
- Example: Throw an error if {invalidPlaceholder} is found.
Version-Aware Prompts:
- Fetch prompts based on the bot version or repository-specific settings.
- Example: Use versioned directories in the action (v1/prompts.json).

Next Steps

Build and Deploy the Template Management Action:
- Centralize prompts and personas for easy reuse.
Integrate Dynamic Placeholder Replacement:
- Ensure prompts and personas adapt to runtime context.
Version and Update Templates:
- Use semantic versioning to manage updates.

Let me know where you’d like to take this—this is shaping up to be a flexible and robust system! 🚀

Design Goals for Custom Prompt and Persona Management​

What to Include in Prompts and Persona Templates​

1. Prompts​

Example Prompt Structure​

Placeholders in Prompts​

2. Persona Templates​

Example Persona Template​

Placeholders in Personas​

How to Store Prompts and Persona Templates​

1. Centralized Storage in a GitHub Action​

Repository Structure​

Example prompts.json​

Example persona.json​

2. Versioning the Prompts and Personas​

Versioning Examples​

Dynamic Retrieval During Runtime​

1. Fetch Prompts and Personas​

Action to Retrieve Templates​

index.js Implementation​

2. Use Prompts and Personas with Placeholders​

Dynamic Placeholder Replacement​

Fetch and Use Prompts in Runtime​

3. Automating Template Updates​

CLI Command to Update Templates​

Implementation​

Suggestions for Innovation​

Next Steps​