Loading...
🤝
Human-in-the-Loop Agent (HULA)(HULA)
Framework for human-in-the-loop evaluation and refinement of LLM-based agents, allowing engineers to guide and assess agent performance at each development stage.
Complexity: highEvaluation and Monitoring
🎯 30-Second Overview
Pattern: Three-agent collaboration framework with AI Planner, AI Coder, and Human Agent for software development
Why: Maintains human control while leveraging AI assistance for JIRA issue resolution and code generation
Key Insight: 79% plan success, 59% PR merge rate - keeping engineers in driver's seat enables reliable AI collaboration
⚡ Quick Implementation
1Setup:Deploy AI Planner, AI Coding, and Human Agent components
2Plan:AI Planner creates coding plan from JIRA issue
3Review:Human agent reviews, refines, and approves plan
4Code:AI Coding agent generates code based on approved plan
5Validate:Human reviews code, provides feedback, approves PR
Example: hula_session = HULA(issue=jira_ticket, agents=[planner, coder, human], stages=[plan, code, review])
📋 Do's & Don'ts
✅Keep human engineer in driver's seat throughout the development process
✅Use three-stage evaluation: offline, online, and practitioner perception
✅Incorporate feedback from compilers, linters, and validation tools
✅Review and refine plans before moving to coding stage
✅Deploy in real JIRA environment for authentic evaluation
❌Allow fully autonomous operation without human oversight
❌Skip plan approval stage - human validation is critical
❌Ignore compiler/linter feedback in code generation loop
❌Expect 100% automation - human collaboration is the goal
❌Deploy without proper three-stage evaluation framework
🚦 When to Use
Use When
- • Software development teams needing AI assistance
- • JIRA-based development workflows and issue tracking
- • Organizations wanting human-controlled AI coding
- • Teams requiring code quality assurance and oversight
- • Enterprise environments with established review processes
Avoid When
- • Fully autonomous coding requirements
- • Simple scripting or one-off coding tasks
- • Teams without structured issue tracking systems
- • Projects requiring immediate code deployment
- • Environments without human review capacity
📊 Key Metrics
Plan Generation Success
79% of work items receive successful coding plans
Plan Approval Rate
82% of generated plans approved by engineers
Code Generation Success
87% of approved plans result in generated code
Pull Request Rate
25% of generated code reaches pull request stage
Merge Success Rate
59% of HULA PRs merged into repositories
SWE-bench Performance
37.2% resolution rate on SWE-bench Verified
💡 Top Use Cases
Enterprise Software Development: Atlassian deployment with 45 engineers, ~900 merged PRs
JIRA Issue Resolution: Automated plan generation and code development for work item tracking
Collaborative AI Coding: Human-guided development maintaining engineer control and oversight
Quality Assurance Workflows: Integrated compiler/linter feedback with human review processes
Research and Development: Academic-industry collaboration for human-AI software engineering
References & Further Reading
Deepen your understanding with these curated resources
Contribute to this collection
Know a great resource? Submit a pull request to add it.
🤝
Human-in-the-Loop Agent (HULA)(HULA)
Framework for human-in-the-loop evaluation and refinement of LLM-based agents, allowing engineers to guide and assess agent performance at each development stage.
Complexity: highEvaluation and Monitoring
🎯 30-Second Overview
Pattern: Three-agent collaboration framework with AI Planner, AI Coder, and Human Agent for software development
Why: Maintains human control while leveraging AI assistance for JIRA issue resolution and code generation
Key Insight: 79% plan success, 59% PR merge rate - keeping engineers in driver's seat enables reliable AI collaboration
⚡ Quick Implementation
1Setup:Deploy AI Planner, AI Coding, and Human Agent components
2Plan:AI Planner creates coding plan from JIRA issue
3Review:Human agent reviews, refines, and approves plan
4Code:AI Coding agent generates code based on approved plan
5Validate:Human reviews code, provides feedback, approves PR
Example: hula_session = HULA(issue=jira_ticket, agents=[planner, coder, human], stages=[plan, code, review])
📋 Do's & Don'ts
✅Keep human engineer in driver's seat throughout the development process
✅Use three-stage evaluation: offline, online, and practitioner perception
✅Incorporate feedback from compilers, linters, and validation tools
✅Review and refine plans before moving to coding stage
✅Deploy in real JIRA environment for authentic evaluation
❌Allow fully autonomous operation without human oversight
❌Skip plan approval stage - human validation is critical
❌Ignore compiler/linter feedback in code generation loop
❌Expect 100% automation - human collaboration is the goal
❌Deploy without proper three-stage evaluation framework
🚦 When to Use
Use When
- • Software development teams needing AI assistance
- • JIRA-based development workflows and issue tracking
- • Organizations wanting human-controlled AI coding
- • Teams requiring code quality assurance and oversight
- • Enterprise environments with established review processes
Avoid When
- • Fully autonomous coding requirements
- • Simple scripting or one-off coding tasks
- • Teams without structured issue tracking systems
- • Projects requiring immediate code deployment
- • Environments without human review capacity
📊 Key Metrics
Plan Generation Success
79% of work items receive successful coding plans
Plan Approval Rate
82% of generated plans approved by engineers
Code Generation Success
87% of approved plans result in generated code
Pull Request Rate
25% of generated code reaches pull request stage
Merge Success Rate
59% of HULA PRs merged into repositories
SWE-bench Performance
37.2% resolution rate on SWE-bench Verified
💡 Top Use Cases
Enterprise Software Development: Atlassian deployment with 45 engineers, ~900 merged PRs
JIRA Issue Resolution: Automated plan generation and code development for work item tracking
Collaborative AI Coding: Human-guided development maintaining engineer control and oversight
Quality Assurance Workflows: Integrated compiler/linter feedback with human review processes
Research and Development: Academic-industry collaboration for human-AI software engineering
References & Further Reading
Deepen your understanding with these curated resources
Contribute to this collection
Know a great resource? Submit a pull request to add it.