GSoC Project proposal: CommitHunter: AI-Powered Commit Debugger

Project Idea Name

CommitHunter: AI-Powered Commit Debugger

Description 

The goal of this project is to develop an automated system that identifies problematic Git commits causing test failures in both performance and non-performance test scenarios. Given a "Good" build (where tests pass) and a "Bad" build (where tests fail), the system will analyze all intermediate commits to pinpoint the problematic commit(s). The approach will evolve from rule-based methods to AI-driven models for higher accuracy.

Phase 1: Rule-Based Approach

Use Rule based approach and below are some examples:

1. String Matching: Identify test failure messages and correlate them with commit messages, logs, or diffs.
2. Binary Search for Performance Tests: Implement a binary search approach to efficiently narrow down the commit range in performance test failures.

Phase 2: Machine Learning Model

1. Train an ML model using historical build data, commit logs, and failure reports.
2. Utilize supervised learning techniques to classify commits as "Likely Problematic" or "Safe."
3. Use natural language processing (NLP) to analyze commit messages and correlate them with test failures.

Phase 3: Automation and Integration

1. Integrate the approach with Git repositories (e.g., GitHub) via APIs.
2. Develop a bot to automatically comment on Git issues with the identified problematic commit(s) if confidence levels meet a reliability threshold.
3. Provide a dashboard for tracking identified problematic commits and their validation over time.

Links to Eclipse Project

• https://github.com/eclipse/openj9
• https://github.com/adoptium/aqa-tests
• https://github.com/adoptium/aqa-test-tools
• https://projects.eclipse.org/projects/technology.openj9
• https://projects.eclipse.org/projects/adoptium.aqavit

Expected outcomes 

• Reduction in manual effort needed to debug and triage test failures.
• Faster identification of problematic commits leading to improved development efficiency.
• A scalable system that can be expanded with improved AI models that can apply to other project and team.

Skills required/preferred 

• Languages & Frameworks: Python (for ML and automation), Git APIs, NLP libraries (e.g., SpaCy, BERT, GPT-based models).
• CI/CD Integration: GitHub Workflow, Jenkins
• Database & Storage: MongoDB (or PostgreSQL/MySQL) for storing historical build data and test results.
• Deployment: integration with current development workflow and pipelines.

Project size 

350 hours

Possible mentors: 

• Shelley Lambert slambert@redhat.com
• Lan Xia lan_xia@ca.ibm.com
• Longyu Zhang longyu.zhang@ibm.com

Rating 

Medium - hard