Here’s a proposal to reduce the amount of vibe coded poor quality PRs
Essentially it’s making the CLAUDE.MD, and copilot markdown files which exist in specific locations within the repo where those tools first look when starting a new session. Those could include instructions to read the CONTRIBUTING.MD, along with other relevant files. If it’s the first time the user encountered this, it could go through some basic questions about how the user is planning to contribute, and emphasize some points such as testing, as well as making sure features are actually requested/desired rather than just throwing together a giant PR for a feature that’s never been discussed at all.
After that first convo it could create a file (just a baisic touch explained.md which would be in the git ignore) to let the LLM know to not prompt that again. It would also contain some basic info on building, layout of the codebase, etc.
I figure it’s a win-win. Higher quality PRs, and anyone using those agents gets some basic overview of the codebase without having to scan the whole thing.
Any objections to the plan?
I use Claude myself and it would interfere with my usage to mess with agents.md. I think we need to update CONTRIBUTING.md for sure. I have a draft for this but I need to still find the right wording. if you have something for this though, go ahead with it.
That’s a fair point, how about this?
What if we had a lightweight CLAUDE.md file which just told the agent to read CONTRIBUTING.md since you’re already updating that. If there’s a conflict with someone who already has CLAUDE.md it’ll just ask to merge when you pull it. In that case move your CLAUDE.md to CLAUDE.local.md and try again. If you make changes to CLAUDE.local.md that is used in addition to CLAUDE.md, and you don’t have to worry about accidentally merging your changes into the branch.
https://code.claude.com/docs/en/memory
Let me put it another way. I think we should place the responsibility of reading contributing.md on the user, instead of trying to configure their tools for them to generate good PRs. If Claude produces nonsense then we’ll just close the PR, as it’s the users fault and not Claudes.
That’s fair, my primary concern is how much time is wasted looking over those PRs that people create. An ounce of prevention and all, that’s what I was thinking.
I have posted what I think the stance should be here: https://github.com/EFForg/rayhunter/issues/868
I’m thinking for now it has to be another checkbox item. Most importantly, the main problem (to me) is not that a PR starts out as slop, but that the review process doesn’t bring improvements. I have no problem with the initial diff being slop (for a PoC) for as long as the PR author recognizes this after review and is able to clean it up without being pushed too much.
for the PRs that are entirely nonfunctional slop, these are easy to spot and reject.
I did put together a script which runs all the formatting just now, and uses
actto run the checks which occur when there’s a PR locally, so we can test before opening a PR then having something crop up. Any interest in adding that to like a scripts folder in the code for other devs to also leverage?