My battle tested review workflow

I got inspired by this post and decided to make a new one, rather than just commenting, so it reaches more people that might find it useful. Asking codex to execute reviews with specified lenses is a excellent way to identify missteps during implementation of a feature or changes. But there are 2 reasons why one review (when it finds more than a couple findings) is never enough: - LLMs are lazy, they will kinda stop trying hard after the find more than 5-6 findings - Solving findings can create new holes So you kind of need to keep prompting it to do more and more reviews until "everything" is found and resolved. The issue with that is that LLM is also overzealous. When the big issues are fixed and you keep asking it for reviews, it will keep finding issues, but most of them will be minor even ones that are not issues, but just improvements. It will even find "issues" that if implemented are actually counterproductive. Reviewing all these will waste your time. So how do you solve this? I have 2 battle tested hacks: After any review prompt add this "The purpose is not to force findings, but to focus only on findings that have material impact on the feature/change/component under review. You just do your unbiased review every time you receive this message. This is not a demand to manufacture findings". Depending on the maturity of the project and how much codex can actually vibe it (case-by-case assessment depending how much codex knows about this project and the feature we are reviewing), I might even add "For the findings you identify, implement the recommended remediation for each finding. When all findings the findings are addressed execute also the appropriate regression testing suites and manage the defects". (see disclaimer at the bottom) Bonus: if I trust codex to do the remediation for these, I just queue the full prompt (review and implement) a few times. Now there is an issue with that is that; at some point codex becomes lazy and does not really review any more. So after the 5th-6th queued prompt I add at the beginning of the prompt - before the review lenses: "You need to execute the review under the following lenses AGAIN. Do do not trust what you know already about this feature/change/component, reread the documentation and the code and redo the full analysis. This is CRITICAL authoritative requirement for the success of the review." These work amasing for me, hope they help others too. Disclosure: a) I have a full set of instructions in my repo about how should the agent code and test b) authorisations are pre-specified within the scope of the work c) almost every feature has documented requirements and design in my repo so the agent really knows what is going on. d) Gemini 3.8 Flash and precious gemini models are not reliable for this kind of work. Letting an agent roam free within a specific scope works fine in my case. The benefit is overwhelming when it works a d overshadowed the couple times it doesn't. I do not advise anyone to just let an AI agent run wild in their repo - especially for really large or complex repo - without some sorts of control what the agent is allowed to do. Hopes this helps! Like, comment, and and subscribe 😝

添加评论
点赞收藏
点踩分享查看原文
评论
?
参与讨论