6.5.3 LLM Debates

LLM Debates for more informed decisions

Your idea of LLM-structured debate with built-in constraints is actually pretty strong for surfacing high-quality arguments efficiently. It would probably produce better discussions than most real open-source projects have today.

But it's still fundamentally: LLMs generate arguments → humans judge which arguments win and whether the constraints were right.

You haven't actually removed the human decision-making; you've just made it more informed and structured. That might be exactly what you want—better information for human decision-makers. But it's not "LLMs deciding," it's "LLMs arguing while humans referee and set the rules."

Is that what you meant, or were you hoping for something where humans step back further?

Making It Work

Constraint injection: One LLM argues "in favor, given that we must maintain backward compatibility and we have limited maintainer capacity" while the other argues "against, but only if we accept breaking changes and have three committed maintainers willing to own this."

Stakeholder representation: Rather than generic debate, you could have one LLM argue from the perspective of "a long-term maintainer of this codebase" and another from "a user depending on this for production work." They'd explicitly weigh concerns differently based on their role.

Value articulation: You explicitly tell both LLMs: "Our project prioritizes stability over features" or "We optimize for contributor experience over user experience," and both debate within those stated values rather than ignoring them.

Maintenance commitment modeling: The pro-LLM could be constrained to only argue for features where it can explicitly identify who will maintain it and for how long. The con-LLM could be required to engage seriously with maintenance plans rather than just saying "this is too complex."

Novel conflict detection: You could ask the LLMs to explicitly identify where their underlying assumptions differ, which might surface the real disagreement (e.g., "We disagree on whether this project should serve enterprise users or just individual developers").

But There's Still a Critical Gap

The human decision-maker still needs to do something LLMs can't: judge whether the constraints themselves are correct.

An LLM-debate could conclude: "Given that we have no maintainers willing to own this, we should reject the feature." But what if a human reviewer thinks: "Actually, this constraint is wrong—I'm willing to maintain it, I just wasn't asked"? Or: "That constraint is outdated; we restructured how we handle maintenance last quarter"?

Or more subtly: "Both of you are debating under the assumption that we need to support all legacy versions, but we should actually have this discussion about whether that assumption still makes sense."

The constraint-setting itself is a human judgment that can't be automated. The LLMs can debate within constraints, but humans must choose which constraints to apply, and that's where the real governance happens.