Some things are easier to describe than to find. Others are easier to recognize than to describe.
Reordering groceries may take one sentence. Choosing a chair may require seeing twenty. Reviewing an agent's changes may require a visual comparison of what existed before and what exists now.
The mistake is to assume that conversation should replace the graphical interface. Conversation reduces the effort of navigation but introduces the effort of articulation. Which is easier depends on what the person knows, what the task requires, and what remains uncertain.
An agent can do the work without making every judgment for us. We may need to explore before we can specify a goal, delegate steps we don't know how to perform, or inspect a result before we can accept it.
The design problem is not choosing between conversation and graphical interfaces. It is making the next judgment easier.
Let people articulate what is easier to say, scan what is easier to recognize, delegate what is easier to execute on their behalf, and inspect what they need to judge.