Understanding is not necessarily the same as correctly following rules. A system can apply rules, produce correct answers, and explain its steps without thereby demonstrating that it grasps what the rules mean or why the answers are true.
Gödel’s First Incompleteness Theorem shows that any consistent, effectively axiomatized formal system powerful enough to express arithmetic has limits. There will be arithmetical statements that the system cannot prove or disprove using only its own axioms and rules. If the system is sound, some statements will be true even though they are unprovable within it.
The system can be strengthened by adding new axioms. This allows it to settle some questions that it could not settle before. But if the stronger system remains consistent, effectively axiomatized, and sufficiently powerful, it will also contain statements that it cannot decide.
A useful analogy is a map. You can expand a map to include territory that previously lay outside it, but the expanded map still has a boundary. Expanding the map moves the edge; it does not eliminate the edge. Similarly, adding axioms allows a formal system to prove more, but no single consistent, effectively axiomatized rulebook can prove every arithmetical truth.
This does not prove that AI cannot be intelligent, conscious, computational, or capable of understanding. It establishes a narrower point: mathematical truth cannot be completely captured by one fixed formal system of the relevant kind.
The connection to intelligence requires a further argument. Mathematical understanding may involve more than deriving conclusions within a fixed system. It may include examining assumptions, comparing formal systems, identifying their limitations, and adopting stronger principles when justified. If understanding has this open-ended and reflective character, it cannot be identified with one fixed formal rulebook. This still leaves open the possibility that such open-ended reasoning could be produced by a more flexible computational process.
An AI might revise its methods, move between formal systems, use probabilistic or non-deductive reasoning, and recognize limitations in particular systems. These abilities would answer the objection that AI is permanently trapped inside one unchanging rulebook. They would not, by themselves, settle whether the AI consciously understands what it is doing.
Whether AI understands depends partly on the standard being used. A functional standard may require the system to represent meanings, connect conclusions to appropriate reasons, correct its mistakes, and use its reasoning successfully in unfamiliar situations. A stronger standard may also require conscious awareness, although whether consciousness is necessary for understanding remains disputed.
Current AI displays many outward behaviors associated with understanding. It can explain, reason, revise, and justify. These behaviors provide evidence that must be taken seriously, but they do not conclusively establish or rule out genuine understanding or consciousness.
Humans consciously experience mathematical reasoning as meaningful, while current AI demonstrates the ability to manipulate and extend the map without established evidence that it consciously understands the territory it represents.