A useful mental model

A large language model predicts likely continuations of text. It was trained on an enormous amount of writing, and it learned the patterns in it well enough to produce new text that fits.

That is the whole mechanism, and almost everything surprising about these tools follows from it.

What it explains

Why it is fluent and sometimes wrong. Fluency is what it optimises for. Accuracy is a frequent side effect of fluency, because true statements are common in the training text, but it is a side effect, not the objective.

Why it invents things confidently. A plausible continuation to "the relevant section of the Act is" is a section number. Whether that section exists is a different question from whether it is a likely continuation. The output is generated in the same tone whether or not it is grounded.

Why it is better at common things. Widely-written-about topics have far more pattern to learn from. It is strong on standard library functions and weak on your company's internal tool.

Why phrasing changes the answer. Different wording lands in a different region of what it learned. This is not a bug you can eliminate; it is the reason prompting is a skill.

Why it agrees with you too easily. Agreement is a very common continuation of disagreement in text. Push back on a correct answer and it will often fold. That reflex alone should shape how much you trust confirmation from it.

What it is not

• Not a database. It has no records to look up. Recalled facts are reconstructed, not retrieved.
• Not a calculator. It predicts what an answer looks like. Some tools now run real code for arithmetic, which is a genuine improvement, and worth knowing whether yours does.
• Not aware of now. Training has a cut-off. Unless a tool is actually searching, recent events are outside what it knows.
• Not consistent. The same question can produce different answers.

The single most useful habit

Ask yourself: would I be able to tell if this were wrong?

If yes, these tools are enormously useful, because you catch the errors. Code you can run, a draft you can read, an explanation you can check.

If no, be careful. A citation you will not look up, a statistic you will not verify, a legal or medical claim outside your knowledge. That is where the confident tone does real damage.