What an LLM Actually Is (No Training Math Required)
You're Calling a Function, Not Talking to a Brain
From where your code sits, a large language model is not a mind you're having a conversation with - it's a stateless HTTP endpoint. You send it text (a prompt, or a structured list of messages), and it sends back text (a completion): one request, one response, nothing more mystical than any other API call you've made. The 'conversation' feeling comes entirely from what your application does with that exchange - not from anything happening inside the model between calls.
{
"model": "some-llm",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What's a closure in JavaScript?"}
],
"max_tokens": 300
}
--- response ---
{
"role": "assistant",
"content": "A closure is a function that remembers variables from where it was defined, even after that outer function has returned...",
"usage": {"input_tokens": 24, "output_tokens": 41}
}Tokens Are the Actual Unit of Everything
Every LLM API meters usage in tokens, not words or characters. A token is roughly a few characters or a word fragment - the word 'unbelievable' might split into 'un', 'believ', 'able'. Pricing, rate limits, and the model's context window (the maximum it can read and respond to in one call) are all specified in tokens, because tokens are the actual unit the model processes internally - words are a human convenience layered on top, not something the API deals in directly.
A rough rule of thumb for English text: 100 tokens is about 75 words. Fine for a back-of-envelope estimate of what a feature might cost; not something you'd want to bill a customer against precisely - actual tokenization depends on the specific model and isn't something you compute by hand.
Why It 'Forgets' the Moment You Close the Tab
Because each call is stateless, the model does not remember your previous messages, your name, or anything from an earlier turn - unless your application resends it. Every 'conversation' in a chat product is really the caller (your frontend or backend) concatenating the full history - every prior user message and every prior model reply - and sending that entire transcript back to the API on every single turn. The model isn't recalling anything; it's reading the same transcript fresh each time and generating whatever comes next.
This has two direct, practical consequences. First, a conversation's effective length is capped by the model's context window - the total tokens it can accept per call, full history included. Second, cost and latency both grow with every turn, since you're resending everything before it, not just the new message - a chat feature at turn 100 costs roughly 100x what turn 1 cost, because it's resending 99 prior turns as context every single time.
The Knobs You're Actually Setting
A handful of parameters in the same request let you control how the model generates its response, without touching anything about how it was trained. `temperature` controls randomness: values close to 0 make output more deterministic and repeatable, higher values make it more varied at the cost of predictability. `max_tokens` caps how long the response is allowed to be, mostly as a cost and latency control, since the model has no built-in notion of how long its own answer 'should' be. Different providers expose slightly different names for the same handful of ideas, but these caller-facing controls - not the model's internals - are what you're actually adjusting when you tune API behavior.
What 'Hallucination' Really Means From This Side
There is no database lookup happening, no fact-checking step, and no built-in notion of 'I don't know' in the basic mechanism - so a model will confidently generate a fluent, plausible-sounding answer even when it's flatly wrong. That's what 'hallucination' refers to. It isn't a rare bug that eventually gets patched out; it's a predictable consequence of how next-token prediction works: the model is always producing the statistically likely continuation of the text so far, not consulting a source of truth.
This is exactly the problem retrieval-augmented generation (RAG) exists to reduce: instead of trusting the model's own memorized 'knowledge,' you fetch actual, current documents and put them directly into the prompt as context, so the model's job shifts from 'recall a fact' to 'summarize what's right in front of it.' More on that in the next lesson in this series.
Where This Actually Matters
None of this changes if you swap providers or model versions - every LLM API works this way. When you're the one wiring an LLM into a feature, these facts turn directly into engineering decisions: cost and latency both scale with how much history you resend, so trimming or summarizing old turns matters at scale; a fluent-sounding answer is not the same as a correct one, so anything user-facing and fact-sensitive needs either retrieval or a human review step; and 'the model doesn't remember' isn't a limitation to work around later - it's the actual contract you're building against from day one.