Introduction
One of the quietest but most important changes in NLP was the shift from symbolic word representations to learned vector representations. Before that shift, words were often treated as isolated units. They could be indexed, counted, and compared, but the representation itself did not carry any real notion of similarity. The model knew that two words were different, but not why they might belong near each other.
Embeddings changed the basic object of language processing: a word stopped being just a discrete label and became a point in a learned geometric space.
Why this topic matters
What interests me here is the mechanism behind that shift: train a model on a prediction task, then use the learned weights as representations. This made similarity, analogy, clustering, and transfer easier to express inside the model.
Distributional hypothesis
The intuition behind word embeddings is often summarized by a familiar slogan: you shall know a word by the company it keeps. Meaning is not assigned in isolation. It emerges from contextual regularities.
A model does not need a dictionary definition of bank to begin learning something about it. It can instead infer structure from repeated co-occurrence patterns. If river and shore appear around one use of bank, while loan and interest appear around another, those neighborhoods begin to shape representation.
So the jump from symbolic NLP to neural representation learning is also a jump from static encoding to contextual statistics.
From counts to neural models
The cleanest place to begin is with one-hot encoding. If a vocabulary has size V, then each word is represented by a vector of length V with exactly one nonzero entry. The representation is sparse, high-dimensional, and completely orthogonal to every other word.
That representation is useful for indexing, but not for meaning. Two words that feel similar to a human, like cat and dog, are just as orthogonal as two unrelated words. One-hot vectors tell the model identity, but not structure.
Neural language models change this completely. Instead of treating representation as a fixed preprocessing choice, they learn dense vectors through prediction. Representation becomes useful because it supports the task itself. That is the bridge from symbolic bookkeeping to learned geometry.
CBOW and Skip-gram
Word2Vec is one of the clearest examples of this shift. Its contribution was not just computational efficiency, but a particularly influential way of learning distributed word representations from local context windows.
The central idea is simple: train a shallow neural model on a predictive task, then keep the learned embedding weights. Those weights become the word vectors.
Continuous Bag of Words, or CBOW, predicts the center word from surrounding context words. The context words are treated as an unordered window, their embeddings are combined, and the model tries to predict which word belongs in the middle.
Skip-gram reverses the direction. Instead of predicting the center word from context, it predicts surrounding context words from the center word. That reversal changes the pressure on the representation: now the center word must support multiple nearby predictions.
I think of CBOW and Skip-gram as two different lenses on the same question. Should context explain the target, or should the target explain its context?
What the model learns
The move to embeddings allows words with similar usage patterns to end up near one another in space. Meaning becomes something the model approximates through distributional structure. Words that appear in similar contexts tend to acquire similar representations.
That is the point where representation starts to feel less like bookkeeping and more like learning.
This is also why analogy demonstrations became so famous. They were not the deepest proof of understanding, but they made the geometry visible. Vector arithmetic seemed to reveal that the model had captured some relational regularities rather than only memorizing isolated facts.
Why neural language models matter
Embeddings gave NLP a smoother representational space. Once words lived in vectors rather than orthogonal slots, a model could transfer what it learned about one semantic neighborhood to nearby ones.
Transformers did not appear out of nowhere. Contextual representations did not appear out of nowhere. They were built on the earlier insight that useful linguistic structure could be learned rather than manually specified.
What I find interesting
At the same time, static embeddings also have real limits. A single vector for a word cannot fully capture polysemy or context-sensitive meaning. The word bank still gets one representation even though its meaning may change depending on the sentence.
Word2Vec is not the final answer, but it is one of the clearest bridges to later contextual and transformer-based models.
Closing thought
CBOW, Skip-gram, and Word2Vec are useful entry points because they make representation learning inspectable: prediction creates structure, and that structure becomes reusable.