Word Embeddings

Word embeddings turn discrete words into dense vectors. The vectors are learned from context, so terms that appear in similar linguistic environments tend to occupy nearby regions of the space. This is the distributional idea in numerical form: usage supplies the signal for meaning.

Word2Vec learns embeddings by predicting words from their context with CBOW or Skip-gram. GloVe learns from global word co-occurrence statistics. Both produce static representations, one vector per word, which means the word bank has the same representation in every sentence.

Contextual models such as ELMo, BERT, and GPT generate a representation conditioned on surrounding text. The word bank in “river bank” can therefore differ from the one in “central bank.” This shift made contextual representations more useful for modern NLP, although static embeddings remain valuable for lightweight systems and for understanding the geometry behind language models.

Distance in an embedding space is evidence of shared usage, not a guarantee of synonymy or truth. The geometry reflects the training corpus, its biases, and the objective used to learn the vectors.