Text as Data
Why Text Is Hard
7 min+60 XPStep 1/10
Language defies every clean assumption
Text is the messiest data a computer will ever meet. A spreadsheet cell means what it says; a sentence plays tricks.
'I saw the man with the telescope' — who held the telescope? 'The chicken is ready to eat' — who's eating whom? Words shift meaning by context ('bank', 'Apple', 'run' has hundreds of senses), word order flips outcomes ('dog bites man' vs 'man bites dog'), and negation hides everywhere: stress a different word in 'I didn't say she took the money' and the accusation moves.
warning
Every classic NLP pipeline starts by pretending text is cleaner than this — then spends the rest of its engineering budget paying for the pretense.