Natural Language Processing

Natural Language Processing (NLP) is the domain of AI that helps computers understand, process and respond to human language, both written and spoken.

To understand text, an NLP system works through five stages: lexical analysis, syntactic analysis, semantic analysis, discourse integration and pragmatic analysis. Step through them with each example sentence.

A chatbot is a program that talks with people in natural language. A script bot follows a fixed script with set options. A smart bot uses AI to understand many ways of asking, learns from data and remembers the conversation.

Before a machine can work with text, we clean and simplify it. Text normalisation reduces the text to its basic form in steps: sentence segmentation, tokenisation, removing stopwords, special characters and numbers, converting to a common case, and stemming or lemmatisation.

Bag of Words turns each document into numbers: how often each vocabulary word appears in it, ignoring word order. TF-IDF improves on this by giving more weight to words that are frequent in one document but rare across the corpus.

Sentiment analysis finds the feeling in a piece of text. A simple model gives each word a score, flips the score after 'not' and boosts it after 'very', then adds everything up. In Orange, the Sentiment Analysis widget does this for thousands of texts at once.