Note that this tutorial is not a reference work. Its coverage of Python and NLP is selective, and presented in a tutorial style. For reference material, please consult the substantial quantity of searchable resources available at https://www.python.org/ and https://www.nltk.org/
What You Will Learn
By digging into the material presented here, you will learn:
• How simple programs can help you manipulate and analyze language data, and how to write these programs
• How key concepts from NLP and linguistics are used to describe and analyze language
• How data structures and algorithms are used in NLP
• How language data is stored in standard formats, and how data can be used to evaluate the performance of NLP techniques
Why Python?
Python is heavily used in industry, scientific research, and education around the world. Python is often praised for the way it facilitates productivity, quality, and maintainability of software. A collection of Python success stories is posted at http://www .python.org/about/success/.
NLTK defines an infrastructure that can be used to build NLP programs in Python. It provides basic classes for representing data relevant to natural language processing; standard interfaces for performing tasks such as part-of-speech tagging, syntactic parsing, and text classification; and standard implementations for each task that can be combined to solve complex problems.
NLTK comes with extensive documentation. In addition to this tutorial the website at http://www.nltk.org/ provides API documentation that covers every module, class, and function in the toolkit, specifying parameters and giving examples of usage. The website also provides many HOWTOs with extensive examples and test cases, intended for users, developers, and instructors.
Software Requirements
- Python
The material presented in this book assumes that you are using Python version 2.4 or 2.5. We are committed to porting NLTK to Python 3.0 once the libraries that NLTK depends on have been ported.
- NLTK
The code examples in this book use NLTK version 2.0. Subsequent releases of NLTK will be backward-compatible.
- NLTK-Data
This contains the linguistic corpora that are analyzed and processed
- NumPy (recommended)
This is a scientific computing library with support for multidimensional arrays and linear algebra, required for certain probability, tagging, clustering, and classification tasks.
- Matplotlib (recommended)
This is a 2D plotting library for data visualization, and is used in some of the tutorial’s code samples that produce line graphs and bar charts.
- NetworkX (optional)
This is a library for storing and manipulating network structures consisting of nodes and edges. For visualizing semantic networks, also install the Graphviz library.
- Prover9 (optional)
This is an automated theorem prover for first-order and equational logic, used to support inference in language processing.
Natural Language Toolkit (NLTK)
NLTK was designed with four primary goals in mind:
Simplicity: To provide an intuitive framework along with substantial building blocks, giving users a practical knowledge of NLP without getting bogged down in the tedious house-keeping usually associated with processing annotated language data
Consistency: To provide a uniform framework with consistent interfaces and data structures, and easily guessable method names
Extensibility: To provide a structure into which new software modules can be easily accommodated, including alternative implementations and competing approaches to the same task
Modularity :To provide components that can be used independently without needing to understand the rest of the toolkit
Automatic Natural Language Understanding
On a more philosophical level, a long-standing challenge within artificial intelligence has been to build intelligent machines, and a major part of intelligent behavior is understanding language. For many years this goal has been seen as too difficult. However, as NLP technologies become more mature, and robust methods for analyzing unrestricted text become more widespread, the prospect of natural language understanding has re-emerged as a plausible goal.
Word Sense Disambiguation
Word Sense Disambiguation In word sense disambiguation we want to work out which sense of a word was intended in a given context.
a. The lost children were found by the searchers (agentive)
b. The lost children were found by the mountain (locative)
c. The lost children were found by the afternoon (temporal)
Pronoun Resolution
a. The thieves stole the paintings. They were subsequently sold.
b. The thieves stole the paintings. They were subsequently caught.
c. The thieves stole the paintings. They were subsequently found.
Generating Language Output
If machines can understand language properly, they can perform advanced tasks like:
- Question Answering
- Machine Translation
1. Question Answering
A machine should understand the meaning of a sentence and answer correctly.
Example
Text:
The thieves stole the paintings. They were subsequently sold.
Question:
Who or what was sold?
Correct Answer:
The paintings
Here, the machine must understand that “They” refers to “paintings”, not “thieves”.
2. Machine Translation
For correct translation, the machine must understand the actual meaning of the sentence.
Example
English:
The thieves stole the paintings. They were subsequently found.
In French, the pronoun changes depending on what “They” refers to:
- Ils → if it refers to thieves
- Elles → if it refers to paintings
So, correct translation depends on correct language understanding.
Limitations of Natural Language Processing
NLP has made significant progress in tasks like machine translation, question answering, and sentiment analysis. However, NLP systems still have important limitations. They cannot perform common-sense reasoning effectively, lack world knowledge, struggle with ambiguity, sarcasm, and context, and depend heavily on training data. Most current NLP systems use superficial yet powerful techniques rather than true understanding. Despite these limitations, NLP remains useful for building practical language-based applications.