ECE 467: Natural Language Processing
Fall 2024
Mondays 2:00 PM - 2:50 PM, Rm. 502;
Wednesdays 11:00 AM - 12:50 PM, Rm. 502
Instructor:
Carl Sable
Email: carl.sable@cooper.edu
Office: Room 614
"Speech and
Language Processing, 3rd Edition"
(draft, in-progress, on-line only)
by Daniel Jurafsky and James H. Martin
We will also rely on content from various published papers, as cited
throughout the course.
Assignments will be posted here when they are assigned.
Note 1: All dates and topics which have not yet occurred are
tentative.
Note 2: All below references to chapters and sections of the Jurafsky
and Martin textbook are based on a draft version of the 3rd edition
dated August 20, 2024, unless otherwise noted.
Part I: Pre-deep-learning NLP
- Wednesday, September 4
Topic 1: Course Introduction
(Slides)
See the syllabus as a
single document
- Monday, September 9
Topic 2: Words, Morphology, Subwords, and Tokenization
(Slides)
Suggested reading: Chapter 2 of Jurafsky and Martin
- Wednesday, September 11
Topic 2 (continued)
Topic 3: Vector Space Models, Information Retrieval, and Text
Categorization
(Slides)
Suggested reading: Sections 4.1 - 4.8, Sections 6.3 - 6.5,
Section 14.1 of Jurafsky and Martin
- Monday, September 16
Topic 3 (continued)
- Wednesday, September 18
Topic 3 (continued)
Project #1 assigned
- Monday, September 23
Topic 4: Linguistic Concepts: Parts of Speech, Syntax, Semantics,
Pragmatics, Discourse, Coreference
(Slides)
Suggested reading: Section 17.1, Sections 18.1 - 18.6, Section
19.1, Section 23.1, Section 23.9, Section 24.1, Sections F.1 - F.4
(appendices are provided as separate files on the textbook's website) of
Jurafsky and Martin
- Wednesday, September 25
Topic 4 (continued)
- Monday, September 30
Quiz #1
- Wednesday, Octobber 2 (first half)
Topic 4 (continued)
Part II: Pre-transformer deep-learning-based NLP
- Wednesday, October 2 (second half)
Topic 5: Feedforward Neural Networks (units, layers,
activation functions, backpropagations, hyperparameters,
training/validation/testing, etc.)
(Slides)
Suggested reading: Sections 7.1 - 7.5 of Jurafsky and Martin
- Monday, October 7
Topic 5 (continued)
- Wednesday, October 9
Topic 5 (continued)
Topic 6: Word Embeddings; Neural Language Models; Word2vec
(Slides)
Suggested reading: Sections 6.8 - 6.10, Sections 7.6 - 7.7 of
Jurafsky and Martin,
original word2vec paper,
negative sampling paper
- Monday, October 14
Topic 6 (continued)
- Wednesday, October 16
Topic 6 (continued)
Topic 7: Recurrent Neural Networks (RNNs); Long Short-Term Memory
Networks (LSTMs)
(Slides)
Suggested reading: Sections 8.1 - 8.6 of Jurafsky and Martin,
popular
LSTM blog
- Monday, October 21
Topic 7 (continued)
Topic 8: Sequence-to-Sequence Tasks and Encoder-Decoder Models;
Cross-Attention; Machine Translation
(Slides)
Suggested reading: Sections 8.7 - 8.8, Sections 13.1 - 13.4,
Section 13.6 of Jurafsky and Martin
- Wednesday, October 23
Topic 8 (continued)
Introduction to Jupyter Notebooks, Google Colab, and PyTorch
Project #2 assigned
- Monday, October 28
Quiz #2
- Wednesday, October 30
Topic 9: Question Answering, Reading Comprehension, and SQuAD;
Contextual Embeddings and ELMo
(Slides)
Suggested reading: Sections 14.3 - 14.4 of Jurafsky and Martin
(February 3, 2024 draft), ELMo paper
Part III: Post-transformer deep-learning-based NLP
- Monday, November 4
Topic 10: Transformers
(Slides)
Suggested Reading: Chapter 9 of Jurafsky and Martin,
original transformer
paper
- Wednesday, November 6
Topic 10 (continued)
Topic 11: BERT and BERT Variations; Pretraining, Fine-tuning and
Transfer Learning; the GLUE Benchmark
(Slides)
Suggested Reading: Chapter 11 of Jurafsky and Martin,
original BERT paper
- Monday, November 11
Topic 11 (continued)
- Wednesday, November 13
Topic 12: Large Language Models (LLMs); GPT (several variations);
Reinforcement Learning from Human Feedback (RLHF)
(Slides)
Suggested Reading: Sections 10.1 - 10.5 of Jurafsky and
Martin,
GPT-1
paper,
GPT-3 paper,
InstructGPT paper
- Monday, November 18
Topic 13: Ethics of NLP; Relevant Philosophical Issues
(Slides)
Suggested Readings: Section 4.10, Section 6.11, Section 10.6,
Section 13.7, Section 15.5.1 of Jurafsky and Martin,
Stochastic
Parrots paper
- Wednesday, November 20
Topic 13 (continued)
Project #3 assigned
- Monday, November 25
Quiz #3
Part IV: Class time to work on final projects
- Monday, December 2
Class time to work on final projects
- Wednesday, December 4
Class time to work on final projects
- Monday, December 9
Class time to work on final projects
- Wednesday, December 11
Class time to work on final projects
- Monday, December 16
Class time to work on final projects
- Wednesday, December 18
Final Project Presentations