GPU Programming

CS 179

The use of Graphics Processing Units for rendering is well known, but their power for general parallel computation has only recently been explored. Parallel algorithms running on GPUs can often achieve up to 100x speedup over similar CPU algorithms, with many existing applications for physics simulations, signal processing, financial modeling, neural networks, and countless other fields.

This course will cover programming techniques for the GPU. The course will introduce NVIDIA's parallel computing language, CUDA. Beyond covering the CUDA programming model and syntax, the course will also discuss GPU architecture, high performance computing on GPUs, parallel algorithms, CUDA libraries, and applications of GPU computing. Problem sets will cover performance optimization and specific GPU applications in numerical mathematics, medical imaging, finance, and other fields.

Labwork will require significant programming. A working knowledge of the C programming language will be necessary. Although CS 24 is not a prerequisite, it (or equivalent systems programming experience) is recommended.

9 units; third term.

Instructors: Kevin Yuh -
Eric Martin -
If you have a question/issue that likely affects other students, please ask it through Piazza. If you have something that only affects you or your project group, please send an email to both instructors.

Supervising professors: Professor Al Barr -
Time and place: MWF 3:00-3:55 PM
Location: ANB107
Office Hours: Kevin Yuh - Mondays, 9-11 PM this week: Thursday, 8-10 PM
Eric Martin - Tuesdays, 7-9 PM
Andrew Zhao - Mondays, 9-11 PM this week: Thursday, 8-10 PM

104 Annenberg, instructional laboratory
Grading policy: There are 2 grading schemes for the class that each student can choose beween:
  • 5 labs (50% of grade) & 5 week project (50% of grade)
  • 7 labs (70% of grade) & 3 week project (30% of grade)
All labs will be scored out of 100 and are weighted equally (meaning each lab is worth 10% of your grade). Either duration of project can be completed individually or as a pair.

Extensions may be granted if the TA's see it appropriate. E grades will not be granted unless under extreme circumstances.  

Project: Project intro slides: PPT PDF
Project ideas & proposal description slides: PPT PDF

Lectures: Lecture 1: PPT PDF (updated - new assignment due date)
Lecture 2: PPT PDF
Lecture 3: PPT PDF
Lecture 4: PPT PDF
Lecture 5: PPT PDF
Lecture 6: PPT PDF
Lecture 7: PPT PDF
Lecture 8: PPT PDF (updated: including cufftDestroy)
Lecture 9: PPT PDF (updated: cufftDestroy, batch issues, new due date)
Lecture 10: PPT PDF
Lecture 11: See "Project ideas & proposal description" above
Lecture 12: PPT PDF (v3.3 5/2 8:02 PM)
Lecture 13: PPT PDF
Lecture 14: PPT PDF
Lecture 15: PPT PDF
Lecture 16: PPT PDF
Lecture 17: PPT PDF
Lecture 18: PPT PDF
Lecture 19a: PPT PDF
Lecture 19b: PPT PDF
Lecture 20: PPT PDF

Assignments: Homework 1: Text Code (text updated: new performance estimates) (code updated: bugfix in block calculation)

Homework 2: Text & Code

Homework 3: Text Code (ver. 2 - text revised 4/24, 12:22 AM)

Homework 4: Text Code (text and images revised 4/27, 5:13 PM, code revised 9:14 PM)

Homework 5: Text & Code

Homework 6: Text & Code

Homework 7: Text Code

Textbook: The CUDA Handbook: A Comprehensive Guide to GPU Programming is recommended but not required. The book is a little more advanced than other CUDA books and might not be very approachable at first, but by the end of the course you should find it to be a valuable reference for both your project as well as any future GPU programming you find yourself doing. Buy through the author's site for a discount.

Resources: CUDA C Programming Guide
Mapping from GPU name to Compute Capability

Less useful, but cool resources: NVIDIA's Parallel Forall Blog
Videos from last several years of NVIDIA's conference on CUDA
How to Write Code the Compiler Can Actually Optimize (2015)
Excellent CPU optimization manuals
What Every Programmer Should Know About Memory
GPU focused systems guide to deep learning