Useful libraries

3D Graphics

OpenGL is a standard part of Windows, Linux and MacOS. You can find lots of samples, tutorials and additional software at www.opengl.org

A higher-level library is OpenInventor, (another free implementation is available at www.coin3d.org but I did not try it). It takes some effort to learn, but provides many useful primitives and 3D interface widgets.

GUI

GLUT is probably the simplest GUI package to use, but has few features.

GLUI provides some additional functionality and a better API.

Of course, you can use "real" GUI toolkits (Qt, GTK etc) but these usually take a while to learn. Furthermore, OpenGL integration is often far from trivial.

Some students had a good experience with wxWindows.

Geometry

GTS is a useful library for triangular meshes; there are many other options.

CGAL is a comprehensive library of geometric algorithms, some of which are very hard to implement robustly on your own.

Denis Zorin