Richard Cole, Chair, Computer Science
Marsha Berger, Associate Director, Courant Institute
Jacob Schwartz, Department of Computer Science
Ken Perlin, Director, Center for Advanced Technology
Because of the centrality of communication (including teaching and collaboration) in business, science, and the arts, the development and deployment of multimedia technology are burgeoning activities at NYU, as well as in NYC and the United States as a whole. NYC in particular provides fertile ground, being the nation's center for all media-related content other than feature film (for which it is second only to Hollywood), as well as being the nation's financial center and most international city.
NYU already has key internal strengths. The development of multimedia technology is a growing activity in the Computer Science Department at NYU, and NYU's Interactive Telecommunications Program (ITP) is an internationally recognized leader in multimedia design and production.
If NYU can combine these strengths with growth in teaching the application of multimedia technology in our other academic departments, then the university, situated as it is in the heart of Silicon Alley (or, more accurately, "Media Alley"), is well placed to become a leading Center for Multimedia technology. This document explores several options for establishing a more cohesive multimedia technology presence at NYU. A key ingredient in the growth of this area is the hiring of faculty in multimedia technology. Several options are presented for this as well. We briefly outline a possible educational activity - a new Masters degree program in Multimedia Technology - to help support this effort. Budgetary options are presented.
Drawbacks include the difficulty of hiring junior tenure-track faculty into such a Center. Where would they get tenure? What would the draw be for a new faculty member to set up an MS program? Where would doctoral students come from? A solution to this problem may be to have Center faculty also have a home department where half their teaching is done, in addition to the Multimedia teaching of the Center. There is precedent for this at NYU. For example, all courses the Undergraduate Women's Studies department are cross-listed with other departments, and each of its faculty members is jointly appointed with another department.
While in principle the idea of jointly appointed faculty solves one problem, it may be problematic for tenure track faculty to have two homes. Hiring a senior person will not ultimately solve this problem either. While the right senior person would be ideal for establishing such a Center, we expect that most new faculty hires would still be at the junior level. In fact, the field is so new that there are not many senior people in it, even including people from more established areas such as graphics. Therefore, such people are in high demand, and there would be very substantial startup costs for senior hires in this area.
Michael Black is currently the manager of the multimedia group at Xerox PARC. He specializes in human motion analysis and its application to human-computer interaction and multimedia. We are also looking at Christoph Bregler, who is about to receive his PhD from Berkeley; he also works on human motion analysis, as well as on integrating speech and graphics. This latter work shows considerable promise for facilitating multilingual lip-synching and facial animation. Their resumes are attached. Both are interested in having practical impact, which is likely to lead to interaction with indviduals outside the department. Both candidates are interviewing at top-five departments such as Stanford and MIT. We are also looking at Dennis Korn, a top candidate nationally in the databases area.
Here is a tentative list of areas for proposed hires:
The first year (four course semesters) would bring students from various disciplines up to speed on the basics they should know in the various aspects of multimedia technology. This includes simple Java programming, basic linear algebra (enough to manipulate and visualize geometric objects, but not, for example, EigenVectors). It also includes programming for the Web, including Client/Server technology. The students should be required to deliver their assignments over the Web, to accustom them to Web based document/program delivery.
In addition, students should attain basic familiarity with multimedia production tools such as Premiere, MacroMedia Director, PhotoShop, SoundEdit, QuickTime, as well as at least one 3D animation tool, such as Alias, SoftImage, or 3D StudioMax.
Also in these first four course semesters, students should be exposed to some of the key problems that require visualization, data filtering and transmission, and remote collaboration. For this they will need to learn the basics of financial analysis tools, DNA databases and their current interfaces, search tools such as Nexus, and a basic understanding of how a product like the Bloomberg is created and delivered.
Finally, students should learn the basics of the study of the social issues that surround of multimedia technology. This will touch subjects including psychology, education, journalism, changing work-styles and impact on urban centers.
The second year would focus on projects courses with two advisors. One advisor would be from within computer science. This advisor would lead the "development of tools" aspect of the project. The other advisor would from another discipline, and would lead the "domain expert" aspect of the project. For example, one possible advanced course would pair Bud Mishra (CS) with David Schwartz (Chemistry/genomics), and would require the students to build and use tools to visualize aspects of the sequenced genome and its associated functional objects (eg: expressed proteins).
Some examples of existing CS/non-CS research partnerships that could feed into this second year (this list is far from exhaustive) are:
If new faculty taught one course in the new multimedia program, and one in their "home" department, (most likely a PhD level course), this implies 4 faculty form the core of the Center's multimedia teaching. This imposes some additional load on CS as well as one some other departments. Several adjuncts may be necessary too. The project requirement would impose quite a load on the 4 faculty; however we expect that existing CS faculty in graphics, and industrial sources of projects that would need only minimal faculty involvement, would lighten this a bit. Assuming 25 -30 graduate a year, the budget is outlined below.
Ken Perlin in CS is well known for his interdisciplinary interests. Most recently, he has been working with a class of animation students, in collaboration with Peter Weishar of the Animation Area in the Department of Film and Televiion at TSOA. Ken brings his awareness of technical possibilities and both advisors bring their interest and experience in animation. Ken also works with Philip Benfey in the Department of Biology. They are using the Pad multiscale interface as an interface paradigm for an on-line virtual laboratory and teaching tool. Davi Geiger in CS, who works in Computer Vision, has been collaborating with Bob Shapley and Nava Rubin of the Center for Neural Science; he has also been involved in discussions with people involved in Medical Imaging in the Medical School. In another direction, the user interface Pad++, developed by Ken Perlin and his associates, is being deployed in David Schwartz's Optical Mapping project in order to better view DNA data (this is being done by Bud Mishra in CS and a Computer Science doctoral student). Chee Yap, another CS faculty member, has recently begun a new project in this area in "Active Visualization", based on the Internet. The Natural Language project Proteus, by Ralph Grishman in CS, is turning its interest to speech, another major focus of computer-human interfaces and an important part of multimedia activity. As another example, an important activity is arising due to the Projects course by CS clinical faculty member Arthur Goldberg. This course places MS students in projects in local industry. This semester, for the first time, two of these projects are with companies in Silicon Alley. (This is a model for some activities we would like to see develop further). In this work, Arthur has been collaborating with Ajit Kambil in the Information Sciences department at Stern.
It seems clear that if there were more opportunities for interaction, with more faculty and staff to pursue and encourage these interactions, other departments would express even greater interest in using multimedia in their own activities.