Richard Cole, Chair, Computer Science
Marsha Berger, Associate Director, Courant Institute
Jacob Schwartz, Department of Computer Science
Ken Perlin, Director, Center for Advanced Technology
Because of the centrality of communication (including teaching and collaboration) in business, science, and the arts, the development and deployment of multimedia technology are burgeoning activities at NYU, as well as in NYC and the United States as a whole. NYC in particular provides fertile ground, being the nation's center for all media-related content other than feature film (for which it is second only to Hollywood), as well as being the nation's financial center and most international city.
NYU already has key internal strengths. The development of multimedia technology is a growing activity in the Computer Science Department at NYU, and NYU's Interactive Telecommunications Program (ITP) is an internationally recognized leader in multimedia design and production.
A recent Coopers and Lybrand study shows that there is a single gap in human resources that is limiting the growth rate of the NY New Media industry: the lack of qualified technically capable college graduates entering the work force. In fact, that study specifically recommends that the Universities in this area increase their focus on turning out graduate level students who will be capable of filling this gap. Our University, situated as it is in the heart of Silicon Alley (or, more accurately, "Media Alley"), is well placed to become a leading Center for Multimedia technology.
NYU needs to combine its existing strengths, with a focus on turning out technologically literate students to the workforce, who are also well grounded in the real-world applications of the new technologies. This effort will require strong participation by the Department of Computer Science, as well as by various faculty in our other academic departments.
This document proposes to outline such a cohesive multimedia technology presence at NYU. A key ingredient in the growth of this area is the hiring of faculty in multimedia technology. We briefly outline a possible educational activity - a new Masters degree program in Multimedia Technology - to help support this effort. Budgetary options are presented.
The first year (four course semesters) would bring students from various disciplines up to speed on the basics they should know in the various aspects of multimedia technology. This includes simple Java programming, basic linear algebra (enough to manipulate and visualize geometric objects, but not, for example, EigenVectors). It also includes programming for the Web, including Client/Server technology. The students should be required to deliver their assignments over the Web, to accustom them to Web based document/program delivery.
In addition, students should attain basic familiarity with multimedia production tools such as Premiere, MacroMedia Director, PhotoShop, SoundEdit, QuickTime, as well as at least one 3D animation tool, such as Alias, SoftImage, or 3D StudioMax.
Also in these first four course semesters, students should be exposed to some of the key problems that require visualization, data filtering and transmission, and remote collaboration. For this they will need to learn the basics of real-world applications such as financial analysis tools, DNA databases and their current interfaces, search tools such as Nexus, and a basic understanding of how a product such as the Bloomberg is created and delivered.
Finally, students should learn the basics of the study of the social issues that surround of multimedia technology. This will touch subjects including psychology, education, journalism, changing work-styles and impact on urban centers.
The second year would focus on projects courses with two advisors. One advisor would be from within computer science. This advisor would lead the "development of tools" aspect of the project. The other advisor would from another discipline, and would lead the "domain expert" aspect of the project. For example, one possible advanced course would pair Bud Mishra (CS) with David Schwartz (Chemistry/genomics), and would require the students to build and use tools to visualize aspects of the sequenced genome and its associated functional objects (eg: expressed proteins).
Some examples of existing CS/non-CS research partnerships that could feed into this second year (this list is far from exhaustive) are:
A potential difficulty is that many students would not be adequately prepared to take graduate courses in Computer Science as well as in a quite distinct field.
If new faculty taught one course in the new multimedia program, and one in their "home" department, (most likely a PhD level course), this implies four faculty form the core of the Center's multimedia teaching. This imposes some additional load on CS as well as one some other departments. Several adjuncts may be necessary as well. The project requirement would impose quite a load on the four faculty; however we expect that existing CS faculty in graphics, and industrial sources of projects that would need only minimal faculty involvement, would lighten this a bit. Assuming 25 -30 graduates a year, the budget is outlined below.
The goal of encouraging educational use of this technology by NYU Departments is approached by building active student involvement of projects, generally to involve faculty from departments other Computer Science, into the degree curriculum. This curriculum assumes 12 courses of 3 credits each, and is as follows:
The last three offerings shown here for the nominal third semester give only one possible student focus. The student will also have the option in this semester to adopt alternate areas of focus. Options can come from courses in ITP, TSOA; TSOA animation and graphic design; Department of Music; School of Business (Multimedia Industry and Policy); School of Education (Music Technology, Educational Multimedia and Multimedia pedagogy.)
The student body to be enrolled would center on persons with undergraduate degrees (or minors) and/or professional experience in programming, but would not be limited to such persons. Students without adequate computer background will be required to take the programming preparatory graduate course already offered by the CS department, in addition to the standard MS degree courses.
A Center approach affords more flexibility. For example, the Center could hire an artist, or graphics designer; something not likely to happen in a CS dept. Conceptually, any significant multimedia effort would need to be quite interdisciplinary. Yet because the focus is on emerging technologies (as opposed to the design focus of ITP), Computer Science faculty would still need to play a central role. We envision a pairwise collaboration between faculty from at least TSOA, Stern, Wagner, School of Ed, and the science departments on the one hand, and CS faculty on the other hand. An MS program, outlined roughly below, fits more into this interdisciplinary mold. One way that such a Center could draw in other parts of NYU is by having faculty in other depts. apply for release time to spend at the Center, to develop multimedia tools for their specialities. This would also provide a mechanism for MS graduate students to find projects.
Drawbacks include the difficulty of hiring junior tenure-track faculty into such a Center. Where would they get tenure? What would the draw be for a new faculty member to set up an MS program? Where would doctoral students come from? A solution to this problem may be to have Center faculty also have a home department where half their teaching is done, in addition to the Multimedia teaching of the Center. There is precedent for this at NYU. For example, all courses the Undergraduate Women's Studies department are cross-listed with other departments, and each of its faculty members is jointly appointed with another department.
While in principle the idea of jointly appointed faculty solves one problem, it may be problematic for tenure track faculty to have two homes. Hiring a senior person will not ultimately solve this problem either. While the right senior person would be ideal for establishing such a Center, we expect that most new faculty hires would still be at the junior level. In fact, the field is so new that there are not many senior people in it, even including people from more established areas such as graphics. Therefore, such people are in high demand, and there would be very substantial startup costs for senior hires in this area.
Michael Black is currently the manager of the multimedia group at Xerox PARC. He specializes in human motion analysis and its application to human-computer interaction and multimedia. We are also looking at Christoph Bregler, who is about to receive his PhD from Berkeley; he also works on human motion analysis, as well as on integrating speech and graphics. This latter work shows considerable promise for facilitating multilingual lip-synching and facial animation. Their resumes are attached. Both are interested in having practical impact, which is likely to lead to interaction with indviduals outside the department. Both candidates are interviewing at top-five departments such as Stanford and MIT. We are also looking at Dennis Korn, a top candidate nationally in the databases area.
Ken Perlin in CS is well known for his interdisciplinary interests. Most recently, he has been working with a class of animation students, in collaboration with Peter Weishar of the Animation Area in the Department of Film and Televiion at TSOA. Ken brings his awareness of technical possibilities and both advisors bring their interest and experience in animation. Ken also works with Philip Benfey in the Department of Biology. They are using the Pad multiscale interface as an interface paradigm for an on-line virtual laboratory and teaching tool. Davi Geiger in CS, who works in Computer Vision, has been collaborating with Bob Shapley of the Center for Neural Science; he has also been involved in discussions with people involved in Medical Imaging in the Medical School. In another direction, the user interface Pad++, developed by Ken Perlin and his associates, is being deployed in David Schwartz's Optical Mapping project in order to better view DNA data (this is being done by Bud Mishra in CS and a Computer Science doctoral student). Chee Yap, another CS faculty member, has recently begun a new project in this area in "Active Visualization", based on the Internet. The Natural Language project Proteus, by Ralph Grishman in CS, is turning its interest to speech, another major focus of computer-human interfaces and an important part of multimedia activity. As another example, an important activity is arising due to Clinical faculty member Arthur Goldbergs Projects course. This course places MS students in projects in local industry. This semester, for the first time, two of these projects are with companies in Silicon Alley. (This is a model for some activities we would like to see develop further). In this work, Arthur has been collaborating with Ajit Kambil in the Information Sciences department at Stern.
It seems clear that if there were more opportunities for interaction, with more faculty and staff to pursue and encourage these interactions, other departments would express even greater interest in using multimedia in their own activities.