For most of us our first encounters with the multimedia literacies of science comes in school. School science and its texts are not examples of the genres of professional science, not even in the advanced secondary school curriculum or in most of tertiary education (Lemke 1994, Roth et al. 1999). They do however initiate students into its multimedia literacy demands (Lemke 2000). Science textbooks contain not just words in sentences and paragraphs, but tables, charts, diagrams, graphs, maps, drawings, photographs and a host of specialized visual representations from acoustical sonograms to chromatography strips and gene maps. In many cases they also contain mathematical formulas and algebraic derivations.
But it is not just the print materials which make these demands. In a recent analysis of videotape data following one student through a day of advanced chemistry and physics classes (Lemke 2000, see also Cumming & Wyatt-Smith 1997), I observed that in his chemistry lesson this student had to interpret a stream of rapid verbal English from his teacher; the writing and layout information on an overhead transparency; writing, layout, diagrams, chemical symbols and mathematical formulas in the open textbook in front of him; the display on his handheld calculator; more writing, layout, diagrams, symbolic notations, and mathematics in his personal notebook; observations of gestures and blackboard diagrams and writing by the teacher; observations of the actions and speech of other students, including their manipulation of demonstration apparatus, and the running by-play commentary of his next-seat neighbor. In fact he had quite often to integrate and co-ordinate most of these either simultaneously or within a span of a few minutes. There is no way he could have kept up with the content development and conceptual flow of these lessons without integrating at least a few of these different literacy modes almost constantly.
In one episode in the physics lesson, there is no role for the notebook, and not even a diagram, but a pure interaction of language and gestural pantomime, including whole-body motion. The teacher, Mr. Phillips, is standing just in front of the first (empty) row of student desktables, at the opposite end of the room from where the student, John, is sitting. John sees his teacher’s hands cupped together to form a sphere, then the hands move a foot to the left and cup together to make another sphere. Then back to the first, and one hand and Mr. Phillips' gaze make a sweeping gesture from one to the other; then Mr. Phillips begins to walk to the left, repeating these gestures and walking down toward John's end of the room. Fortunately, Mr. Phillips is also talking and John is not deaf; by integrating the teacher’s precise and conventionalized mime with his accompanying technical speech, John can interpret that the cupped hands are atoms, the sweeping hand a photon, emitted by the first, traveling to the second, absorbed there, re-emitted after a while, passing on down through a ruby crystal, producing a "snowball effect" of more and more photons of exactly the same energy. In other words, the crystal is a laser.
Mr. Phillips says he's going to add more complexity to the picture now. An atom "might shoot out a photon in this direction" -- gesture away from the axis of the room-sized imaginary ruby crystal toward the students -- "or in this one" -- gesture back toward the blackboard -- "or ..." -- oblique gesture. How do we get a laser beam then? He walks back and forth between the ends of his now lasing, imaginary ruby crystal, describing the mirrors he gestures into being at each end, but saying they differ in reflectivity and transmissivity, to build up and maintain the avalanche of photons, while letting some out in the form of the laser beam. John has seen mimes like this before; he has seen diagrams of atoms and crystals, of photons being absorbed and emitted by atoms. Intertextually, he can use the visual literacy of these past diagrams, together with his literacy in pantomime, and his verbal discourse literacy in atomic physics to synthesize a model of how a laser works.
John is lucky. He does appear to have the required literacies, and to be able to combine and synthesize them across media, events, and semiotic modalities. There is a great deal that John must already know in order to make sense of what he is learning in these lessons minute to minute. Not just language and verbally expressed discourse formations (such as the intertextual thematic formations I have described in Lemke 1983, 1995 and elsewhere), but conventional diagrams of atomic arrangements in a crystal, standard graphs of energy levels of atoms, typical ways of gesturing directionality in space, and common notations for the algebraic and symbolic representation of chemical reactions and stoichiometric calculations of concentrations and the pH of solutions. His literacy extends to motor routines in operating a calculator, social discourse routines of question and answer in a classroom, and technical practices in manipulating a spectroscope and diluting a solution. He must constantly translate information from one modality to another: numerical to algebraic, algebraic to graphical, graphical to verbal, verbal to motor, pantomime to diagrammatic, diagrammatic to discursive. But simple translation is not enough; he must be able to integrate multiple media simultaneously to re-interpret and re-contextualize information in one channel in relation to that in the other channels, all in order to infer the correct or canonical meaning on which he will be tested. In most cases, the complete meaning is not expressed in any one channel, but only in two or more, or even only in all of them taken together (see detailed examples in Lemke 2000, Roth & Bowen 2000, Wells 2000).
Even if we restrict our attention to the interpretation of printed text, scientific genres remain highly multimodal.