J.L.Lemke On-line Office

 The GSFC Gateway Web

Let’s begin with a fairly rich and complex page from the domain of the U.S. National Aeronautics and Space administration (NASA) in a website at the Goddard Space Flight Center (GSFC): 

GSFC Earth Sciences (GES) Distributed Active Archive Center




This page is a gateway to data collected by NASA sattelites which record the spectrum of radiation reflected from the earth. This data can be interpreted as giving measures of the temperature, density of atmospheric ozone, degree of vegetation cover, etc. all over the earth, as seen from space. The primary content of the page is a menu of seven links to such datasets with brief textual descriptions and associated color images showing one important application of each type of data. This is a dynamic webpage: when the mouse is moved over the links (but not clicked), a new color image and caption text appear for each link. 

The first link anchor is: 

Atmospheric Chemistry   Ozone and other trace gas compositions, dynamics and energy interactions of the upper atmosphere.  

The accompanying image is a polar-view globe map with a few dotted longitude and latitude lines and is colored to represent data values. Below is a spectrum key, with color spectrum labeled in hundreds of Dobson units. The caption text has a header, Ozone Depletion, and reads: 

“TOMS data represents a long term, continuous record of information used to monitor global and regional trends in ozone. The image above shows the October 1996 Antarctic ozone hole as viewed by the TOMS ADEOS instrument.” 

The second link anchor in the menu is: 

Atmospheric Dynamics   3-dimensional dynamic and thermodynamic state of the earth-atmosphere system, from satellite measurements and assimilation systems. 

The image in this case shows  flow patterns of atmospheric and oceanic ‘currents’ against a land-ocean topography. Flow lines are represented by arrows forming a vector field.

Let’s consider the page as a whole and the dynamic images and captions for these two links. I will present a brief analysis of the textual-linguistic and visual-semiotic features of the text and their integrative interrelationships in terms of the three generalized semiotic functions common to both: presentational, orientational, and organizational.

(Please note that the page is occasionally updated; for example recent versions have added a warning note about improper use or access, i.e. hacking.) 

It is often useful to begin a multimodal analysis with the visual-organizational composition of the whole page, because, as Arnheim noted, this is the salient structure which guides the eye in its traversals across the page. 

There is a top frame with NASA logo at left, title in large serif type, below the title a motto in small quoted italics, at right an iconlink for What’s New (a colored globe with superimposed text, similar in size to the NASA logo icon at left). Below these is a single thin blue navigation bar (probably an image map), dividing the top matter from the page content. Below the bar is a graphical link for highlighted information on a new sattelite instrument (MODIS). It is a rectangle with a bluesky-and-clouds background overprinted with large and smaller type. 

The main content of the page, all of which has a uniform sky-blue background, consists of the menu of textual-phrase link anchors noted above, each of which is both a link and an active mouse-over site, and each of which is followed by a descriptive gloss. 

Most interesting are the results of the on-mouse-over script, which place in a frame at right a complex graphical image and below it, as a caption, a short explanatory text. These image-text pairs illustrate the kinds of data to be found under the header link.

There is no explanation on the page of the procedure for displaying different images. (Note that the caption texts are themselves separate image files). 

At the bottom of the page is another navigation bar with different choices from the topbar, and the usual NASA bottom matter, including contact information, responsible people, and modification date in small italic type. 

The overall organization here is very much like a typical printed page of text: horizontally aligned paragraph-like units of text, stacked vertically down the virtual page. The deviations from this are the two globe-shaped icons at top left and right, and the inset dynamic images with their caption text below them in a separate “column” to the right of the menu of textual links. You cannot normally see the whole length of the page at once, but you do initially see the first of the dynamic images. The scrollbar at right indicates that there is more content on the page, below. It defines a vertical unoriented vector (up or down); this as well as the left-aligned edges of the textlinks, menu bars, and MODIS rectangle, and the conventional reading orientation (down the page, especially for the caption text), impose a vertical ordering on a page dominated visually by discrete horizontal elements. 

The page has a compositional unity deriving from this horizontal-and-vertical-grid structure, and this  is reinforced by the uniform sky-blue background and the blue color tones of the text, menu bars, NASA logo, What’s New globe icon, and MODIS rectangle. This uniformity throws into salient relief the non-blue colors of the dynamic images. The first, titled Ozone Depletion, shows yellow as its primary color, with secondary green, central pink and white, a lesser area of dark red and a tiny bit of sky blue. All these colors also appear in a color-bar below the image, but in extremely small tokens. This color contrast sets off the dynamic image from everything else on the page, including its own caption which is alike in color but slightly larger in typeface. The eye is strongly drawn to the dynamic images, and as the user discovers that the images change with movement of the mouse down the textlink menu, there is set up another dynamic intersection of horizontal shifts of attention back and forth between the textlinks and the images with the vertical up and down exploration of the page’s affordances and returns to previously seen images. 

Each of the other dynamic images also includes an element of sky-blue, and there is both a sort of tying together (visual cohesive tie) and an alternation of the eye back and forth between these blue elements and the dominant blue of the rest of the page. The blue elements in the images take on a special prominence because of the color context of the rest of the page.



The use of color to produce a cohesive tie which has a definite semiotic significance is clear in the Monsoon Dynamics image (associated with the second textlink). Here, apart from the use of blue to indicate two of the most thematically important elements (the blue ribbons for airmass drift and the blue arrows for wind movement), there are two yellow ribbons which are not otherwise visually connected to one another, but which are meant to represent movements of air masses with some similar (not noted) property, in contrast to those represented in blue. 

The twin organizing principles of both text and visual images are (1) structural and (2) cohesive. The former is the familiar principle of wholes and parts (e.g. lists, figure-and-caption); the latter that of similarities of salient features (e.g. color, or typeface). 

The textual unitshere are structurally organized into headers or titles or link-names and explanatory extended nominal groups. The structural relation is Halliday’s Expansion: Elaboration, here realized paratactically (Halliday 1994). The cohesive organization of the list is based on co-hyponymy (Halliday & Hasan 1976), or more generally co-thematic collocations, with an intertextual basis (knowing the earth system division into lithosphere, hydrosphere, and atmosphere, supplemented by the cross-cutting biosphere). 

There are also other very direct ties between principal elements. For example, the first two textlinks are “Atmospheric Chemistry” and “Atmospheric Dynamics”, corresponding closely to the thematic complementarity of chemistry and physics. The former is cohesive with “ozone” in its image’s title, the latter with “dynamics” repeated in its image’s title. These lexical semantic ties reinforce and are reinforced by the visual ties noted above, and by the interactive connection of the dynamic change of images with movement of the mouse over the vertically stacked textlinks. 

The cohesive linguistic ties, whose function here is organizational, are of course also functioning presentationally (thematically) to help construct the semantic relationship between the words of the textlink and its elaboration on the one hand and the thematic meaning of the image caption on the other. There is a very high degree of lexical cohesion of the strongest type (word repetition, identity chaining, reference chaining) between these, e.g. in the case of the “Monsoon Dynamics”. The overall semantic relationship is one of re-statement, with some specification or subcategorization, and particularly the semantic relationship of Exemplification (dynamics: monsoon: India). 

How is the presentational semiotic link built between the textlink phrases and the visual image itself? First, there is the supporting organizational connection, implying that there is a presentational or thematic relationship. Thus the lexical identity from the linktext to the caption title (“dynamics”) combines with the visual organizational convention of the genre (structural: caption to figure), to connect to the image as such. When we interpret the image thematically, we do so in relation (organizational connection) to the caption text. We identify “flow patterns” “trajectories … yellow and blue ribbons” and “wind vectors” visually in the image. The other element in the image, inter-pictorially recognizable as mountains or topographic relief of some sort, cohesively links back to the elaboration of the textlink at left: “dynamic … state of the earth-atmosphere system”, in which the other elements are “atmosphere” and the mountains “earth”. 

But what of the orientational meanings here? How do the verbal and visual signs combine to produce (a) a stance to the viewer/user, and (b) a stance toward the thematic content presented? 

The page as a whole has an explicit orientation: “Your source for Earth Science data and information” in italics just below the page title. The second-person address here defines a Source-User relationship and makes a proposal: offer (Halliday 1994). But the page also makes demands, most clearly that you use the mouse to reveal the images and captions, and also that you click on one of the links to proceed to those and only those pages the designer has provided links to (apart from the default Back operation, or options afforded by the browser rather the page). The sudden, and unexpected, appearance of the Monsoon image at right clearly makes a demand on the user’s attention. These are demands for action, not for information. (We will see an example of the latter in the Earth Observatory website discussed below.) The caption text, when it refers to visual elements in the image (“yellow … ribbons”) also makes an implicit demand that we match these to visual elements. 

Semantically, the textlink and its elaboration and the caption text present pure offers of information. As such we should consider the putative source’s stance toward this information: its warrantability (high, there is no qualification of probability), its desirability (unmarked, except that monsoons may be considered dangerous, and so information about them useful and desirable), its normativity (no proposal, so no issue of obligation), its usuality (monsoons are expectable, the conventions of scientific imaging of the data presented also as of unmarked usuality), importance (monsoons are important events, information about them presumably also), comprehensibility (high for the source, the data and image help us understand monsoons, but maybe confusing or mysterious for the user, depending on expertise and familiarity with these terms and conventions), and seriousness (unmarked, high for this genre and register). 

Visually, we may ask how the image itself expresses such stances. 

Warrantability: this is not a photorealistic image, it does not have high visual resolution or realistic detail. It is an abstract representation, which we infer (from the thematic content of the caption and intertextual knowledge) is based on sattelite data and faithful to that data, which would make the warrantability high, even though purely by visual conventions it might not seem so (Kress & van Leeuwen 1996). We can make comparisons with the other dynamic images. In the Ozone Depletion image, also abstract, there is a quantitative coding that adds a high degree of warrantability, and this coding is presented visually and numerically (for a similar instance, see the NDVI page analysis below). The image represents point for point real scientific data. It is not just an imagined scenario. The “Interdisciplinary” link has an associated image “Global Warming”, which shows a data graph that again has very high warrantability in the conventions of its genre. (Lower warrantability is shown by error-bars on data points, or indications that graphed lines and points correspond to theory or estimates rather than actual data.) In the Monsoon image we might still have some doubt about the exact contours of the landscape shown, because conventionally such images abstract and simplify. The text-warrant in the caption is only for the accuracy of the ‘flow patterns’, leaving this issue open. 

Desirability: nothing in the image itself shows a “monsoon” as a dangerous storm or flood-causing rain, except for the choice of a black background for the sky. This choice is presumably purely conventional, not dictated by the conventions of scientific representation of data on air flows. I think we can plausibly interpret it as a marker of the dangerous or serious or undesirable nature of the monsoon, and so indirectly also as reinforcing the sense of the importance of havingNASA data to understand it. 

Normativity: there does not seem to be an “ought to” element in this image, though there is a demand of sorts, that the eye follow the arrows (there are so many of them, almost all aligned in the same direction). The other orientational stance to the viewer here is signaled by the overhead view: that we see this as from above (from the sattelite, but much closer up), with a “God’s eye” view, which is culturally associated both with warrantability (from on high you see more of what is) and with science (an objective view from above the fray, of what is true). So the Source:User relationship is being visually constructed here as one of a giver of information to one who needs this information, one who is in a position to know, to one who is not but needs the knowledge. One who can guide understanding to one who needs to follow that guidance (e.g. the image’s arrows, but also the links themselves). So there is a kind of normativity here, that the user ought to trust and use the knowledge the source offers. 

Usuality: this is not a very conventional image for most viewers, though it may be for specialists, and it is not so unusual for anyone with scientific training. Within the image itself, there are non-naturalistic features in contrast with the naturalistic ones, mainly the ribbons and arrows, which are the features explained in the caption. The fact that they need explanation would seem to indicate that the source regards them as at least to some degree unusual. In this respect usuality is linked to Comprehensibility: what is unusual is also often less comprehensible. The explanation makes these features more comprehensible, but cannot make them more usual. The degree of comprehensibility also tends to define the implied user: someone for whom the explanation given is adequate. 

Finally, Importance and Seriousness: the color choice in the Monsoon image is a palette that might be associated in other genres with a cartoon image, and so not fully serious. This is clearly not the case here, and seriousness is indicated in part by the many arrows, suggesting a systematic pattern, not a playful arrangement. How does the image show the importance of its content? Not, it seems, by attention to photorealistic detail, and if we compare with the other images, not by the use of numerical codes either. Importance here seems mainly to be conveyed verbally through the connection to “monsoon”, as it is also in an image that is titled “Ozone depletion” (intertextually marked as important in the community). In the case of the image labelled “Global Warming” we have the explicit verbal marker in the caption text: “A remarkable global warming trend …” which explicitly marks for low Usuality, and probably indirectly increases our sense of the importance of the data displayed.