The fundamental issue that I want to explore in this
paper is how multimodal semiotics interacts with hypertext semantics to produce
the semiotic affordances of hypermedia.
What is unique about hypertext is that it is multiply connected by design; that is, (a) the reader’s sequential pathway through the meaningful units (on many scales) of the hypertext web cannot be predicted by the author/designer and (b) the author/designer normally plans for potential meaning construction along many possible reading sequences. (See Appendix for a more detailed analysis of the hypertext medium.) This changes the relationships among author(s), work/web, and user(s) in a variety of ways. From oral speech, across the various genres of written text, to hypertext webs, there is a wide range of balance struck between imposition of sequence by the author and improvisation of sequence by the reader. An author may create a work whose value to most readers depends on their reading it in a single author-determined sequence. Or authors may permit or encourage readers to create their own sequences through the meaningful units of the work. Every text, no matter how it may seem only to offer information to a reader, always also imposes upon and makes demands of the reader in respect of meaning. It seeks to manipulate, constrain, and control the meanings made by the reader at each textscale, including the actions taken by the reader in operating the technology of the medium.
In most uses of the hypertext medium, there are
severe limits on the useful degree of control of reader-constructed sequences,
so that designers must anticipate many possible moves by the reader, and the
combinatorial explosion of these rapidly passes beyond the capability of most
designers, even over timescales very long compared to those over which readers
construct traversals, to anticipate more than the most likely sequences. So
hypertext, while it can work just like other more closed media, has its unique
potential as an open medium. Matching the degree of openness to the anticipated
uses of the web is a principal design challenge. Exploring the possible uses of
the most open webs is a challenge for us all.
By the semantics of hypertext, I mean the affordances of
the hypertext medium for constructing meaning-relationships along traversals.
This is the analogue of what might be called longer-scale text semantics in the
case of more conventional verbal media. Just as we make meanings across many
paragraphs or chapters that we do not make within a single paragraph or chapter,
so we can make meanings in hypertext along long traversals (across say 10, 30,
100 or more units or lexias ) that are not made in any one lexia or even
across links between two lexias. We know relatively little about long-scale
conventional text semantics (Lemke, in press-c). I believe that on intermediate
scales (dozens of paragraphs, say) we know two basic things: (1) that meanings
are made through the nested embedding of multivariate structures on different
scales, particulary genre structures and extended rhetorical-argumentative
structures, and (2) that meanings are also made through extended cohesion chains
and cohesive harmonies that are orchestrated by their co-distribution patterns
through a text and by their occasional intersection in a multivariate nexus at a
shorter text scale (Lemke 1995b).
The first type of extended-text meaning is not easily made
in hypertext webs that are rich in interconnection. It is possible to make use
of genre-like structures, but difficult to hierarchically organize them
sequentially. Cohesion chains, on the other hand, which are based on relations
of similarity of units across extended text, work equally well in hypertext. I
have argued elsewhere (Lemke 1995b) that thematic formations are the product of
either intra- or inter-textual cumulations of meanings interpreted according to
both these principles. It is also commonly said of long narrative fictions that
their distinguishing meaning affordance is the building of convincingly detailed
fictional universes, ones in which there is enough detail to provide explanation
and motivation for long sequences of fictional events among fictional
characters. One might say the same even of non-fiction histories or
autobiographical narratives. In such cases the exact sequencing does not seem to
matter, and this is the basis of much hypertext fiction (e.g. Michael Joyce's Afternoon), where
global hypertext meaning is made progressively and cumulatively, but most fully
only retrospectively. We encounter, as in a real-life exploration or mystery,
one clue after another to help us build a consistent understanding (or several
alternative theories) of events, and we cumulate and revise the overall pattern
as we go.
It is less easy to create compelling logical arguments (cf.
Kolb 1997) or coerce the reader to agreement on matters of analysis and
interpretation, but it is still possible to raise for the user of a (branching,
multi-sequential) hypertext web a great many interrelated questions that perhaps
better reflect the complexity of the real. One may offer multiple perspectives,
and indeed accommodate multiple authorship of the web, without loss of
usefulness for such purposes.
I am not going to attempt here the task of developing a
general, long-scale hypertext semantics, for either textual or multimodal
hypermedia. I want only to sketch out some foundational notions and then see
what happens in the webpage traversal examples I will provide. We need to begin,
however, withat least an inventory of the shortest-scale cross-link meaning
relations.
What kinds of meaning relations are typically construed
across a single hypertext link? Such a link provides what is essentially an
intertextual meaning relation, and we know (Lemke 1995b) that the kinds of
meaning relations made between texts include those made within texts on longer
scales. We also know that within texts there is a certain scale-invariance of
meaning relations from the clause complex (Halliday 1994) to the rhetorical
formation (Mann & Thompson 1986) and beyond. These basic relations are
specializations of the semantic relations of Expansion and Projection between
clauses (Halliday 1994, chap 7). A clause is a complete semiotic unit, it
provides a construal of what we ordinarily consider to be an extra-linguistic
state of affairs (though in fact we make sense of experience and construct what
we call knowledge of the world largely though the semantic affordances of
natural language). Such verbal construals may have only certain semantic
relations to one another. I believe that as a first approximation the kinds of
relations construed between consecutive webpages or hypertext lexias are these:
Presentational:
· Links which tie one topic-specific set of semantic relationships to another in the same way such sets are internally connected (Cf. thematic formations, Lemke 1983, 1995b); e.g. activity-to-actors, object-to-qualities, event-to-manner, activities linked by common actors and vice versa, etc.
· logical relations of expansion and projection: restatement, specification, exemplification, commentary; addition, exception, alternative; conditionality, causality, contextualization; quotation, opinion (Halliday 1994, chap. 7)
· rhetorical relations which further specify the logical relations, such as concession, opposition, disjunction, problem-solution, cause-consequence, proposal-evidence, events-generalization, etc. (Mann & Thompson 1986)
Orientational:
·
Offer & response (accept, consider, demur, decline, reject,
counter-offer); Demand & response (comply, refuse. etc.); more generally
Offer/Demand -information, -action & response; Degrees of Offer/Demand:
entice, suggest, propose, insist, etc. (cf. Halliday 1994, chap. 4)
· State-of-affairs/Evaluation (warrant, desire, importance, normativity, usuality, comprehend, humor); evaluative propagation chain elements; heteroglossic alliance/opposition (Lemke 1998a)
Organizational:
·
Functional relations among the elements of such structures as:
nominal group, clause, complex, rhetorical formation, genre (Halliday 1994,
Martin 1992)
· Covariate chain element: similarity chain, co-hyponymic chain, co-meronymic chain; based on presentational or orientational features as above (Halliday & Hasan 1976, Lemke 1995)
As we proceed from lexia to lexia along a traversal, each linked pair may have many of the above relations. Coherence is achieved along a traversal, at least retrospectively, by consistency of patterning, thematically and orientationally, with the long-range cumulations depending primarily on heteroglossic formations: thematic-cum-evaluative stance, singly or in systems of alliance, alternative, opposition, as seen from one or from many discrete meta-viewpoints. This is a complex subject in itself, even for mono-sequentially designed texts (see Lemke 1988, 1995, in press-c). In simplest terms, a heteroglossic formation is an identifiable, socially-positioned discursive viewpoint or way-of-speaking about something, including both how we construe presentationally and what our evaluative stance is. Along a traversal we are sensitive to consistency of such viewpoints and to systematic relationships built among viewpoints (whether presented as objective, as the author(s) own, or attributed to the reader or to others).
For each of the general classes of semantic connections listed above, there are corresponding visual principles and forms. In fact, many of the verbal relations can be conceptualized as visual metaphors (e.g. chains, multi-slot structures, narrative scenes, viewpoints) . Nonverbal visual works of comparable complexity and scale show all these features, as for example in rich, traversable visual environments (e.g. architectural spaces, artist-exhibition spaces; designed landscapes, online gaming worlds), dynamic visual displays (e.g. silent films, animations, auto-scrolling displays, theatrical and dance performances), and extended static visual series or sequences (cartoon books, graphic novels, ukiyo-e print collections, long scroll paintings).
What happens when we combine textual and pictorial-graphical resources across longer scales? For example, in dialogue-scripted or narrated film, gaming worlds with dialogue and embedded texts; hypermedia genres (e.g. CD-ROM encyclopedias) and, most commonly today, webpages and websites.
I would like to illustrate some of the principles I have put forward here regarding multimodality and hypertext semantics at least for the short-range scale of single pages, page-pairs across single links, and a very short traversals of webpages. I draw on my experience of having analyzed these instances in the context of much longer traversals (e.g. Lemke, in press-a).