J.L.Lemke On-line Office

Hypertext Semantics

 The fundamental issue that I want to explore in this paper is how multimodal semiotics interacts with hypertext semantics to produce the semiotic affordances of hypermedia. 

What is unique about hypertext is that it is multiply connected by design; that is, (a) the readerís sequential pathway through the meaningful units (on many scales) of the hypertext web cannot be predicted by the author/designer and (b) the author/designer normally plans for potential meaning construction along many possible reading sequences. (See Appendix for a more detailed analysis of the hypertext medium.) This changes the relationships among author(s), work/web, and user(s) in a variety of ways. From oral speech, across the various genres of written text, to hypertext webs, there is a wide range of balance struck between imposition of sequence by the author and improvisation of sequence by the reader. An author may create a work whose value to most readers depends on their reading it in a single author-determined sequence. Or authors may permit or encourage readers to create their own sequences through the meaningful units of the work. Every text, no matter how it may seem only to offer information to a reader, always also imposes upon and makes demands of the reader in respect of meaning. It seeks to manipulate, constrain, and control the meanings made by the reader at each textscale, including the actions taken by the reader in operating  the technology of the medium.

 In most uses of the hypertext medium, there are severe limits on the useful degree of control of reader-constructed sequences, so that designers must anticipate many possible moves by the reader, and the combinatorial explosion of these rapidly passes beyond the capability of most designers, even over timescales very long compared to those over which readers construct traversals, to anticipate more than the most likely sequences. So hypertext, while it can work just like other more closed media, has its unique potential as an open medium. Matching the degree of openness to the anticipated uses of the web is a principal design challenge. Exploring the possible uses of the most open webs is a challenge for us all. 

By the semantics of hypertext, I mean the affordances of the hypertext medium for constructing meaning-relationships along traversals. This is the analogue of what might be called longer-scale text semantics in the case of more conventional verbal media. Just as we make meanings across many paragraphs or chapters that we do not make within a single paragraph or chapter, so we can make meanings in hypertext along long traversals (across say 10, 30, 100 or more units or lexias ) that are not made in any one lexia or even across links between two lexias. We know relatively little about long-scale conventional text semantics (Lemke, in press-c). I believe that on intermediate scales (dozens of paragraphs, say) we know two basic things: (1) that meanings are made through the nested embedding of multivariate structures on different scales, particulary genre structures and extended rhetorical-argumentative structures, and (2) that meanings are also made through extended cohesion chains and cohesive harmonies that are orchestrated by their co-distribution patterns through a text and by their occasional intersection in a multivariate nexus at a shorter text scale (Lemke 1995b). 

The first type of extended-text meaning is not easily made in hypertext webs that are rich in interconnection. It is possible to make use of genre-like structures, but difficult to hierarchically organize them sequentially. Cohesion chains, on the other hand, which are based on relations of similarity of units across extended text, work equally well in hypertext. I have argued elsewhere (Lemke 1995b) that thematic formations are the product of either intra- or inter-textual cumulations of meanings interpreted according to both these principles. It is also commonly said of long narrative fictions that their distinguishing meaning affordance is the building of convincingly detailed fictional universes, ones in which there is enough detail to provide explanation and motivation for long sequences of fictional events among fictional characters. One might say the same even of non-fiction histories or autobiographical narratives. In such cases the exact sequencing does not seem to matter, and this is the basis of much hypertext fiction (e.g. Michael Joyce's Afternoon), where global hypertext meaning is made progressively and cumulatively, but most fully only retrospectively. We encounter, as in a real-life exploration or mystery, one clue after another to help us build a consistent understanding (or several alternative theories) of events, and we cumulate and revise the overall pattern as we go. 

It is less easy to create compelling logical arguments (cf. Kolb 1997) or coerce the reader to agreement on matters of analysis and interpretation, but it is still possible to raise for the user of a (branching, multi-sequential) hypertext web a great many interrelated questions that perhaps better reflect the complexity of the real. One may offer multiple perspectives, and indeed accommodate multiple authorship of the web, without loss of usefulness for such purposes. 

I am not going to attempt here the task of developing a general, long-scale hypertext semantics, for either textual or multimodal hypermedia. I want only to sketch out some foundational notions and then see what happens in the webpage traversal examples I will provide. We need to begin, however, withat least an inventory of the shortest-scale cross-link meaning relations. 

What kinds of meaning relations are typically construed across a single hypertext link? Such a link provides what is essentially an intertextual meaning relation, and we know (Lemke 1995b) that the kinds of meaning relations made between texts include those made within texts on longer scales. We also know that within texts there is a certain scale-invariance of meaning relations from the clause complex (Halliday 1994) to the rhetorical formation (Mann & Thompson 1986) and beyond. These basic relations are specializations of the semantic relations of Expansion and Projection between clauses (Halliday 1994, chap 7). A clause is a complete semiotic unit, it provides a construal of what we ordinarily consider to be an extra-linguistic state of affairs (though in fact we make sense of experience and construct what we call knowledge of the world largely though the semantic affordances of natural language). Such verbal construals may have only certain semantic relations to one another. I believe that as a first approximation the kinds of relations construed between consecutive webpages or hypertext lexias are these: 

Presentational:

                     Links which tie one topic-specific set of semantic relationships to another in the same way such sets are internally connected (Cf. thematic formations, Lemke 1983, 1995b); e.g. activity-to-actors, object-to-qualities, event-to-manner, activities linked by common actors and vice versa, etc.

                    logical relations of expansion and projection: restatement, specification, exemplification, commentary; addition, exception, alternative; conditionality, causality, contextualization; quotation, opinion (Halliday 1994, chap. 7)

                    rhetorical relations which further specify the logical relations, such as concession, opposition, disjunction, problem-solution, cause-consequence, proposal-evidence, events-generalization, etc. (Mann & Thompson 1986)

 Orientational: 

                    Offer & response (accept, consider, demur, decline, reject, counter-offer); Demand & response (comply, refuse. etc.); more generally Offer/Demand -information, -action & response; Degrees of Offer/Demand: entice, suggest, propose, insist, etc. (cf. Halliday 1994, chap. 4)

                    State-of-affairs/Evaluation (warrant, desire, importance, normativity, usuality, comprehend, humor); evaluative propagation chain elements; heteroglossic alliance/opposition (Lemke 1998a)

 Organizational:

                     Functional relations among the elements of such structures as: nominal group, clause, complex, rhetorical formation, genre (Halliday 1994, Martin 1992)

                    Covariate chain element: similarity chain, co-hyponymic chain, co-meronymic chain; based on presentational or orientational features as above (Halliday & Hasan 1976, Lemke  1995)

 As we proceed from lexia to lexia along a traversal, each linked pair may have many of the above relations. Coherence is achieved along a traversal, at least retrospectively, by consistency of patterning, thematically and orientationally, with the long-range cumulations depending primarily on heteroglossic formations: thematic-cum-evaluative stance, singly or in systems of alliance, alternative, opposition, as seen from one or from many discrete meta-viewpoints. This is a complex subject in itself, even for mono-sequentially designed texts (see Lemke 1988, 1995, in press-c). In simplest terms, a heteroglossic formation is an identifiable, socially-positioned discursive viewpoint or way-of-speaking about something, including both how we construe presentationally and what our evaluative stance is. Along a traversal we are sensitive to consistency of such viewpoints and to systematic relationships built among viewpoints (whether presented as objective, as the author(s) own, or attributed to the reader or to others).

 For each of the general classes of semantic connections listed above, there are corresponding visual principles and forms. In fact, many of the verbal relations can be conceptualized as visual metaphors (e.g. chains, multi-slot structures, narrative scenes, viewpoints) . Nonverbal visual works of comparable complexity and scale show all these features, as for example in rich, traversable visual environments (e.g. architectural spaces, artist-exhibition spaces; designed landscapes, online gaming worlds), dynamic visual displays (e.g. silent films, animations, auto-scrolling displays, theatrical and dance performances), and extended static visual series or sequences (cartoon books, graphic novels, ukiyo-e print collections, long scroll paintings). 

 What happens when we combine textual and pictorial-graphical resources across longer scales? For example, in dialogue-scripted or narrated film, gaming worlds with dialogue and embedded texts; hypermedia genres (e.g. CD-ROM encyclopedias) and, most commonly today, webpages and websites.

 I would like to illustrate some of the principles I have put forward here regarding multimodality and hypertext semantics at least for the short-range scale of single pages, page-pairs across single links, and a very short traversals of webpages. I draw on my experience of having analyzed these instances in the context of much longer traversals (e.g. Lemke, in press-a).