Chen, C. and Rada, R. (1996) Interacting with hypertext: A meta-analysis of experimental studies. Human-Computer Interaction, 11(2), 125-156.

Interacting with Hypertext:
A Meta-Analysis of Experimental Studies
Chaomei Chen and Roy Rada
 
ABSTRACT

The meta-analysis compared and synthesized the results of 23 experimental studies on hypertext. The analysis was based on 56 pairs of effect sizes and significance levels of the impact of users, tasks, and tools on interactions with hypertext. This analysis focused on three factors that prevailingly influence the use of hypertext: 1) the cognitive styles and spatial ability of users, 2) the complexity of tasks, and 3) the structure of information organization and the visualization of the structure. The meta-analysis found that this group of experimental studies reported significantly discrepant findings, indicating that substantial differences exist among individual experiments. Individual differences in cognition did not yield enough evidence to conclude that the effect sizes are significantly apart from zero. The meta-analysis showed that the overall performance of hypertext users tended to be more effective than that of non-hypertext users, but the differences in efficiency measures were consistently in favor of non-hypertext users. Users benefited more from hypertext tools for open tasks. Overall, the complexity of tasks has the largest combined effect sizes. Graphical maps which visualize the organization of hypertext have significant impact on the usefulness of a hypertext system. This meta-analysis raised two issues concerned with the present hypertext literature: 1) the absence of a taxonomy of tasks for analyzing and comparing hypertext usability across studies and 2) the weaknesses of the connections between abstract hypertext reference models and specific hypertext systems. These weaknesses may considerably undermine the significance of individual findings on hypertext usability. Results of the meta-analysis suggest that the discrepancies among empirical findings are related to these weaknesses. Future work on hypertext usability should emphasize task taxonomies alone with longitudinal and ethnographic studies for a deep understanding of the interactions between users and hypertext. Recommended research issues for the future are highlighted in the conclusion.

 
 

  1. INTRODUCTION
  2. Several generations of hypertext systems have evolved over the last decade. Empirical issues associated with the use of these hypertext systems have been pursued in a broad range of laboratory experiments and field studies. A fundamental research question is whether the empirical findings reported in the literature suggest any invariant underlying relations (Hunter, Schmidt, & Jackson, 1982; Hunter & Schmidt, 1990).

    A pioneering survey of empirical findings on a broad range of hypertext usability issues was conducted by Nielsen (1989). The survey revealed several strong relationships between independent variables and dependent variables in relation to interacting with hypertext systems. Do these relationships generally hold for hypertext systems? What, if any, discrepancies exist among the reported empirical findings concerning the same relationship? Do the empirical findings in the literature suggest new relationships as hypertext systems become more powerful? This paper reports the results of a meta-analysis on experimental studies published between 1988 and 1993 that answer these questions in a quantitative way.

    Building on the work of Nielsen (1989), this meta-analysis studies the impact of users, tasks, and tools on users’ behavior in interactions with hypertext. In order to give an interpretive context for the results of the meta-analysis, this paper introduces a conceptual framework which is drawn on established principles and models of hypertext systems (Conklin, 1987; Halasz, 1988; Halasz & Schwartz, 1990; Halasz & Schwartz, 1994). Empirical implications of this framework are then stressed in a qualitative review of experimental studies. The major propositions to be investigated in this study are derived from this conceptual framework and qualitative review of experimental studies.

     

    1. Conceptual Framework
    2. The principles and models of hypertext systems were first reviewed in the landmark work of Conklin (1987). According to Conklin, the essence of hypertext is the structure of nodes and links as a medium of thinking and communication for users. Similarly, Nielsen noted that the purpose of hypertext is to provide users an open, exploratory information space (Nielsen, 1990). Conklin (1987) also identified the well-known problems of interacting with hypertext, particularly disorientation and cognitive overhead. Wright (1991) discussed the significance of these issues to the evaluation of hypertexts.

      Halasz (1988) reviewed some pressing issues that must be addressed in the design of forthcoming hypertext systems. These issues primarily arose from the experience with a workstation-based hypermedia system NoteCards (Trigg, Suchman, & Halasz, 1986). The provision of graphical overviews was regarded as one of the most essential features of hypertext systems to tackle the problems of disorientation and cognitive overhead. Halasz also highlighted the weaknesses of hypertext systems in supporting search tasks and suggested that hypertext systems need to incorporate search facilities to help users managing loosely structured information.

      A formal treatment of hypertext models was given by the Dexter Hypertext Reference Model (Halasz & Schwartz, 1994; Gronbaek & Trigg, 1994). The purpose of the Dexter model was to provide a reference framework so that one can judge whether or not a particular system has all the essentials to be a hypertext system. The Dexter model was also developed to help information interchange among hypertext systems. The Dexter model has yet to be fully used in the design of hypertext systems or in the design of experiments for formal evaluations.

      An essential way of interacting with hypertext is browsing. Marchionini and Shneiderman (1988) suggested several factors that could influence users’ browsing strategies, including 1) contextual settings, 2) information system, 3) task domain, 4) user, and 5) intermediate outcome of browsing. A pioneering survey of empirical findings concerning hypertext systems was conducted by Nielsen (1989) based on the results of 30 empirical studies. Nielsen surveyed a broad range of usability issues with hypertext and general information systems, including the effects of the medium of display on the speed of reading, the effects of different pointing devices on the speed of selecting highlighted fields on the screen, the effects of tasks on users’ performance, and the effects of users’ motivation on interactions. Nielsen found that the greatest magnitudes of the effects included in the survey are associated with users’ motivation, expertise for required tasks, and organizational structures of files.

      The conceptual framework of this study consists of users, tasks, and tools as contextual variables. The framework also includes effectiveness and efficiency as dependent variables. This study postulates that these three contextual variables have significant effects on the two aggregated dependent variables. Through this meta-analysis experimental studies should collectively reveal relationships between these three contextual variables and the dependent variables.

      By using a meta-analytical method, this study will be able to synthesize the empirical findings on a specific hypothesis of a relationship between an independent variable and a dependent variable. In addition, the external validity of analyzed experimental studies can be tested as to whether the construction and measurement of a particular variable is theoretically sound in general. The instantiation of this conceptual framework is given by a focused qualitative review of experimental studies on hypertext.

       

    3. A Focused Qualitative Review
    4. The effects of users, tasks, and tools on the performance in terms of process and outcome have been the predominant concern of researchers in the field of hypertext usability. The characteristics of users, the complexity of tasks, and the functionality of hypertext systems have proven to have influential power on users’ browsing strategies and the outcome of interactions.

       

      1. Users
      2. Interests in hypertext raise many psychological issues (e.g., Jonassen, 1993). The effects of individual differences on the use of hypertext systems have been investigated in many experimental studies. This meta-analysis focuses on cognitive aspects of individual differences. According to Messick (1976), cognitive styles are the preferred, consistent, individual characteristics in organizing and processing information. Users are expected to benefit differently from the nonlinear structure of hypertext due to different cognitive styles (Goldstein & Blackman, 1978).

        Several empirical studies investigated the effects of cognitive styles on users’ performance with hypertext. Gray, Barber, and Shasha (1991) studied the effect of locus of control (LOC) on assimilating the underlying structure of a hypertext called dynamic text. By definition, internal LOC users are people who believe they are in control of events, while external LOC users are people who believe that the occurrences of events are controlled by contextual factors. Gray et al. (1991) found that internal LOC users were faster than external LOC users and suggested that internals were more comfortable with the way that information was organized. Lee (1989) explored the effects of characteristics of field-independent and field-dependent users. A field-independent user tends to identify the needed information quickly from a complex context, whereas a field-dependent user tends to perform less efficiently. Lee found that field-independent users performed better with hypertext than with linearly organized text. Leidig (1992) investigated the relationship between cognitive styles and structural maps in hypertext. Users were divided according to their learning styles: 1) divergers, 2) assimilators, 3) convergers, and 4) accommodators. Leidig found a significant interaction between the learning styles and structural maps. For instance, assimilators performed well with both textual and graphical maps of the hypertext structure, whereas convergers and accommodators were less effective in similar conditions.

        The influences of spatial ability on users’ performance with hypertext systems have been investigated by several researchers (Vicente & Williges, 1988; Campagnoni & Ehrlich, 1989). Users with high spatial ability completed tasks more quickly than users with lower spatial ability. Vicente et al. found a main effect of spatial ability on completion time. Campagnoni et al. reported that users with good visualization ability used the top-level table of contents less frequently than users with lower visualization ability, suggesting that a good spatial ability may help one to learn the structure of a subject domain quickly. In contrast, Leidig (1992) only found a marginal main effect of spatial ability on the accuracy of performance, although there was a significant main effect of spatial ability on users' general satisfaction.

        This meta-analysis defines users with internal LOC, high spatial ability, or active learning styles as having active cognitive characteristics. Empirical evidence suggests that individual differences influence users’ performance with hypertext. The nonlinear structure of hypertext should encourage users to interact actively with the information space. The results of experimental studies are synthesized in the meta-analysis with the hypothesis that these active characteristics will help users in their interactions with hypertext.

        Researchers have also been interested in the role of domain knowledge in dealing with information systems. For instance, Carmel, Crawford, and Chen (1992) found that domain experts and novices employed different browsing strategies in a hypertext on the Vietnam War. Novices navigated along more referential links than experts did. Experts were interested in more specific topics and they usually read the most information about a chosen topic. In contrast, novices changed topics in browsing more frequently. In this meta-analysis, however, the issue of the effects of domain knowledge is not taken further because the majority of experimental studies sampled in this study did not focus on users’ knowledge of particular substantive domains.

         

      3. Tasks
      4. Hypertext systems have been designed, tested, and used for people to perform a broad range of tasks. Both strategies and consequences of interactions with hypertext depend on the complexity of tasks. According to Rasmussen et al. (1990), the complexity of a task consists of many contributing factors, including the amount of information to be considered, the number of goals to be fulfilled, and the coupling of goals and contextual constraints.

        Marchionini (1989) classified tasks as closed and open according to related goals. Closed tasks have specific goals, whereas open tasks have general goals. According to Carmel et al. (1992), observable search goals are persistent, while observable browsing goals are transient. Marchionini and Shneiderman (1988) distinguished the goals and strategies of finding facts and browsing knowledge in hypertext systems. Campagnoni and Ehrlich (1989) studied analytical search strategies and browsing strategies. Rada and Murphy (1992) compared the behaviors of search tasks and browsing tasks among several hypertext systems. For a browsing task, users need to find and integrate information from several sources in a hypertext, whereas for a search task, the information is stored in a single place. Similarly, Leventhal, Teasley, Instone, Rohlman, and Farhat (1993) divided test questions as single and multiple entry questions. Although neither search tasks exhaust closed tasks nor browsing tasks represent all the open tasks, the majority of tasks empirically studied can be described in terms of this broad classification of closed and open tasks.

        Given an open task, users need to decompose a general goal into some concrete goals and to consider and integrate information from several sources. Furthermore, the whole process of task accomplishment is subject to the contextual constraints to a greater extent for an open task. Based on this rationale, this meta-analysis assumes that the complexity of open tasks is higher than that of closed tasks. The study also expects that the empirical findings will collectively indicate a significant overall effect size of the complexity of tasks with hypertext systems.

        Closed tasks analyzed in this study include search questions with keywords (Egan, Remde, Gomez, Landauer, Eberhardt, & Lochbaum, 1989), analytical search questions (Campagnoni & Ehrlich, 1989), single entry questions (Leventhal et al., 1993), and questions from technical articles (Gordon, Gustavel, Moore., & Hankey, 1988). In parallel, open tasks include questions without keyword cues (Egan et al., 1989), browsing questions (Campagnoni & Ehrlich, 1989), multiple entry questions (Leventhal et al., 1993), and questions from general readings (Gordon et al., 1988). Creating links in Yusof (1992) are open tasks since users need to consider various issues that might arise on the structure of a hypertext.

        Jonassen (1993a) found an interesting relationship between the effects of tasks and tools on the outcome of performance. In this meta-analysis, the effects of both tasks and tools are synthesized and it will be interesting to see whether the relationship still holds in general.

         

      5. Tools
    The Dexter Hypertext Reference Model is a framework for comparing the architecture of hypertext systems (Halasz & Schwartz, 1990; Halasz & Schwartz, 1994). The Dexter model has three layers: the storage layer, the run-time layer, and the within-component layer. The focus of the Dexter model is on the storage layer (see Figure 1). The storage layer consists of nodes and links to represent the logical structure of the hypertext. The simplest structure is a sequence of nodes and all the links are sequential. The nonlinearality of a hypertext denotes how nodes are linked and how many links are not sequential.
    FIGURE 1 ABOUT HERE

    Hypertext systems provide tools for users to access, view, and manipulate the network structure of nodes and links. This functionality is captured by the run-time layer of the Dexter model. On the other hand, as the developers of the Dexter model noted (e.g., Halasz & Schwartz, 1994), the range of possible tools for accessing, viewing, and manipulating hypertext network is so broad that all the possible functionalities cannot be adequately covered in a single, unified model.

    A general proposition in the hypertext literature has been that an explicit representation of the underlying structure of hypertext will help users interact with hypertext more efficiently and more effectively (Halasz, 1988; Nielsen, 1990). Simpson and McKnight (1990) found that users browsed a hypertext efficiently if a hierarchical structure of the hypertext was added to the user interface, but the browsing was less efficient with an alphabetical index. In contrast, Leventhal et al. (1992) reported that none of the performance measures were significantly correlated to the use of hierarchical structures, including a home card and several overview cards.

    Hammond and Allinson (1989) showed that tasks and hypertext facilities were significantly interrelated. For instance, the greatest effectiveness of indexing facilities was with closed tasks called directed tasks, and guided tours were intensively used for open tasks, such as exploratory tasks. In this meta-analysis the hypothesis is advanced that effect sizes and significance levels will increase as hypertext systems provide indices, tables of contents, or graphical maps. In other words, these facilities should help users understand and access hypertext structure.

    In addition to static, non-interactive structural cues, some hypertext systems use fisheye views to provide dynamic, interactive structural cues (e.g., Egan et al., 1989; Fairchild et al., 1988). Chen, Rada, and Zeb (1994) found a main effect of the provision of a fisheye view browser on users’ performance and a significant interaction with the evolution of a shared hypertext. Fisheye view browsers are natural extensions of graphical overviews. However, due to the lack of empirical studies on fisheye view browsers, the quantitative synthesis of empirical findings on hypertext functionalities will not include fisheye view browsers. Instead, this meta-analysis will be restricted to static structural cues of indices, tables of contents, and graphical overviews.

     

  3. Method
  4. This meta-analysis focuses on experimental studies in which independent variables are related to one of the three contextual variables (namely, users, tasks, and tools) and dependent variables are effectiveness and efficiency measures. Effectiveness measures include two sub-categories: achievement scores and the coverage of a hypertext. The meta-analysis synthesizes the effect sizes for each causal relationship. In addition, by assessing the heterogeneity of a collection of results, the meta-analysis explores the extent to which the three contextual variables can capture the dynamics and complexities of the empirical findings in hypertext studies.

     

    1. A Meta-Analytical Synthesis
The meta-analysis reported in this paper examines 27 published experimental studies and 4 doctoral dissertations that appeared between 1988 and 1993. The purpose of the study is to find invariant underlying relations suggested collectively by the empirical findings and to relate these findings to the principles and models of hypertext systems. The meta-analysis concerns the following propositions.

Effects of Users’ Cognitive Characteristics (Active versus Passive):

Effects of the Complexity of Tasks (Open versus Closed): Effects of Hypertext Tools (Nonlinear versus Linear Organization, Graphical versus Textual Organizational Cues): This meta-analysis is built on the previous review of empirical findings concerning the use of hypertext systems, in particular, the survey of Nielsen (1989). On the other hand, the meta-analysis differs from Nielsen’s work in terms of goals, samples, and methods. The goal of this meta-analysis is to synthesize or combine empirical findings from a number of individual studies on relationships between contextual variables and dependent variables. The goal of Nielsen’s survey was primarily to identify the relationships that have manifested the greatest magnitude of effects.

The selection criteria result in different samples between this meta-analysis and Nielsen’s survey. In addition, studies included in Nielsen’s survey were up to 1989, while this meta-analysis includes studies from 1989 to 1993. Nilesen’s survey concerns a broad range of usability issues. Some of them are beyond the scope of this study. Nielsen (1989) included several studies on the effects of computer screen versus paper presentation (e.g., Wright & Lickorish, 1983, Gould & Grischkowsky, 1984; Gould, Alfaro, Fonn, Haupt, Minuto, & Salaun, 1987; Wilkinson & Robinshaw, 1987; Haas, 1989), screen size (Reisel & Shneiderman, 1987; Hansen & Haas, 1988), and the number of windows on the screen (Tombaugh, Lickorish, & Wright, 1987). Although the issues such as the size of the screen and the number of windows on a screen relate to the design and evaluation of hypertext systems, this meta-analysis focuses on the node-and-link hypertext structure of information.

Not all the studies surveyed by Nielsen (1989) were based on hypertext systems. Among Nielsen’s 20 studies that were conducted with hypertext systems, four met the selection criteria of this meta-analysis (Campagnoni & Ehrlich, 1989; Egan et al., 1989; Monk, Walsh & Dix, 1988; Gordon et al., 1987). Studies are eliminated due to two reasons. A study is eliminated if it did not investigate any relationship to be tested by this meta-analysis. For instance, Ewing, Mehrabanzad, Sheck, Ostroff, and Shneiderman (1986) compared the speed of activating links using mouse versus arrow keys and this relationship is beyond the scope of this meta-analysis, therefore, the study of (Ewing et al., 1986) is not included. A study is also eliminated if the statistics are not reported adequately (e.g., Conklin & Begeman, 1988; Brown, 1989).

 

    1. Sampling
This meta-analysis employed the procedures of sampling, coding, and analysis developed in the social and behavioral sciences (e.g., Rosenthal, 1987; Glass, McGaw, & Smith, 1981). Experimental studies were located from journals, conference proceedings, bibliographical databases, and previous surveys and reviews of hypertext studies. The following journals on human-computer interaction and hypertext were searched: Human-Computer Interaction,

International Journal of Man-Machine Studies,

IEEE Transactions on Systems, Man, and Cybernetics,

Interacting with Computers,

Communications of the ACM,

Hypermedia,

ACM Transactions on Information Systems, and

IEEE Computers.

Conference proceedings searched include the proceedings of the ACM conferences on hypertext (1987, 1989, 1991, and 1993), the proceedings of the British hypertext conferences I and II (McAleese & Green 1989), and the proceedings of CHI (1986, 1988, and 1991). Dissertation Abstract International (DAI) between 1988 and 1993 were also reviewed.

This study also used several bibliographical databases to collect experimental studies, including the PsycINFO database (1987-1993), the Educational Resources Information Center (ERIC) database (1982-1993), and a computer science and electronic engineering database INSPEC (1989-1993). In on-line searches, the terms hypertext, hypermedia, experimental, empirical, and effects were used.

Experimental studies were selected for the meta-analysis according to the following criteria:

The resultant sample consists of 23 experimental studies reported between 1988 and 1993. Of them, 20 studies (67%) appeared in the literature between 1990 and 1993.

 

    1. Coding Individual Studies
    2. The following information was coded for each study: independent variables, dependent variables, sample sizes, methods of assigning subjects, the background of the researchers, characteristics of hypertext systems used, the year of publication, tasks, and statistics of significance tests. A total of 56 effect sizes and associated significance levels were combined in the meta-analysis.

      This study selectively grouped independent variables into three types of contextual variables ? users, tasks, and tools (see Figure 2). The meta-analysis also grouped the dependent variables into two categories of effectiveness and efficiency (see Figure 3).

      FIGURE 2 ABOUT HERE

      Coverage of browsing in hypertext measures how much information stored in hypertext is actually accessed by users. This meta-analysis includes several this type of measures under the category of effectiveness, for instance, the ratio of the number of relevant nodes visited to the number of nodes visited and the ratio of the number of nodes visited to the total number of nodes in a hypertext document. Graphical maps are expected to have greater effect sizes on the effectiveness.

      The validity of such clustering was assessed via subjective ratings of five hypertext experts. Each used a 5-point scale to measure the similarity among variables of individual studies. These five experts were from the department of computer science and included a professor, a visiting professor, an associate professor, and two final-year Ph.D. students. The reliability coefficient alpha was 0.80 (standard item alpha = 0.78) for the five cases of 30 pairs of measurements. Thus the clustering procedure was valid.

       

    3. Analysis
An effect size is the estimate of the magnitude of a specific relationship between two variables. Usually one is the independent variable and the other is the dependent variable. Effect size r can be calculated from a given one-tailed p value and corresponding sample size. Tests of significance alone are not informative enough for practitioners and designers of hypertext systems to judge the usefulness of a design. This meta-analysis compares and combines effect sizes and significance levels in the form of Fisher’s standard score zr and the standard normal deviate score Z. An effect size r was transformed to Fisher’s zr. For instance, an effect size r of .30 corresponds to Fisher’s zr of .31. Z scores can be obtained from reported one-tailed p values of significance tests according to cumulative distribution functions, such as TCDF and FCDF.

Effect sizes in Fisher’s zr were combined according to standard formulae in textbooks on meta-analytical methods. The Z scores were combined according to Stouffer’s method. These two procedures of combination are recommended for their computational simplicity (e.g., Rosenthal, 1987). Finally, the results of the combination were converted back to a correlation coefficient r as the combined effect size and a one-tailed p value as the combined significance level. For studies which only reported group means and standard deviations, the significance levels were calculated as paired t-tests. When results were reported as non-significant, a p value of .50 and a Z of 0.00 were coded.

FIGURE 3 ABOUT HERE

According to Rosenthal (1987), the heterogeneity among a set of effect sizes ? fluctuations from the average of the group ? has a distribution of c 2 with K-1 degrees of freedom, where K is the number of studies. The heterogeneity among significance levels has the same distribution. The heterogeneity test addresses whether the grouping factor is theoretically sound. A large heterogeneity usually suggests that the grouping factor may not capture the variance of a group of results.

Cognitive characteristics of users were measured in terms of locus of control (LOC), field independency, and learning styles (see page *). Open and closed tasks were coded in terms of questions without/with keywords, single/multiple entry questions, and browsing/search questions (see page *). Hypertext systems used in this meta-analysis are largely non-commercial systems. The definition of hypertext in Conklin (1987) is used to classify hypertext or non-hypertext.

As shown in Figure 3, the following measures of effectiveness were used by the selected studies: the percentage of the number of questions correctly answered, the number of concepts correctly recalled, the average number of correct answers, achievement test scores (multiple choice questions), the additional hypertext cards visited compared to a baseline number, and the total number of commands used. The measures of efficiency include the total completion time, the average time to complete, the time spent on each question, the number of tasks completed per hour, and the number of repeated moves between nodes.

This meta-analysis includes a series of tests for linear trends in effect sizes and significance levels in association with a few browsing assistant tools. The purpose of testing for linear trends is to investigate whether the outcomes of experimental studies are affected by increasingly sophisticated tools. A linear trend in variable Y in association with variable X indicates that X and Y are correlated and the values of Y tend to increase or decrease linearly as the values of X increases. For instance, a group of effect sizes are contrasted on whether or not graphical maps are available in corresponding hypertext systems. The contrast weighted effect sizes can be compared to determine whether the influence of graphical maps is substantial.

The meta-analysis included several tests for linear trends regarding the hypothesis that indices, tables of contents, and graphical maps affect interactions with a hypertext system. For example, effect sizes for hypertext systems with graphical maps were assigned contrast weights +1, whereas effect sizes for hypertext systems without graphical maps were assigned weights -1. Thus, a linear trend in effect sizes would verify that the effect sizes were getting larger for hypertext systems with graphical maps.

To assess the effects of users, the meta-analysis compared the effects of cognitive styles, learning styles, locus of control, and spatial ability. For example, active cognitive characteristics are defined by higher measures in LOC (Gray et al. 1991), field dependency (Lee 1990) or active learning styles (Leidig 1992) and passive cognitive characteristics are defined by lower scores in the corresponding variables. For the effects of task complexity, studies which compared open and closed tasks, general and specific substantive domains, browsing and search tasks were included in the meta-analysis. For the effects of tools, experimental studies in the meta-analysis compared hypertext and linear text, graphical maps and textual lists.

 

  1. RESULTS
  2. This meta-analysis focuses on the following questions: to what extent in general do users, tasks, and tools influence the effectiveness and efficiency of using hypertext systems? How significant are these reported effect sizes? Are these effect sizes significantly influenced by particular features of hypertext systems, such as indices, tables of contents, and graphical maps? The results of the analysis are grouped into three parts corresponding to users, tasks, and tools. In each part, findings of individual studies on effect sizes and significance levels are summarized in one table with the combined results.

    1. Users
    2. The effects of cognitive styles and spatial visualization ability were studied in this meta-analysis as the influence of individual differences. The combined effect sizes are small. The differences in dependent measures were smaller in hypertext systems with graphical maps.

      1. Effects of Cognitive Styles on Effectiveness
      2. Cognitive styles were compared as active versus passive. The hypothesis states that active users will benefit significantly more from hypertext than passive users. The results from 4 studies were compared and combined (See Figure 4).

        FIGURE 4 ABOUT HERE

        The combined effect size of cognitive style on effectiveness scores is small ( r = .24 ) according to Cohen (1977) and it is not statistically significant (p = .40). There is a linear trend in effect sizes for indices, tables of contents, and graphical maps ( p = .03 ). Significance levels were not affected by the type of orientation facilities ( p = .48). The heterogeneity test on effect sizes was not statistically significant (c 2 = 5.27, df = 3, p = .15), nor was the heterogeneity test on significance levels (c 2 = 0.09, df = 3, p = .99).

      3. Effects of Cognitive Styles on Efficiency
      4. The meta-analysis investigated the hypothesis that users with active cognitive styles will perform more efficiently than users with passive cognitive styles. This hypothesis was supported by the results from two studies and was rejected in the third study (see Figure 5). Results supporting the hypothesis were assigned positive signs and the negative sign means active users performed less efficiently than passive users.

        FIGURE 5 ABOUT HERE

        Cognitive characteristics have a small combined effect size ( r=.04 ). The effect size is not statistically significant ( p = .14, one-tailed). Although the individual effect sizes differ from each other significantly, the significance levels are not heterogeneous according to the heterogeneity test (c 2 = 2.23, df = 2, p = .33). The effect sizes of cognitive characteristics on efficiency were smaller in hypertext systems with graphical maps( Zlinear-trend = - 4.82, p = .00001), suggesting that graphical maps may augment users’ cognitive ability.

         

      5. Effects of Spatial Ability on Efficiency
      Three studies tested the effects of spatial ability. The hypothesis is that users with good spatial ability will interact with hypertext more efficiently than users with lower spatial ability measures. This hypothesis was supported in all three studies (see Figure 6).
      FIGURE 6 ABOUT HERE

      The combined effect size ( r = .45 ) is medium-large according to Cohen (1977). This finding indicates that spatial ability in general increases the efficiency of interacting with hypertext. Neither the effect sizes nor the significance levels are significantly discrepant (c 2 = 2.24, df = 2, p = 0.33 and c 2 = 0.54, df = 2, p =.76, respectively). Linear trend tests found that graphical maps reduced the differences in dependent measures (Z = - 3.37, p = .03, one-tailed). No linear trends were found in effect sizes with indices or tables of contents (Z = 0.80, p = .47 one-tailed).

    3. Tasks
    4. The meta-analysis compared the effects of tasks with various complexities. Tasks were divided as open and closed according to whether tasks have general or specific goals. Closed tasks were associated with lower complexity, while open tasks were associated with higher complexity. The hypothesis for the synthesis is that users will perform more effectively and more efficiently with hypertext for open tasks than for closed tasks.

      1. Effects of Task Complexity on Effectiveness
      2. Seven studies tested the hypothesis that users will perform more effectively with hypertext for open tasks than for closed tasks. This hypothesis was supported in all the 7 studies (see Figure 7).

        FIGURE 7 ABOUT HERE

        The combined effect size of task complexity ( r = .63 ) is large on effectiveness and these findings are overall significant ( p < .0000001 ). The effects sizes are essentially consistent ( c 2 = 8.60, df = 6, p = .20 ), but the significance levels are heterogeneous (c 2 = 14.07, df = 7, p = .03 ).

        Linear trend tests showed that the effect sizes were enlarged in hypertext systems with indices or tables of contents ( Z = 4.02, p < .00003 ). In contrast, the effect sizes were reduced in hypertext systems with graphical maps ( Z = -5.47, p < .00003). These findings suggest that open tasks can be so cognitive resources demanding that neither indices nor tables of contents have sufficient power to guide users browsing through hypertext or to augment users’ cognitive ability to integrate information from multiple resources. On the other hand, graphical maps were appropriate tools for open tasks so that a certain amount of complexity was reduced by the graphical maps. In other words, there is a strong interaction between task complexity and tools. Significance levels were not substantially affected by indices, tables of contents, or graphical maps.

      3. Effects of Task Complexity on Efficiency
      Seven studies were combined on the hypothesis that users with hypertext systems will complete faster for closed tasks than for open tasks (see Figure 8). The results were basically predictable. Positive signs were assigned to the effect sizes if closed tasks were more efficiently conducted.
      FIGURE 8 ABOUT HERE

      The combined effect size ( r = .58 ) is large according to Cohen’s criteria (Cohen 1977). The combined significance level is equivalent to p = .001 one-tailed. Users performed faster for closed tasks than for open tasks. Neither the effect sizes nor the significance levels are heterogeneous ( c 2 = 3.30, df = 6, p = .77 and c 2 = 11.15, df = 6, p = .08 ).

      Linear trend tests showed that hypertext systems with indices or tables of contents enlarged the differences between experimental and control groups ( Z = 1.74, p = .04, one-tailed ), whereas hypertext systems with graphical maps reduced the effect sizes. However, the significance levels were increased with graphical maps ( Z = - 3.87, p = .0007, one-tailed ). Graphical maps seem to absorb some variances of task complexity.

    5. Tools
    6. This meta-analysis addresses the role of tools in hypertext systems in two aspects. The analysis compared the effects of hypertext systems and that of non-hypertext systems on effectiveness and efficiency measures. The hypothesis for the synthesis is that users with hypertext systems will perform more effectively and efficiently than users with non-hypertext systems. The analysis also verified the hypothesis that graphical maps are useful tools in general for users interacting with hypertext. In addition, the meta-analysis examined the effects of graphical maps on coverage of browsing hypertext.

      1. Effects of Information Structure on Effectiveness
      2. Thirteen studies tested the null hypothesis that there will be no difference in effectiveness measures for people using hypertext and non-hypertext systems. The effect sizes in 8 studies indicate that people using hypertext systems had higher effectiveness, whereas the effect sizes in the rest of 5 studies suggest that people using non-hypertext systems performed more effectively (see Figure 9).

        FIGURE 9 ABOUT HERE

        The combined effect size ( r = .12 ) is small-to-medium according to Cohen (1977) and it is statistically significant ( p < .0000001 ). Both effect sizes and significance levels in this group are heterogeneous ( c 2 = 135.70, df = 14, p < .00001 one-tailed and c 2 = 101.82, df = 14, p < .00001 one-tailed). The heterogeneity suggests that the effects may be influenced by additional underlying factors, for instance, the differences in the design of hypertext systems, different substantive material in hypertext documents, and the design of experiments.

        No linear trend was found with indices, tables of contents, or graphical maps. This result suggests that the organizational structure of information dominates the extent that users’ performance is affected and that individual components of hypertext or non-hypertext systems such as indices, tables of contents, and graphical maps may have relatively weaker influence.

      3. Effects of Information Structure on Efficiency
      4. Five studies tested the null hypothesis that there will be no significant difference in efficiency between people using hypertext or non-hypertext systems. Four studies found that users were faster with non-hypertext systems and one study found contrary evidence (see Figure 10). Positive signs were assigned to the direction of the majority of the findings.

        FIGURE 10 ABOUT HERE

        The combined effect size ( r = .24 ) is medium with a small significance level ( p = .02 ). The effect sizes were essentially different from each other ( c 2 = 16.15, df = 6, p = .01 ), but the significance levels were not ( c 2 = 10.23, df = 6, p = .12 ).

        A negative linear trend in effect sizes was found with whether hypertext systems have graphical maps. In other words, hypertext systems with graphical maps tend to have smaller effect sizes, whereas hypertext systems without graphical maps tend to have larger effect sizes. The superiority of non-hypertext systems on efficiency measures was narrowed by hypertext systems with graphical maps ( Ztrend(Graphical Map) = - 2.97, p = .002 ).

      5. Graphical Maps versus Textual Interface on Effectiveness
      6. The principal design rationale of providing graphical browsers in hypertext systems has been to resolve the problems of disorientation and high cognitive overhead in using hypertext. This meta-analysis compared the effects of graphical maps and that of textual user interface on effectiveness based on the results of 5 studies (see Figure 11). The hypothesis is that graphical maps have greater effect sizes on effectiveness.

        FIGURE 11 ABOUT HERE

        The hypothesis was supported in all the 7 studies. The graphical maps in general have a medium-to-large combined effect size ( r = .38 ) with a combined significance level of p = .00003. Both effect sizes and significance levels are heterogeneous ( c 2 = 38.82, df=6, p = .000001 one-tailed and c 2 = 21.91, df=6, p = .001 one-tailed, respectively ). No linear trend was found with indices, tables of contents, or graphical maps in effect sizes nor significance levels.

      7. Graphical Maps versus Textual Interface on Efficiency
    The meta-analysis combined the results of 3 studies which tested the hypothesis that graphical maps will lead to greater effect sizes on efficiency of using hypertext. Graphical maps indeed had greater effect sizes in all the 3 studies (see Figure 12).
    FIGURE 12 ABOUT HERE

    Graphical maps have a small-to-medium combined effect size ( r = .28 ) with a significance level of p = .006 one-tailed. The effect sizes and the significance levels were not heterogeneous according to corresponding tests ( c 2 = 1.35, df = 2, p = .51 and c 2 = 0.27, df = 2, p = .87 ).

     

  3. DISCUSSION
  4. To summarize, for effectiveness measures, the meta-analysis has found that the complexity of tasks has the greatest effect size ( r = .63 ), followed by graphical maps ( r = .38 ). For efficiency measures, the greatest effects are due to spatial ability of individuals ( r = .45 ) and the complexity of tasks ( r = .58 ). These are large effect sizes according to the guidelines of Cohen (1977) and practitioners and designers of hypertext systems should not overlook these factors.. Hypertext versus non-hypertext systems have medium effect sizes on both effectiveness and efficiency measures in general. Graphical maps also have medium effect sizes on efficiency and coverage. Cognitive styles have small effect sizes on both effectiveness and efficiency measures.

    According to the tests for linear trends in effect sizes, tool facilities such as indices, tables of contents, and graphical maps influenced some types of effect sizes, for instance, the complexity of tasks on both effectiveness and efficiency, hypertext systems and spatial ability on efficiency. In other words, the tools and tasks have the strongest interaction, whereas the interaction between tools and users is particularly found in association with spatial ability of users. In general, users, tasks, and tools showed stronger interaction on efficiency measures than on effectiveness measures.

    The results of the meta-analysis have shown that users, tasks, and tools have various effects. Heterogeneous findings in the empirical literature suggest that further work is needed to overcome existing shortcomings with empirical studies of hypertext. The quantitative synthesis highlighted the areas that need to be strengthened to reduce the discrepancies among empirical findings.

     

    1. Users
    2. This analysis found a small overall effect size of cognitive styles on both effectiveness and efficiency measures. On the other hand, spatial ability consistently had a large effect size on efficiency measures. Based on the synthesized findings, spatial ability influences the processes of interaction with hypertext more directly than individual differences such as cognitive styles, field dependency, or learning styles. Graphical maps were found to have a linear trend in effect sizes of spatial ability. This trend suggests that organizational structure of information visualized by graphical maps helped to narrow the gaps caused by the differences in spatial ability.

      Several factors may contribute to the small combined effect sizes. For instance, effect size may vary from a short interaction to a long interaction; The contents of hypertext nodes, the size of the information space, and the integrative complexity of tasks may also influence users. A closer look at individual effect sizes shows that the absolute values of these effect sizes are not as small as the combined effect size, and suggests that these experiments may profoundly differ in many aspects.

      Nielsen (1987) found that characteristics of users have dominant effects on hypertext usability, for instance, age and motivation. Further work is needed to compare the effects of individual differences with other characteristics of users. In the real world, users interact with hypertext over a considerably longer period of time than has been tested in most experiments. Empirical work should also address how the effects of users’ characteristics evolve over time.

       

    3. Tasks
    4. This meta-analysis found that in the empirical literature the complexity of tasks has the largest effect sizes on effectiveness and efficiency measures. Users completed closed tasks faster with non-hypertext systems, while they benefited more from hypertext systems for open tasks. Nielsen (1987) found that task is one of the most influential factors on empirical findings regarding the usability of hypertext systems. On the other hand, the complexity of tasks and tools of a hypertext system are interrelated. The provision of appropriate tools can reduce the complexity of a particular task. This interrelationship was partially investigated in this meta-analysis through tests for linear trends in effect sizes and significance levels.

      The meta-analysis found that the effect sizes of the complexity of tasks were affected in different ways by various facilities in hypertext systems. As far as the number of information sources that users need to deal with for a single task is concerned, indices or tables of contents did not seem to be appropriate tools to augment users’ ability to assimilate information from several sources. Effect sizes were reduced in hypertext systems with graphical maps. The reduction in effect size indicate that graphical maps are useful because the differences between users’ performance on tasks with high and low complexities are narrowed by the provision of graphical maps.

      In general, hypertext systems are appropriate for open tasks such as browsing and assimilating. On the other hand, closed tasks such as indexing and searching across nodes and links have been inadequately supported in hypertext systems. Search facilities were under-emphasized in hypertext systems (Halasz, 1988).

      This meta-analysis classified tasks as being open and closed according to the complexity of goals. Although the effect sizes are not heterogeneous according to the heterogeneity tests, the significance levels have substantial discrepancies among individual studies. The heterogeneity of significance levels is partially due to discrepancies in task design and the absence of a taxonomy for cognitive complexity of tasks. More detailed task analysis is needed to understand interactions with hypertext systems in the real world. A finer-grained comparison can only be based on a taxonomy resulting from the task analysis.

       

    5. Tools
    6. This meta-analysis found that users of hypertext systems tend to have higher effectiveness scores than users of non-hypertext systems. On the other hand, effect sizes and significance levels differ significantly. Experimental designs varied as the hypertext systems varied. Some systems have only the most primitive hypertext facilities implemented. For instance, a page of a document is displayed on the screen at a time and hypertext links are provided at page-to-page level. Some systems are advanced and include graphical browsers, a persistent window to display the current focal point, footprints, and search facilities.

      This meta-analysis took a small step toward accounting for the characteristics of each experimental design. Linear trends in effect sizes and significance levels were tested with indices, tables of contents, and graphical maps. Graphical maps appear to be necessary for users dealing with large and complex information structures in hypertext and to be useful to resolve the problems of disorientation and high cognitive overhead. The combined effect size of graphical maps ( r = .38 ) is based on 7 individual effect sizes: 2 large ones ( r = .77, .77), 4 small ( r = .01, .14, .15, .16), and 1 medium ( .38 ). The 2 large ones have the highest significance-level scores ( Z = 5.03 and 4.10, respectively ). The meta-analysis clearly shows that a graphical map is a very important factor that can improve the effectiveness and efficiency of interacting with a hypertext system.

      For the effects of nonlinearality of information organization, the combined Z's of weighted significance levels are no longer significant. However, for tests of the effects of task complexities, the combined Z's of weighted significance levels are significant, thus the degrees of statistical significance tend to increase linearly as the availability of navigation tools increases. In particular, contrast weighting by graphical browsers switched the directions of effects. That is, graphical browsers may play a significant role in the relationship between task complexities and effectiveness. Experimental studies will be needed to test the hypothesis in more accurate forms.

      An interesting finding is that graphical browsers and cognitive styles are interrelated. Using a hypertext system with graphical browsers tends to be correlated with a larger difference between the experimental group and the control group. This agrees with the general interpretation that graphical browsers reduce the impact of cognitive styles.

       

    7. Strengths and Limitations of the Meta-Analysis
    A synthesis of the empirical findings on hypertext from a single, unifying perspective would help researchers, designers, and users of hypertext systems. A synthesis highlights the salient findings in the literature so that the communication becomes more efficient and effective. A synthesis may help people to understand the relationships among existing findings so that the knowledge can be better assimilated. A synthesis may also provide useful suggestions for future research.

    The main limitation of a meta-analytical study is sampling bias. It is largely a subjective decision on where to look for relevant studies and which studies to be included in the analysis. Some sampling difficulties are particularly related to the literature of hypertext. Most hypertext systems and studies have been available only since the late 80's. Hypertext conferences in the USA and Europe have been dominated by design and implementation issues. On the other hand, empirical work on hypertext has been published in journals of almost every major special field in computer science, such as database management systems, information retrieval, expert systems, computer-aided learning, and human-computer interaction.

    Exclusive use of published work is known to have a bias toward statistically significant, hypothesis-confirming results (McLeod, 1992). This meta-analysis includes several unpublished doctoral dissertations from universities in the USA in order to enlarge the scope of the relevant experimental studies for the study. These doctoral dissertations are available in microfiche from the producer UMI.

    This meta-analysis investigates the outcomes of experimental studies on 9 hypotheses regarding the role of users, tasks, and tools in the use of hypertext systems. Seven or more individual experiments are analyzed for each of 5 of the 9 hypotheses, but a small number of studies are entered for each of the remaining 4 hypotheses. In particular, the three hypotheses regarding the effects of users’ cognitive characteristics are based on the results of 4 or 3 individual studies.

    Users’ cognitive characteristics are associated with small effect sizes in this meta-analysis. However, the evidence is not sufficient to conclude that cognitive characteristics of users have little impact on the effectiveness and efficiency of interacting with a hypertext system. It is important to bear in mind the small sample size. There are few experimental studies in this area. Experimental studies addressing the role of users’ characteristics in using hypertext are particularly needed. The effects of cognitive characteristics may be better detected by a longitudinal study; cognitive characteristics may have a strong interaction with the user interface of different hypertext systems. In fact, if the sample size is small, the result is likely biased toward the individual study that has the predominant estimate. This meta-analysis implies that more work is needed for understanding the effects of cognitive characteristics of users.

    Subjective classification of various independent and dependent variables from individual experiments is another source of discrepancies. Few empirical studies of hypertext used identical benchmark measures. This problem is partially resolved in the meta-analysis by focusing on variables which have been consistently measured or that can be easily transformed, such as effectiveness scores and the speed of completion.

    This meta-analysis decided to classify the three experimental studies in Egan et al. (1989) as independent studies. As evident in the results, the SuperBook used in Study I and Study II of Egan et al. (1989) was significantly different. For example, users were slower with the first version of SuperBook than users with Prints, whereas users were faster with the second version at the same significance level. Therefore, the decision was made so as to emphasize the differences in the design of the two versions, instead of treating the different outcomes as the discrepancies of using the same system in different trials by different users. Readers should bear in mind these considerations of this meta-analysis. If a meta-analysis is based on a small sample size, dependent studies may attract a small weight in effect in the analysis. In this meta-analysis, sample sizes are large for the hypotheses in which these three studies are involved.

     

  5. CONCLUSION
This meta-analysis highlights two major issues concerning the empirical literature of hypertext: 1) the diversity of experimental designs and 2) the heterogeneity of empirical findings. The diversity of experimental designs makes experimental studies difficult to compare with each other. The great heterogeneity undermines the significance of individual findings.

Based on the results of the meta-analysis, theoretical and empirical work on hypertext needs to be improved in two aspects: 1) the development of a task taxonomy of interacting with hypertext and 2) the instantiation of particular hypertext systems in standard hypertext models. This study recommends that the diversity of experimental designs could be reduced by fitting specific hypertext systems into standard models of hypertext such as the Dexter model and by developing new benchmarks to measure both the processes and outcomes of interacting with hypertext systems.

Benchmark measures currently in the empirical studies of this meta-analysis were largely conventional information retrieval measures. These benchmark measures may not remain valid for addressing hypertext usability. Anderson et al. (1989) made an effort in the development of benchmarks of hypertext systems, but their benchmarks are inadequate for dealing with the major issues of hypertext usability. The heterogeneity among outcomes of individual studies may be reduced, if the benchmark measures can be developed in relation to a hypertext reference model, such as the Dexter model.

Major conclusions derived from the results of the meta-analysis can be summarized as follows:

  1. differences in the performance and outcome of users' interactions with hypertext are significantly related to the tasks performed;
  2. individual differences in cognition alone do not have significant effects on the use of hypertext;
  3. significant discrepancies exist in experimental designs for hypertext usability studies, and individual findings differ significantly from each other in general;
  4. a taxonomy of cognitive tasks for analyzing and comparing different findings is missing in the hypertext literature; and
  5. connections between an abstract hypertext reference model and individual experimental designs are inadequate.
Interacting with hypertext is a dynamic and complex process. The process can be characterized from behavioral, cognitive, and social perspectives. The majority of experimental studies in the literature focused on traditional benchmark measures of effectiveness and efficiency. As Nielsen (1989) noted, a useful hypertext system in reality tends to be embedded into a particular domain of activity. Hypertext is expected to have profound influence on how people interact with computer systems. Benchmark measures can only partially reveal the interrelationships between users and hypertext-based systems. It is more important and also more difficult to understand the dynamics of how users respond to the changes in their situations.

A process-oriented approach should play a more active role in hypertext studies. Although the Dexter model could suggest a way of organizing the functionalities of hypertext systems in the storage layer, much of the interactions take place in the run-time layer, where the Dexter model intentionally leaves out the specification to particular hypertext systems. Dynamic models of interactive behaviors with hypertext-based systems are required. Characterizing the sequential structure of user behavior and analyzing paths of interactions in hypertext are particularly useful (e.g., Chen & Rada, 1995; Olson, Herbsleb, & Rueter, 1994). They are essential to the understanding of some fundamental issues regarding hypertext, for example, the strategies of users, how the functionality of a system is utilized, how a system design can be improved, and, ultimately, how the process of work can be organized with effectiveness and efficiency.

The goal of this meta-analysis has been to provide a quantitative synthesis of the empirical findings on hypertext. In the long run, hypertext usability research will benefit from interdisciplinary collaboration, which involves computer sciences, psychology, and educational research. More work will be necessary to adapt research methodologies from other disciplines to address usability issues particularly related to hypertext.

NOTES

Background. This work was conducted as part of the Ph.D research of the first author at the University of Liverpool, UK.

Acknowledgements. Gary Olson and three anonymous referees provided valuable and constructive comments on an earlier version of the paper.

Support. The authors would like to acknowledge the financial support of the European Commission DELTA project D2006.

Authors’ Present Address. Chaomei Chen, Department of Computer Studies, Glasgow Caledonian University, City Campus, Cowcaddens Road, Glasgow, G4 0BA, UK. E-mail: C.Chen@gcal.ac.uk; Roy Rada, School of EE & Computer Science, Washington State University, Pullman, WA 99164-2752. E-mail: rada@wsu.edu.

REFERENCES

Anderson, T., Berre, A., Mallison, M., Porter, H. III, and Schneider, B. (1990). The hypermodel benchmark. In F. Bancilhon, C. Thanos, & D. Tsichritzis (Eds.), Advances in database technology ? EDBT’90 (pp. 317-331). London: Springer-Verlag.

‡Brown, P. (1989). Keynote address. Presented at the Hyeprtext II Conference, 29-30 June, 1989. York, UK.

†‡Campagnoni, F., & Ehrlich, K. (1989). Information retrieval using a hypertext-based help system. ACM transactions on information systems, 7, 271-291.

Carmel, E., Crawford, S., & Chen, H. (1992). Browsing in hypertext: a cognitive study. IEEE Transactions on Systems, Man, and Cybernetics, 22, 865-884.

Chen, C., & Rada (1995). Understanding collaborative authoring in shared workspaces. Proceedings of the Fifth IFIP TC13 International Conference on Human-Computer Interaction (INTERACT’95). London: Chapman & Hall.

Chen, C., Rada, R., & Zeb, A. (1994). An extended fisheye view browser for collaborative writing. International Journal of Human-Computer Studies, 40, 859-878.

Cohen, J. (1977). Statistical power analysis for the behavioral sciences (revised ed.). New York: Academic Press.

Conklin, J. (1987). Hypertext: An introduction and survey. IEEE Computer, 20, 17-41.

‡Conklin, J., & Begeman, M. (1988). gIBIS: A hypertext tool for exploratory policy discuss. ACM Transactions on Office Information Systems, 6, 303-331.

†‡Egan, D., Remde, J., Gomez, L., Landauer, T., Eberhardt, J., & Lochbaum, C. (1989). formative design-evaluation of SuperBook. ACM Transactions on Information Systems, 7, 30-57.

‡Ewing, J., Mehrabanzad, S., Sheck, S., Ostroff, D., & Shneiderman, B. (1986). An experimental comparison of a mouse and arrow-jump keys for an interactive encyclopedia. International Journal of Man-Machine Studies, 24, 29-45.

Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-analysis in social research. London: Sage.

Goldstein, K. M., & Blackman, S. (1978). Cognitive style: five approaches and relevant research. New York: John Wiley & Sons.

†‡Gordon, S., Gustavel, J., Moore, J., & Hankey, J. (1988). The effects of hypertext on reader knowledge representation. Proceedings of the Human Factors Society 32nd Annual Meeting, 296-300. Santa Monica, CA: HFS.

‡Gould, J. D., Alfaro, L., Fonn, R., Haupt, B., Minuto, A., & Salaun, J. (1987). Why reading was slower from CRT displays than from paper. Proceedings of the ACM Conference CHI+GI’87, 7-11. Toronto, Canada.

‡Gould, J. D., & Grischkowsky, N. (1984). Doing the same work with hard copy and with cathod ray tube (CRT) computer terminals. Human Factors, 26, 323-337.

†Gray, S., Barber, C., & Shasha, D. (1991). Information search with dynamic text vs paper text: an empirical comparison. International Journal of Man-Machine Studies, 35, 575-586.

Gronbaek, K., & Trigg, R. (1994). Design issues for a Dexter-based hypermedia system. Communications of the ACM, 37, 40-49.

‡Hass, C. (1989). Does the medium make a difference? Two studies of writing with pen and paper and with computer. Human-Computer Interaction, 4, 149-169.

Halasz, F. (1988). Reflections on NoteCards: seven issues for the next generation of hypermedia systems. Communications of the ACM, 31, 836-852.

Halasz, F., & Schwartz, M. (1994). The Dexter hypertext reference model. Communications of the ACM, 37, 30-39.

Hammond, N., & Allinson, L. (1989). Extending hypertext for learning: An investigation of access and guidance tools. In A. Sutcliffe, & L. Macaulay (Eds.), People and computers V (pp. 293-304). Cambridge, England: Cambridge University Press.

‡Hansen, W. J., & Haas, C. (1988). Reading and writing with computers: a framework for explaining differences in performance. Communications of the ACM, 31, 1080-1089.

†Hendry, D. G., Carey, T. T., & TeWinkel, S. T. (1990). A Study of Measures for research in hypertext navigation: In D. Diaper, D. Gilmore, G. Cockton, & B. Shackel (Eds.), Human-computer interaction - INTERACT'90, 101-106. Amsterdam: North-Holland.

Hunter, J. E., & Schmidt, F. L. (1990). Methods of meta-analysis: correcting error and bias in research findings. London: Sage.

Hunter, J. E., Schmidt, F. L., & Jackson, G. B. (1982). Meta-analysis: cumulating research findings across studies. Beverly Hills, CA: Sage Publications.

Hutchings, G. A., Hall, W., & Colbourn, C. J. (1993). Patterns of students' interactions with a hypermedia system. Interacting with Computers, 5, 295-313.

†Instone, K., Teasley, B. M., & Leventhal, L. M. (1993). Empirically-based re-design of a hypertext encyclopedia. Bridging the gap - INTERCHI'93, 500-506. Amsterdam: North-Holland.

Jonassen, D. H. (1993a). Acquiring structural knowledge from semantically structured hypertext, Journal of Computer-Based Instruction, 20, 1-8.

Jonassen, D. H. (1993b). Hypertext: A psychological perspective. London: Ellis Horwood.

†Klappenberger, F. (1990). The effect of domain knowledge on searching for specific information in a hypertext environment. Unpublished doctoral dissertation, University of Maryland. UMI-91-10319.

†Lee, Y-B. B. (1989). Effects of learning style and instructional cues on achievement and learning interactivity in a hypermedia instructional system. Unpublished doctoral dissertation, Purdue University. UMI-90-08652.

†Leidig, P. (1992). The relationship between cognitive styles and mental maps in hypertext assisted learning. Unpublished doctoral dissertation, Virginia Commonwealth University. UMI-92-25398.

†Leventhal, L., Teasley, B., Instone, I., & Farhat, J. (1992). Age-related differences in the use of hypertext: Experiment and design guidelines. (Technical Report 92-Dec-01). Bowling Green, OH: Department of Computer Science, Bowling Green State University.

†Leventhal, L., Teasley, B., Instone, I., Rohlman, D., & Farhat, J. (1993). Sleuthing in HyperHolmes: An evaluation of using hypertext vs. a book to answer questions. Behaviour and Information Technology, 12, 149-164.

Marchionini, G. (1989). Information-seeking strategies of novices using a full-text electronic encyclopedia. Journal of the American Society for Information Science, 40, 54-66.

Marchionini, G. & Shneiderman, B. (1988). Finding facts versus browsing knowledge in hypertext systems. IEEE Computer, 21, 70-79.

McAleese, R., & Green, C. (Eds.). (1990). Hypertext: State of the art. Oxford: Intellect.

†McKnight, C., Dillon, A., & Richardson, J. (1990). A comparison of linear and hypertext formats in information retrieval. In R. McAleese & C. Green (Eds.), Hypertext: State of the art (pp. 10-19). Oxford: Intellect.

McLeod, P. (1992). An Assessment of the experimental literature on electronic support for group work: Results of a meta-analysis. Human-Computer Interaction, 7, 257-280.

Messick, S. (1976). Individuality in learning: implications of cognitive styles and creativity for human development. San Fransisco: Jossey-Bass.

†‡Monk, A., Walsh, P., & Dix, A. (1988). A comparison of hypertext, scrolling and folding as mechanisms for program browsing. In D. M. Jones & R. Winder (Eds.), People and computers IV (pp. 421-435). Cambridge, England: Cambridge University Press.

Nielsen, J. (1989). The matters that really matter for hypertext usability. Proceedings of the ACM conference on Hypertext ? Hypertext'89, 239-248. New York: ACM Press.

Nielsen, J. (1990). Hypertext and hypermedia. London: Academic Press.

†Nosek, J., & Roth, I. (1990). A comparison of formal knowledge representation schemes as communication tools: predicate logic vs semantic network. International Journal of Man-Machine Studies, 33, 227-239.

†Nowaczyk, R. H., & Snyder, A. D. (1993). Using hypertext for college-course information, Computers & Education, 21, 273-279.

Olson, G., Herbsleb, J., & Rueter, H. (1994). Characterizing the sequential structure of interactive behaviours through statistical and grammatical techniques. Human-Computer Interaction, 9, 427-472.

Rada, R., & Murphy, C. (1992). Searching versus browsing in hypertext. Hypermedia, 4, 1-30.

Rasmussen, J., Brehmer, B., de Montmollin, M., & Jacques, L. (1990). Taxonomy for analysis of work domains. Proceedings of the First MOHAWC Workshop. Roskilde, Denmark: Riso National Laboratory.

‡Reisel, J. F., & Shneiderman, B. (1987). Is bigger better? The effects of display size on program reading. In G. Salvendy (Ed.), Social, ergonomic and stress aspects of work with computers (pp. 113-122). Elsevier Science Publishers.

Rosenthal, R. (1987). Judgment studies: design, analysis, and meta-analysis. Cambridge, England: Cambridge University Press.

†Simpson, A. (1990). The interface to a hypertext journal. In D. Diaper, D. Gilmore, G. Cockton, & B. Shackel (Eds.), Human-Computer Interaction ? INTERACT'90, 869-874. Amsterdam: North-Holland.

†Simpson, A., & McKnight, C. (1990). Navigation in hypertext: structural cues and mental maps. In R. McAleese & C. Green (Eds.), Hypertext: State of the art (pp. 73-83). Oxford: Intellect.

‡Tombaugh, J., Lickorish, A., & Wright, P. (1987). Multi-window displays for readers of lengthy texts. International Journal of Man-Machine Studies, 26, 597-615.

†Vicente, K. J., & Williges, R. C. (1988). Accommodating individual differences in searching a hierarchical file system. International Journal of Man-Machine Studies, 29, 647-668.

†Wey, Pi-Shin (1992). The effects of different interface presentation modes and users' individual differences on users' hypertext information access performance. Unpublished doctoral dissertation, University of Illinois at Urbana-Champaign. UMI-93-05730.

‡Wilkinson, R., & Robinshaw, H. (1987). Proof-reading: VDU and paper text compared for speed, accuracy and fatigue. Behaviour and Information Technology, 6, 125-133.

Wright, P. (1991). Cognitive overheads and prostheses: some issues in evaluating hypertexts, Proceedings of the ACM conference on Hypertext ? Hypertext '91, 1-12. New York: ACM Press.

‡Wright, P., & Lickorish, A. (1983). Proof-reading texts on screen and paper. Behaviour and Information Technology, 2, 227-235.

Yusof, S. (1992). Learning from hypertext: Effects on cognitive structures. Unpublished doctoral dissertation, Southern Illinois University at Edwardsville. UMI 93-00071.

Footnotes

1. If the entry of Z=0.00 for Study 2 is not included as an estimation of a non-significant p value, the combined Z is 5.56 and the heterogeneity test results in c 2 =99.75, p < .00001.

2. Studies with † are included in the sample of the meta-analysis and studies with ‡ were included in Nielsen (1989).

 

Figure Captions

Figure 1. The focus of the Dexter Hypertext Reference Model (Adapted from Halasz & Schwartz, 1994).

 

Figure 2. Studies included and independent variables. Studies with `*’ are reported in Ph.D. dissertations.

Figure 3. Studies included and dependent variables. Studies with `*’ are reported in Ph.D. dissertations.

Figure 4. The effects of users’ cognitive characteristics on effectiveness measures.

Figure 5. The effects of users’ cognitive characteristics on efficiency of use.

Figure 6. Effects of spatial ability on efficiency of interacting with hypertext.

Figure 7. The effects of task complexity on effectiveness. Interacting with hypertext is more effective for open tasks.

Figure 8. The effects of task complexity on efficiency measures with hypertext.

Figure 9. The effects of the nonlinear structure of hypertext on effectiveness. Positive signs indicate that using hypertext resulted in better performance.

Figure 10. The effects of nonlinearality of information organization on performance efficiency. Users seem to perform better with hypertext systems in large samples.

Figure 11. The effects of graphical maps on effectiveness scores.

Figure 12. The effects of graphical maps on efficiency of use.

 

Figures

Figure 1. The focus of the Dexter Hypertext Reference Model (Adapted from Halasz & Schwartz, 1994).

 

Figure 2. Studies included and independent variables. Studies with `*’ are reported in Ph.D. dissertations.

  Study Users Tasks Tools
1
Campagnoni & Ehrlich, 1989 spatial ability    
2
Egan, Remde, Gomez, Landauer, Eberhardt, & Lochbaum, 1989 (Study 1) with/without

keywords

hypertext
3
Egan et al., 1989 (Study 2) with/without

keywords

hypertext
4
Egan et al., 1989 (Study 3) with/without

keywords

hypertext
5
Gordan, 1988   general/specific hypertext
6
Gray, Barber, & Shasha, 1991 locus of control   hypertext
7
Hendry, Carey, & TeWinkel, 1990   browse/study  
8
Instone, Teasley, & Leventhal, 1993   hypertext; 

graphical maps

9
Klappenberger, 1990* spatial ability general/specific  
10
Lee, 1989* field dependency    
11
Leidig, 1992* (Study 1) learning style;

spatial ability

  hypertext;

graphical maps

12
Leidig, 1992* (Study 2) learning style    
13
Leventhal, Teasley, Instone, & Farhat, 1992   search/inference  
14
Leventhal, Teasley, Instone, Rohlman, & Farhat, 1993   search/inference hypertext
15
McKnight, Dillon, & Richardson, 1990     hypertext
16
Monk, Walsh, & Dix, 1988 (Study 1)   hypertext
17
Monk, Walsh, & Dix, 1988 (Study 2)   graphical maps
18
Nosek & Roth, 1990     hypertext
19
Simpson, 1990   search/browse hypertext; 

graphical maps

20
Simpson & McKnight, 1990   search/browse graphical maps.
21
Vicente & Williges, 1988 spatial ability   hypertext
22
Wey, 1992* learning style   graphical maps
23
Nowaczyk & Snyder, 1993   open/closed  hypertext
 

 

Figure 3. Studies included and dependent variables. Studies with `*’ are reported in Ph.D. dissertations.

  Study Effectiveness Efficiency
1
Campagnoni & Ehrlich, 1989 solution time
2
Egan, Remde, Gomez, Landauer, Eberhardt, & Lochbaum, 1989 (Study 1) percentage correct  average time to complete
3
Egan et al., 1989 (Study 2) percentage correct average time to complete
4
Egan et al., 1989 (Study 3) percentage correct  average time to complete
5
Gordan, 1988 number of recalled concepts  total time of reading
6
Gray, Barber, & Shasha, 1991 average number of correct answers average time to complete
7
Hendry, Carey, & TeWinkel, 1990   node visiting time
8
Instone, Teasley, & Leventhal, 1993 accuracy speed
9
Klappenberger, 1990* percentage time of

viewing relevant nodes

10
Lee, 1989* achievement completion time
11
Leidig, 1992* (Study 1) learning outcome completion time
12
Leidig, 1992* (Study 2) learning outcome completion time
13
Leventhal, Teasley, Instone, & Farhat, 1992 accuracy scale time on each question
14
Leventhal, Teasley, Instone, Rohlman, & Farhat, 1993 accuracy time on each question
15
McKnight, Dillon, & Richardson, 1990 number of items correctly answered time to complete task
16
Monk, Walsh, & Dix, 1988 (Study 1) tasks correct tasks per hour
17
Monk, Walsh, & Dix, 1988 (Study 2) tasks correct tasks per hour
18
Nosek & Roth, 1990 precision  
19
Simpson, 1990 percentage correct  number of moves

between nodes

20
Simpson & McKnight, 1990 additional cards opened
21
Vicente & Williges, 1988 total number of commands search time
22
Wey, 1992* completion time
23
Nowaczyk & Snyder, 1993   completion time
 

 

Figure 4. The effects of users’ cognitive characteristics on effectiveness measures.

Study
Sample Size
N
Effect Size
r
Significance Test
Z
6
80
.13
0.13
10
96
.19
0.06
11
32
-.28
-0.06
12
37
.06
0.36
Combined
.24
0.25
 

Figure 5. The effects of users’ cognitive characteristics on efficiency of use.

Study
Sample Size
N
Effect Size
r
Significance Test
Z
6
80
.24
1.83
10
96
.31
0.003
22
61
-.39
-0.001
Combined
.04
1.06

Figure 6. Effects of spatial ability on efficiency of interacting with hypertext.

Study
Sample Size
N
Effect Size
r
Significance Test
Z
1
12
.65
2.33
9
32
.23
1.30
21
40
.44
1.95
Combined
.45
3.22

Figure 7. The effects of task complexity on effectiveness. Interacting with hypertext is more effective for open tasks.

Study
Sample Size
N
Effect Size
r
Significance Test
Z
2
10
.74
2.24
3
20
.74
3.58
4
10
.77
3.46
5
24
.47
2.04
13
41
.30
1.95
14
29
.67
6.36
19
32
.54
2.97
Combined
.63
5.54

Figure 8. The effects of task complexity on efficiency measures with hypertext.

Study
Sample Size
N
Effect Size
r
Significance Test
Z
2
10
.70
2.02
3
20
.60
2.57
4
10
.56
2.46
5
24
.63
3.13
7
44
.38
2.27
14
29
.63
6.00
23
30
.50
2.59
Combined
.58
3.01

Figure 9. The effects of the nonlinear structure of hypertext on effectiveness. Positive signs indicate that using hypertext resulted in better performance.

Study
Sample Size
N
Effect Size
r
Significance Test
Z
2
10
.00
0.00
3
20
.71
3.33
4
10
.75
4.15
5a
24
-.42
-1.76
5b
24
-.20
-1.88
6
80
.42
3.66
8
24
.86
5.33
11
142
-.18
2.19
14
29
.34
1.43
15a
8
-.86
-2.74
15b
8
-.78
-2.32
15c
8
.86
2.73
16
20
.16
0.61
18
47
.16
1.08
19
32
.81
5.03
Combined
.22
5.38

Figure 10. The effects of nonlinearality of information organization on performance efficiency. Users seem to perform better with hypertext systems in large samples.

Study
Sample Size
N
Effect Size
r
Significance Test
Z
2
10
.74
2.24
3
20
.00
0.00
6
80
.04
0.65
14
29
.33
1.40
15
8
.14
0.39
16
20
.49
2.18
21
40
-.29
-1.45
Combined
179
.24
2.04

Figure 11. The effects of graphical maps on effectiveness scores.

Study
Sample Size
N
Effect Size
r
Significance Test
Z
12
140
.01
0.14
17
20
.14
0.58
18
46
.16
0.58
19
32
.77
5.03
20
24
.77
4.10
21
40
.31
1.61
22
61
.15
1.12
Combined
363
.38
4.97

Figure 12. The effects of graphical maps on efficiency of use.

Study
Sample Size
N
Effect Size
r
Significance Test
Z
17
20
.41
1.81
21
40
.29
1.45
22
61
.14
1.07
Combined
121
.28
2.50