An Input-Process-Output Structural Framework
for Evaluating Web-based Instruction

Chong Ho Yu, Ph.D.s


Introduction

Various evaluation methodologies have been proposed for assessing the impact of Web-based instruction (WBI) on learners (Svanum, Chen, & Bublitz, 1997; Swigger, Brazile, Lopez, & Livingston, 1997). Due to the relative newness of the medium and the vagueness of the scope of WBI, there isn't a unified theoretical framework for WBI evaluation. To fill this gap, a structural framework with an emphasis on input-process-output is proposed. The framework also functions as a reference for conceptualizing an evaluation from the abstract level to the specific level.

The input-process-output structural framework (see Figure 1) is a specification of how different input, intermediate, and output variables form causal relationships in a system. The structural framework has four levels of abstraction: paradigm, theory, model, and measurement.
 

paradigm
theory
model
measurement

The highest layer of the framework has the highest of abstraction and vice versa. The degree of specification increases as the layer goes down. A theory is an implementation of a paradigm, a model is a specification of a theory, and a measurement is the quantification or the empirical representation of a model. Implementations of the structural framework may vary in the theory, model, and measurement levels. However, the emphasis on input-process-output is firmly embedded in the paradigm level.

Paradigm

In this writing, the meaning of "paradigm" is adopted from the usage by Thomas Kuhn (1962). In Kuhn's view, a paradigm is a set of metaphysical beliefs and underlying assumptions that make up a theoretical framework. In other words, a paradigm establishes the "boundary of the playground" and "rules of the game" for researchers in the discipline.

The proposed structural framework endorses a paradigm that emphasizes input-process-output of human learning inspired by cognitive psychology. This paradigm challenges the input-output emphasis under a "black-box" approach.

Under a "black box" approach, an input, which is usually a treatment, is delivered to the intended audience. Afterwards, the output, which is usually an observable outcome, is evaluated and the effectiveness of the treatment is inferred. This input-output approach leaves many important questions unanswered such as "What are the properties of the instructional media?" "What effects are resulted from these properties?" "How do the learners use the program?" and "How do the learners process the information?"

This "black box" approach could be traced back to Behaviorism and classic learning theory, in which the learner or the subject is treated as a black box. In Behaviorism, only stimulus (input) and responses (output) are emphasized for controlling behaviors. In this system the details of underlying structure, mechanism, and dynamics are either unknown or regarded as unimportant (Fontana, 1984).

With the increasing popularity of cognitive psychology, questions about human mental structure and process are being addressed. For instance, "How is knowledge represented?" "How does an individual acquire new knowledge?" (Kellogg, 1995) Because of the process-orientation of cognitive psychology, educational psychologists pay much attention to learning processes (Anderson, 1990; West, Farmer, & Wolff, 1991). In the context of Web-based instruction, questions about relationships between instructional media and learning processes should be raised. For example, "How does hyperlinking enhance connectivity of human schema? How does asynchronous collaboration on the Internet improve intrinsic motivation?"

Approaching from two dimensions

Under the input-process-output structural framework, the procedures of evaluating WBI could be viewed in two dimensions. The first dimension is the level of abstraction. As shown in Figure 1, approaching the research from the abstract layer to the specific layer can be visualized as a vertical process:

  • defining various constructs and concepts in the theory
  • defining variables, which indicate the constructs and concepts, in the model
  • developing or applying instruments to measure the variables in the measurement,
  • determining the relationships among the variables based upon the data collected by the instruments.

The second dimension is the input-process-output stream. As seen in Figure 1, this streaming can be viewed as a horizontal process:

  • defining the media properties
  • defining the mental constructs and mapping them to the media properties
  • defining the criteria of achievement specifying the behavioral outcomes.
In the following section, each component will be explained in terms of the second dimension.

Media properties

Different instructional designers may have different implementations of Web-based instruction. Inexperienced evaluators may ask an excessively board question such as "Can Web-based instruction enhance learning?" It is important to define what WBI is by identifying what certain features are introduced by the implementation of the medium. Rather than assessing the overall WBI program, the evaluation should focus on those specific properties. Typical examples of WBI's properties are:

  • learner control modules that present self-customized sequences to various learners,
  • virtual communities that provide chat rooms, threaded discussions, and ListServ groups,
  • hypermedia with graphical representations and animated illustration of information,
  • hyperlinks to world-wide resources.

Different use of those resources may contribute to different learning processes and outcomes. There are two ways to define variables and to develop instruments for the above media properties. The first way is to create another version of treatment without these media properties and randomly assign users to this control group. In this fashion, group membership is a variable. Another way is to treat the tools which carry specific media properties as numeric variables and track the frequency of using the tools. For example, the number of chat, threaded discussion, and ListServ messages could indicate the utilization of virtual communities. The number of time spent and the number of page accessed within the WBI could reflect the user engagement. Several web traffic tracking technologies can be used to collect data for studying user patterns on WBI (Yu, Jannasch-Pennell, DiGangi, & Wasson, 1998). .

Mental constructs

It is hypothesized that different media properties could change different aspects of user mentality. For instance, adaptive learning could lead to a positive attitude toward self-directed learning, virtual communities could strengthen the sense of collaboration, hypermedia can enrich visual-oriented cognitive process, and hyperlinks to world-wide resources can enhance cross-cultural awareness. A mental construct can be broken down into several variables for in-depth investigation. For instance, collaboration can be defined by eight factors: openness to learning opportunities, self-concept as an effective learner, initiative and independence in learning, informed acceptance of responsibility for one's own learning, a love to learn, creativity, future orientation, and the ability to use basic study skills and problem-solving skills (Guglielmino, 1977; Guglielmino & Murdick, 1997). These variables could be measured by validated and reliable instruments on attitudes and cognitive styles.

However, using standardized instruments is not absolutely flawless. Usually the variables that are said to be latent dimensions of a broader construct are extracted by factor analysis. Kelley (1940) warned that variables resulting from factor analysis are not timeless, spaceless, populationless truth. Thus, it is recommended that evaluators use standardized inventories as a reference for developing instruments adaptive to their local treatment and audience.

Achievement

It is expected that the preceding attitudinal and cognitive change could result in improvement in Internet knowledge and skills, which may include the knowledge and skills of using web search, E-mail, newsgroups, threaded discussion, and FTP. Usually the criteria of achievement is tied to specific instructional content and therefore the instrument is locally developed. The strengths and weaknesses of locally-developed tests are discussed by Kubiszyn and Borich (1993).

Behavioral outcome

Finally, it is believed that test scores of Internet skills could predict the application of Internet skills reflected by observable behaviors . For instance, if the participants of a WBI are K-12 teachers, then incorporation of Internet technologies into teaching is expected. The variables may be the number of Web pages developed for class, the frequency of using the Internet as a source of teaching materials, the frequency of using web-based communication among peers. These variables could be measured by survey, interview and on-site observation.

Summary

Evaluation of learning is often limited by a "black-box" approach. However, there are many variables between the input and the output such as user patterns of the treatment and attitudinal changes prior to behavioral changes. Further, certain methodologies subsumed by a paradigm may be unquestioned by evaluators. To remediate these shortcomings, this proposed structural framework is an attempt to provide a guideline for evaluators to approach a research question in levels of abstraction and the stream of input-process-output.

References

Anderson, J. R. (1990). Cognitive psychology and its implications. New York : W.H. Freeman.

Fontana, D. (Eds.). (1984). Behaviorism and learning theory in education. Edinburgh: Academic Press.

Guglielmino, L.M. (1977). Development of the Self-directed Learning Readiness Scale, unpublished doctoral dissertation, University of Georgia, Athens, GA.

Guglielmino, P. J.; Murdick, R. G. (1997). Self-directed learning: The quiet revolution in corporate training and development. Advanced Management Journal, 62, 10-18.

Kelley, T. L. (1940). Comment on Wilson and Worcester's Note on Factor Analysis. Psychometrika, 5, 117-120.

Kellogg, R. T. (1995). Cognitive psychology. Thousands Oaks: Sage Publications.

Kubiszyn, T., & Borich, G. (1993). Educational testing and measurement: Classroom application and practice. New York: Harper Collins College Publishers. Kuhn, T. S. (1962). The structure of scientific revolutions. Chicago: University of Chicago Press.

Svanum, S., Chen, S. H., & Bublitz, S. (1997). Internet-based instruction of the principle of base rate and prediction: A demonstration project. Behavior Research Methods, Instruments, & Computers, 29, 228-231.

Swigger, K. M., Brazile, R., Lopez, V., & Livingston, A. (1997). The virtual collaborative university. Computers Education, 29, 55-61.

West, C. K., Farmer, J. A., & Wolff, P. M. (1991). Instructional design: Implications from cognitive sciences. Englewood Cliffs, New Jersey: Prentice Hall.

Yu, C. H., Jannasch-Pennell, A., DiGangi, S., & Wasson, B. (1999). Using On-line interactive statistics for evaluating Web-based instruction, Journal of Educational Media International, 35, 157-161.

Navigation

Index

Simplified Navigation

Table of Contents

Search Engine

Contact