Checklist

The following questions can be used to help you design and implement your research study. The answers are provided as examples. They are by no mean the model answers. Also, you don't have to answer every question. Some questions may not be relevant to your study.

Theory

• What is the supporting instructional/psychological theory? What previous research can support the theory?

  The supporting theory is Anderson's Adaptive Character of Cognition. The theory proposes that human cognition is a schemata that consists of numerous nodes. Non-linear learning strengthens the association of nodes, thereby form a coherent schemata. Research on HyperCard and Authorware supports this theory.

• What is the treatment program?

  A website with various hypertext features.

• What is/are the media propertie(s)?

  Hypertext in World Wide Web.

• What is/are the observed variable(s) to represent the media property(ies)?

  Use of websites. The usage will be checked by user access log.

• What is/are the mental construct(s)?

  Non-linear thinking.

• What is/are the behavioral outcomes to represent the mental construct(s)?

  Problem solving that requires extensive search of scattered information such as Web search.

• Could the media property(ies) and the mental construct(s) form a direct causal relationship? If not, is/are there more mediating variable(s)?

  The relationship is direct. There is no mediating variable.

Hypothesis

• What is your null hypothesis?

  There is no significant difference in problem solving skills between learners who use hypertext and those who don't.

• What is your alternate hypothesis?

  There is a significant difference in problem solving skills between learners who use hypertext and those who don't.

• Can the hypothesis be falsified? Will it lead to a meaningful result?

  Either one of the hypotheses can be wrong. According to Karl Popper, this risky position will lead to findings that carry scientific values.

Design of experiment

• Is your study experimental, quasi-experimental, or non-experimental?

  The study is experimental because randomization will be used. It will be a factorial design with two factors. The between subject factor has two levels: hypertext group and conventional learning group. The within subject factor has two levels, too: pretest and posttest.

• What are the potential threats against the validity of your study? How can you counteract those threats?

  The study is subject to the threats of instrumentation and subject selection mentioned by Cook and Campbell. Inter-rater reliability will be used to identify measurement errors. Randomization will be used as a counter measure against selection bias. The attributes of groups will be examined to avoid the Simpsonšs paradox.

• Do you want to study many interaction effects or to have a small study? Which experimental design will you use to optimize the study?

  There is only one interaction effect and thus a full-factorial design is acceptable.

Population and sample

• What is the hypothetical population to which you will make inference?

  All college students who have basic computer skills.

• What is the accessible population from which you will draw samples?

  ASU's EMC graduate students who have basic computer skills.

• What is your sampling method?

  A mixture of convenience sampling and purposive sampling. I will go to several classes to recruit subjects who are qualified to be the users of the treatment (e.g. basic computer skills).

• How will you choose the alpha level, effect size, power level, and sample size?

  Given that the alpha level is .05, the effect size is .25, and the power level is .80, the desirable sample size should be 128. However, I can get only 80 participants.

• If you do not conduct a power analysis or your sample size is less than what the power analysis suggests, what remedy will you use?

  Resampling.

Measurement

• Will you establish the content validity, the criterion validity, or/and the construct validity?

  The content validity will be established by a panel of content experts. The construct validity will be verified by factor analysis.

• Which reliability estimation method will you use?

  Cronbach coefficient Alpha for internal consistency and Kappa coefficient for inter-rater reliability.

• If your instrument is a performance test, which item difficulty method will you use?

  Latent trait model item analysis.

• If you conduct a test item bias analysis, which method will you use?

  MH procedure.

• How many constructs does your instrument measure? Can factor analysis support the factor structure?

  There are five different domains in the problem solving test. Factor analysis confirms that there are five subscales in the test and they are not strongly correlated.

Analysis

• If you use parametric tests, can the nature of the study and data structure meet the parametric assumptions? If not, what remedy will you use?

  The study is experimental and thus an ANOVA model is appropriate. However, although outliers are removed, the data cannot meet the assumptions of normality and homogeneous variances. According to Monte Carlo simulations, my sample size is not large enough to make the test robust against the violations. ANOVA with data transformation and resampling will be used.

• By resampling, can you verify the parametric test results?

  The bootstrapping results confirm the parametric test results.

• By the Bayesian inference, can you infer the treatment effectiveness?

  The Bayesian inference concludes that the hypertext treatment is more effective in enhancing problem solving.

• By exploratory data analysis, can you find an interesting pattern of the data?

  By brushing and linking, it is found that the scores of female humanity majors tend to center around the post-test means.

Interpretation

• Are noises such as threats against validity of experiment and measurement errors isolated from the study? What other noises remain in the study?

  The measurement errors are acceptable but the pre-test effect is not controlled.

• Are the predicted relationships among variables confirmed by the statistical methods?

  The association between the dependent and independent variables are confirmed by the ANOVA with data transformation. The Bayesian inference and bootstrapping concurred with the finding.

• Can you establish a cause and effect relationship among variables?

  Because the study is experimental in nature, the noises are isolated and the association is confirmed, the causal inference is established.

• Can you generalize the findings to the entire population?

  The subjects are randomized and their characteristics meet the requirements of the target audience, thereby the findings can be generalized.

Go up to the main menu

Navigation Other courses Simplified Navigation Table of Contents Search Engine

Contact me