Deviation scores
Using centered scores, also known as deviation
scores, is another way to avoid collinearity in regression which
involves interaction. A centered score is simply the result of
subtracting the mean from the raw score (X - X_mean). For the ease of
illustration, the following example will use only two subjects and two
variables:
|
X1
|
X2
|
X1X2
|
| Sandy |
8 |
4 |
32
|
| David |
5 |
8 |
40
|
|
If we plot the above data into subject space, we will get two short
vectors and a very long vector. There are two problems. First, the
scales of X1, X2 and X1
* X2 are very different (The problem of
mismatching scales is addressed in the write-up Centered-scores regression).
Second, there exists collinearity, of course.
To overcome these problems, you can apply
centered scores into the regression model. The following table shows
how the raw scores are transformed into centered scores.
|
 |
|
X1
|
C_X1=X1-X1_MEAN
|
X2
|
C_X2=X2-X2_MEAN
|
C_X1 * C_X2
|
| Sandy
|
8
|
8 - 6.5 = 1.5
|
4
|
4 - 6 = -2
|
1.5 * -2 = -3
|
| David
|
5
|
5 - 6.5 = -1.5
|
8
|
8 - 6 = 2
|
-1.5 * 2 = -3
|
|
When we plot the centered
scores into subject space, we can find that the scales of all vectors
are closer to each other. In addition, collinearity is no longer a
threat. The interaction term is orthogonal (90 degree) to both centered
X1 and centered X2.
|
These movie clips use a physical body as a metaphor to
illustrate an uncentered model and centered model.
You must have a sound card to hear the audio.
The SAS code to run a regression with centered scores is as the
following:
DATA ONE;
INPUT Y X1 X2;
....
PROC MEANS; VAR X1 X2;
OUTPUT OUT=NEW MEAN=MEAN1-MEAN2;
DATA CENTER; IF _N_ = 1 THEN SET NEW; SET ONE;
C_X1 = X1 - MEAN1;
C_X2 = X2 - MEAN2;
C_X1X2 = C_X1 * C_X2;
PROC GLM; MODEL Y = C_X1 C_X2 C_X1X2;
The preceding code works fine with a small dataset (e.g. a few hundred
observations). However, The following revised code is more efficient
for a large dataset (e.g. thousands of observations):
DATA ONE;
INPUT Y X1 X2;
....
PROC MEANS; VAR X1 X2;
OUTPUT OUT=NEW MEAN=MEAN1-MEAN2;
DATA CENTER/VIEW=CENTER;
IF _N_ = 1 THEN SET NEW(KEEP = MEAN1-MEAN2); SET ONE;
C_X1 = X1 - MEAN1;
C_X2 = X2 - MEAN2;
C_X1X2 = C_X1 * C_X2;
PROC GLM DATA=CENTER; MODEL Y = C_X1 C_X2 C_X1X2;
First, using a view instead of creating a new dataset can save memory
space. A view works like a dataset except that it creates the dataset
only and only if the data are read. Second, because only the means will
be used later, other unused variables can be dropped and only the means
are kept. Again, it can save memory space to make the program more
efficient.
Application
In a project entitled Eruditio, I applied the
combination of centered scores and partial orthogonalization into the
regression model. In this project we delivered instruction on the Web.
It is hypothesized that the more the students use the Web-based
instruction, which is indicated in the user activity log, the better
the performance on the subject matter is. The variables are listed as
the following:
- Gain--Gain scores between
pretest and posttest (outcome variable)
- Page--The number of pages
accessed by learners as recorded in the user log (predictor)
- Time--Total time learners spent
in our website as recorded in the user log (predictor)
- Interaction--Interaction between
Page and Time (predictor)
Interaction is definitely collinear with Page or Time. And the degree
of association between Page and Time is also intermediate. To construct
a legitimate regression model free of collinearity, I transformed the
independent variables into
centered scores of Page and Time, and the residuals of the interaction
of centered Page and centered Time.
Orthogonal, Independence and Uncorrelated
|
In most cases, "orthogonality," "independence, "and "Uncorrelated" are
interchangeable. However, there is a slight difference among them.
"Uncorrelated" is when two variables are not related and information
about one of them cannot provide any information about the other.
"Orthogonality" means that the two variables provide non-overlapping
information. In some cases, two variables may be orthogonal but
correlated. For example, let X = {-1, 1}. A new variable Y is derived
from X * X, which is {1, 1}. Y is definitely correlated to X. But when
you plot the data in subject space, you see two orthogonal vectors as
shown on the right panel.
When a new variable created by raising power of
another variable is included in a regression model, this is called
polynomial regression. Polynomial regression will be discussed in the
next section.
|
 |
Rodgers, Nicewander, and Toothaker (1984) explained the difference
among the three concepts using both algebraic and geometrical
approaches. These explanations are complicated but they clearly
illustrated the relationships among the three concepts in the following
graph:
Navigation
Index
Simplified Navigation
Table of Contents
Search Engine
Contact
