Leech, Barrett, & Morgan (2015). Multiple Regression (Chapter 6).

Leech, N. L., Barrett, K. C., & Morgan, G. A. (2015). Multiple Regression (Chapter 6). In IBM SPSS for Intermediate Statistics: Use and Interpretation (5th edition, pp. 109–143). New York: Routledge.

“Multiple regression is one type of complex associational statistical method. … the general purpose of predicting a dependent or criterion variable from _several_ independent or predictor variables.” (p 109)

“There are several different ways of computing multiple regression that are used under somewhat different circumstances. … If the researcher has no prior ideas about which variables will create the best prediction equation and has a reasonably small set of predictors, then simultaneous regression, which SPSS calls Enter, is the best method to use.” (p 109)

“It is preferable to use the hierarchical method when one has an idea about the _order_ in which one wants to enter predictors and wants to know how prediction by certain variables _improves on_ prediction by others. Hierarchical regression appropriately corrects for capitalization on chance, whereas stepwise, another method available in SPSS in which variables are entered sequentially, does not.” (p 109)

[“Conditions of Multiple Linear Regression” (p 110) ..]

“… the _dependent_ or outcome variable should be an interval or scale level variable, which is normally distributed in the population from which it is drawn. The _independent_ variables should be mostly interval- or scale-level variables, but multiple regression can also have dichotomous independent variables, which are called dummy variables.” (p 110)

“A condition that can be extremely problematic as well is multicollinearity, … when there are high intercorrelations among some set of the predictor variables.” (p 110)

“… it is important to test for multicollinearity when doing multiple regression.” (p 110)

[“Assumptions of Multiple Linear Regression” (p 110) …]

“… the relationship between each of the predictor variables and the dependent variable is linear and that the error, or residual, is normally distributed and uncorrelated with the predictors.” (p 110)

[“Problem 6.1: Using the Simultaneous Method to Compute Multiple Regression” (p 110) …]

“… we will use the method that SPSS calls Enter (often called simultaneous regression), which tells the computer to consider all the variables at the same time.” (p 110)

“If variables are highly correlated (e.g., correlated at .50 or .60 and above), then one might decide to combine (aggregate) them into a composite variable or eliminate one or more of the highly correlated variables if the variables do not make a meaningful composite variable.” (p 111)

[“Output 6.la: Correlation Matrix” (p 111)]

[“Interpretation of Output 6.1a” (p 112) …]

“If predictor variables are highly correlated and conceptually related to one another, we would usually aggregate them, not only to reduce the likelihood of multicollinearity but also to reduce the number of predictors, which typically increases power. If predictor variables are highly correlated but conceptually are distinctly different (so aggregation does not seem appropriate), we might decide to eliminate the less important predictor before running the regression. However, if we have reasons for wanting to include both variables as separate predictors, we should run collinearity diagnostics to see if collinearity actually is a problem.” (p 112)

[“Output 6.1b: Multiple Linear Regression, Method= Enter” (p 113)]

[“Interpretation of Output 6.1b” (p 115) …]

“Multiple regression uses only the participants who have complete data for all the variables.” (p 115)

“… the adjusted R² is .36, meaning that 36% of the variance in _math achievement_ can be predicted from _gender_, _competence_, etc. combined.” (p 115)

“If the Tolerance value is low (< 1 – _R_²), then there is probably a problem with multicollinearity. In this case, since adjusted _R_² is .36, and 1 – _R_² is .64, then tolerances are low for _competence_ and _mother’s_ and _father’s education_ indicating we have a problem with multicollinearity.” (p 115)

“It is important to note that all the variables are being considered together when these values are computed. Therefore, if you delete one of the predictors that is not significant, it can affect the size of the betas and levels of significance for other predictors. This is particularly true if high collinearity exists.” (p 116)

“Another important aspect of the Coefficients table are the Correlations: zero order (bivariate), partial and part. The Partial correlation values, when they are squared, give us an indication of the amount of unique variance (variance that is not explained by any of the other variables) in the outcome variable (_math achievement_) predicted by each independent variable.” (p 116)

[SPSS example: “Collinearity Diagnostics” (p 116)]

“This table gives you more information about collinearity in the model. Eigenvalues should be close to 1, Variance Proportions should be high for just one variable on each dimension, and Condition Indexes should be under 15.” (p 117)

[“Problem 6.2: Simultaneous Regression Correcting Multicollinearity” (p 117)]

[“Output 6.2: Multiple Linear Regression with Parents’ Education, Method= Enter” (p 119) …]

[Scatterplot matrix (p 119) …]

“The top row shows four scatterplots (relationships) of the dependent variables with each of the predictors. To meet the assumption of linearity, a straight line, as opposed to a curved line, should fit the points relatively well.” (p 119)

“Dichotomous variables have two columns (or rows) of data points. If the data points bunch up near the top of the left column and the bottom of the right, the correlation will be negative (and vice versa). Linearity would be violated if the data points bunch at the center of one column and at the ends of the other column.” (p 119)

[SPSS example “Correlations” (p 120) …]

[Referring to 1-tailed Sig levels …]

“Note that all the predictors are significantly related to _math achievement_. Multiply Sig by 2 to get the two-tailed significance level.” (p 120)

[Referring to correlation values …]

“None of the relationships among predictors is greater than .25.” (p 120)

[SPSS example “Coefficients” table (p 122) …]

Leech_regression-coefficients_122

[“Interpretation of Output 6.2 continued” (p 122) …]

“In this example, we do not need to worry about multicollinearity because the Tolerance values are all close to 1.” (p 122)

“In the Sig. column of the table we can see that now all of the predictors are significantly contributing to the equation, and the Betas are higher than in Problem 6.1 so each of the variables has increased the amount of unique variance that it is explaining.” (p 122)

[“Example of How to Write About Output 6.2” (p 124) …]

“Multiple regression was conducted to determine the best linear combination of gender, grades in h.s., parents’ education, and motivation for predicting math achievement test scores. (Assumptions of linearity, normally distributed errors, and uncorrelated errors were checked and met.) The means, standard deviations, and intercorrelations can be found in Table 6.1. This combination of variables significantly predicted math achievement, _F_(4,68) = 14.44, _p_ < .001, with all four variables significantly contributing to the prediction. The adjusted R squared value was .43. This indicates that 43% of the variance in math achievement was explained by the model. According to Cohen (1988), this is a large effect. The beta weights, presented in Table 6.2, suggest that good grades in high school contributes most to predicting math achievement and that being male, having high math motivation, and having parents who are more highly educated also contribute to this prediction.” (p 124)

[“Problem 6.3: Hierarchical Multiple Linear Regression” (p 125) …]

“… hierarchical approach … This approach is an appropriate method to use when the researcher has a priori ideas about how the predictors go together to predict the dependent variable.” (p 125)

[“Output 6.3: Hierarchical Multiple Linear Regression” (p 126)]

[“Interpretation of Output 6.3” (p 127) …]

“This test is based on the unadjusted _R_², which does not adjust for the fact that there are more predictors, so it is also useful to compare the adjusted _R_² for each model, to see if it increases even after the correction for more predictors.” (p 127)

[“Example of How to Write About Output 6.3” (p 129) …]

“To investigate how well grades in high school, motivation, parents’ education, and math courses taken predict math achievement test scores, after controlling for gender, a hierarchical linear regression was computed (The assumptions of linearity, normally distributed errors, and uncorrelated errors were checked and met.) Means and standard deviations are presented in Table 6.1. When gender was entered alone, it significantly predicted math achievement, _F_(1, 71) = 1.16, _p_ = .009, adjusted _R_²= .08. However, as indicated by the _R_², only 8% of the variance in math achievement could be predicted by knowing the student’s gender. When the other variables were added, they significantly improved the prediction, _R_² change= .56, _F_(4,67) = 27.23, _p_ < .001, and gender no longer was a significant predictor. The entire group of variables significantly predicted math achievement, _F_(5,67) = 25.33, _p_ < .001, adjusted _R_² = .63. This is a large effect according to Cohen (1988). The beta weights and significance values, presented in Table 6.3, indicates which variable(s) contributes most to predicting math achievement, when gender, motivation, parents’ education, and grades in high school are entered together as predictors. With this combination of predictors, math course taken has the highest beta (.68), and is the only variable that contributes significantly to predicting math achievement.” (p 129)

[“Problem 6.4: Forward Multiple Linear Regression” (p 130) …]

“… forward approach, which is when SPSS adds variables one at a time by assessing which variable has the smallest probability of F (i.e., _p_ value) continuing until all variables are added that have a _p_ value equal to or less than .05. … in multiple regression we are looking at the combination of all of the predictor variables together: therefore, removing variables to find the best model may not be helpful as there are times that a variable that is not statistically significant can make a better overall model.” (p 130)

[From SPSS “Coefficients” table (p 133) …]

“Recall that the Tolerance values should be high.er than 1 – _R_². For these variables (1 – .459 = .541). All Tolerance values are larger than .541, therefore we do not have a problem with multicollinearity.” (p 133)

[“Interpretation of Output 6.4 continued” (p 133) …]

“The constant and the Unstandardized coefficients for a model can be used to predict an individual’s score on math achievement using the formula Y =a + bx + e where Y is the dependent variable, a is the constant [or intercept], b is the slope of one of the unstandardized beta coefficients for that independent variable (x), and e is the error term.” (p 133)

[“Example of How to Write About Output 6.4” (p 134) …]

“Forward multiple regression was conducted to investigate how well grades in high school, motivation, parents’ education, and gender predict math achievement test scores. (The assumptions of linearity, normally distributed errors, and uncorrelated errors were checked and met.) Means and standard deviations and intercorrelations are presented in Table 6.1 (see Problem 6.2). The beta weights and significance values for all models are presented in Table 6.4. After conducting forward regression, the model that included all predictor variables explained the most variance in math achievement, _F_ (4, 68) = 14.44, _p_ < .001, adjusted _R_² = .43. This indicates that 43% of the variance in math achievement was explained by this model. This is a large effect according to Cohen (1988). The equation for prediction a person’s math achievement score from the model was

Math achievement = -5.44 + 1.99 (Grades) – 3.63 (Gender) + 2.15 (Motivation) + .58 (ParentEd) + e

“All variables statistically significantly contribute to the final model. The beta weights for the model are presented in Table 6.4; these suggest that as motivation is increased by one unit, math achievement increases by 2.15, holding everything else constant.” (p 134) [Parentheses added to formula for clarity. – oki]

[“Problem 6.5: Backward Elimination Multiple Linear Regression” (p 135) …]

“… backward approach, which is when all the variables are added into the model, then are eliminated one by one, with the variable that has the largest probability of _F_ (i.e., _p_ value) removed until all variables have a _p_ value equal to or less than .10. … parsimonious …” (p 135)

[“Interpretation of Output 6.5” (p 137) …]

“The next table is a correlation matrix. Note that only math achievement would be statistically significantly correlated with _scholastic aptitude test-math_ with a two-tailed test.” (p 137)

“… Model Summary_R_ … is .82 (_R_²= .67), … adjusted _R_² is .64 … _scholastic aptitude test – math_ can be predicted from all the predictor variables combined. … second model … (R), using all the predictors except _competence_, is essentially the same, .82 (_R_² = .67), and the adjusted R² is .65. The third model with mosaic removed has a multiple correlation coefficient (_R_) of .81 (_R_² = .66), and the adjusted _R_² is .64. These values are extremely important when using Backward regression as after assessing which model(s) are statistically significant, we need to assess which model explains the most variance in the outcome variable. Model 2 explains the highest variance in _scholastic aptitude test – math_ (adjusted _R_² is .65). Yet, because we want the most parsimonious model, model 3 explains approximately the same amount of variance with an adjusted R² of .64. Therefore, model 3 looks like the best model.” (p 138)

[“Example of How to Write About Output 6.5” (p 140) …]

“Backward multiple regression was conducted to identify the most parsimonious combination of parents’ education, competence, pleasure, mosaic, and math achievement in predicting scholastic aptitude test – math. (The assumptions of linearity, normally distributed errors, and uncorrelated errors were checked and met.) Means and standard deviations are presented. The beta weights and significance values for all models are presented in Table 6.5. After conducting backward regression, the model that included parents’ education, pleasure, and math achievement was found to be the most parsimonious, _F_ (3, 69) = 44.41, _p_ < .001, adjusted _R_² = .64. This indicates that 64% of the variance in scholastic aptitude test – math was explained by this model. This is a large effect according to Cohen (1988). The equation for the model was

Scholastic aptitude test = 296.36 – 5.79 (ParentEd) + 22.16 (Pleaseure [sic]) + l l.86 (MathAch) + e

“Only math achievement statistically significantly contributes to the final model (_t_ = 10.70, p < .001). The beta weights for the model, presented in Table 6.5, suggest that as math achievement is increased by one unit, scholastic aptitude test increases by 11.86, holding everything else constant.” (p 140) [Parentheses added to formula for clarity. – oki]

 

See this page at https://kinasevych.ca/index.php