Simple Linear Regression

Simple Linear Regression

Simple linear regression is a statistical method used to model the relationship between two variables. One variable is independent (predictor), and the other is dependent (response). The goal is to find a linear equation that best predicts the dependent variable based on the independent variable.

Understanding the Components

In simple linear regression, you will encounter several key components:

• Independent Variable (X): The variable you manipulate or control. It is also known as the predictor variable.
• Dependent Variable (Y): The variable you measure or observe. It is the outcome you want to predict.
• Regression Equation: The equation representing the relationship between X and Y, typically in the form Y = a + bX, where:

a = y-intercept (the value of Y when X = 0)
b = slope of the line (the change in Y for a one-unit change in X)

Collecting Data

Before conducting a regression analysis, you need to collect data for both the independent and dependent variables. For example, let’s say you want to study the relationship between hours studied (X) and exam scores (Y) for a group of students.

Example Data Set

Consider the following data collected from five students:

Calculating the Regression Equation

To find the regression equation, you need to calculate the slope (b) and the y-intercept (a). The formulas for these are:

b = (NΣXY - ΣXΣY) / (NΣX² - (ΣX)²)
a = (ΣY - bΣX) / N

Where:

• N = number of data points
• ΣXY = sum of the product of X and Y
• ΣX = sum of X values
• ΣY = sum of Y values
• ΣX² = sum of squared X values

Step 1: Calculate the Sums

First, we need to calculate the necessary sums:

ΣX = 2 + 3 + 5 + 7 + 8 = 25
ΣY = 65 + 70 + 80 + 85 + 90 = 390
ΣXY = (2×65) + (3×70) + (5×80) + (7×85) + (8×90) = 130 + 210 + 400 + 595 + 720 = 2055
ΣX² = (2²) + (3²) + (5²) + (7²) + (8²) = 4 + 9 + 25 + 49 + 64 = 151

Step 2: Plug Values into the Slope Formula

Now, substitute the sums into the slope formula:

N = 5 (number of students)
b = (5×2055 - 25×390) / (5×151 - 25²)
b = (10275 - 9750) / (755 - 625)
b = 525 / 130 = 4.0385 (approximately)

Step 3: Calculate the Y-Intercept

Next, we calculate the y-intercept using the y-intercept formula:

a = (390 - 4.0385×25) / 5
a = (390 - 100.9625) / 5
a = 289.0375 / 5 = 57.8075 (approximately)

Final Regression Equation

The regression equation is:

Y = 57.8075 + 4.0385X

Interpreting the Results

In the equation Y = 57.8075 + 4.0385X:

• The y-intercept (57.8075) indicates the predicted exam score when no hours are studied.
• The slope (4.0385) suggests that for every additional hour studied, the exam score increases by approximately 4.04 points.

Assessing the Fit of the Model

To determine how well the regression model fits the data, you can calculate the coefficient of determination, denoted as R². This value indicates the proportion of the variance in the dependent variable that can be explained by the independent variable.

Calculating R²

R² is calculated as follows:

R² = 1 - (SS_res / SS_tot)

Where:

• SS_res = residual sum of squares (the sum of squared differences between observed and predicted values)
• SS_tot = total sum of squares (the sum of squared differences between observed values and the mean of observed values)

Step 1: Calculate the Mean of Y

First, calculate the mean of the observed Y values:

Mean of Y = ΣY / N = 390 / 5 = 78

Step 2: Calculate SS_tot

Now, calculate SS_tot:

SS_tot = Σ(Y_i - Mean_Y)²
SS_tot = (65 - 78)² + (70 - 78)² + (80 - 78)² + (85 - 78)² + (90 - 78)²
SS_tot = 169 + 64 + 4 + 49 + 144 = 430

Step 3: Calculate Predicted Y Values

Using the regression equation, calculate the predicted Y values:

When X = 2: Y = 57.8075 + 4.0385×2 = 65.8845
When X = 3: Y = 57.8075 + 4.0385×3 = 69.9230
When X = 5: Y = 57.8075 + 4.0385×5 = 77.0000
When X = 7: Y = 57.8075 + 4.0385×7 = 84.0765
When X = 8: Y = 57.8075 + 4.0385×8 = 88.1150

Step 4: Calculate SS_res

Now, calculate SS_res:

SS_res = Σ(Y_i - Y_pred)²
SS_res = (65 - 65.8845)² + (70 - 69.9230)² + (80 - 77.0000)² + (85 - 84.0765)² + (90 - 88.1150)²
SS_res = 0.7744 + 0.0059 + 9.0000 + 0.8460 + 3.5540 = 14.1803

Step 5: Calculate R²

Finally, substitute SS_res and SS_tot into the R² formula:

R² = 1 - (14.1803 / 430) = 1 - 0.0329 = 0.9671 (approximately)

This means that approximately 96.71% of the variance in exam scores can be explained by the number of hours studied.

Conclusion

Simple linear regression is a powerful tool for predicting outcomes based on relationships between variables. By understanding how to calculate and interpret the regression equation, as well as assessing the model fit with R², you can effectively use this method to analyse data.

Check your understanding

1. What is the formula for the slope in simple linear regression?
2. How do you interpret the y-intercept in the regression equation?
3. What does an R² value of 0.85 indicate about the model?
4. List the steps to calculate the regression equation from a data set.