347
Height, Weight, Waist size, and Chest size. A correlation matrix for these variables is likely to show large positive correlations between High school GPA, IQ, and SAT scores. Similarly, Height, Weight, Waist and Chest measurements will probably be positively correlated. So, the question is whether High school GPA, IQ, and SAT scores are related because of some underlying, common factor. The answer, of course, is yes, because they are all measures of intelligence. Similarly, Height, Weight, Waist, and Chest measurements are all related to physical size. So the conclusion is that there are only two underlying factors that are being measured by the eight variables, and these factors are intelligence and physical size. These common factors are sometimes called latent variables. Since “intelligence” is an abstract concept, it cannot be measured directly: instead, measures such as GPA, IQ, etc. are used to estimate the intelligence of an individual.
In the simple example presented above, it’s not too difficult to isolate the pattern of correlations that link the variables in the two groups; but when you have hundreds of variables and there are multiple underlying factors, it is much more difficult to identify the factors and the variables associated with each factor.
The purpose of Factor Analysis is to identify a set of underlying factors that explain the relationships between correlated variables. Generally, there will be fewer underlying factors than variables, so the factor analysis result is simpler than the original set of variables.
Principal Component Analysis is very similar to Factor Analysis, and the two procedures are sometimes confused. Both procedures are built on the same mathematical techniques. Factor Analysis assumes that the relationship (correlation) between variables is due to a set of underlying factors (latent variables) that are being measured by the variables.
Principal Components Analysis is not based on the idea that there are underlying factors that are being measured. It is simply a technique for finding a linear combination of the original variables that produce orthogonal (uncorrelated) variables that explain the maximum amount of variance in the original variables. It is often used to reduce the number of variables while retaining most of the predictive power.
The goal of PCA is to rigidly rotate the axes of an n-dimensional space (where n is the number of variables) to a new orientation that has the following properties:
1. The first axis corresponds to the direction with the most variance among the variables, and subsequent axes have progressively less variance in their direction.
2. The correlation between each pair of rotated axes is zero. This is a result of the axes being orthogonal to each other (i.e., they are uncorrelated).
PCA is performed by finding the eigenvalues and eigenvectors of the covariance or correlation matrix. The eigenvectors represent a linear transformation from the original variable coordinates to rotated coordinates that satisfy the criteria listed above. For example, if you have variables X1 through Xn. Then the eigenvector components would be: