There are a couple of assumptions inherent to strict factor models. The first is that the residual return is assumed to be uncorrelated with each of the factor returns. This does not restrict the set of possible models as much as it may seem. The factor exposures can be adjusted to achieve the condition where the residuals are uncorrelated with the factor returns. In fact, in simple settings using historic data, this can be achieved with multiple regression procedures. The key assumption of a linear factor model is that the residual for one asset’s return is uncorrelated with the residual of any other asset. This means that the only correlations among asset total returns are those that arise due to their exposures to the factors. The residual component of an asset’s return is assumed to be unrelated to the residual of any other asset, and hence totally specific to that asset. In other words, the risk associated with the residual return is specific to or idiosyncratic to the asset in question. This assumption makes a linear factor model powerful in the sense that it rules out many possible combinations of outcomes. But this power comes at a cost. The more restrictive a model is, the greater the chance that it may be wrong. For this reason, if using a strict factor model, it is important to try to account for the most important sources of correlations by including a sufficient number of factors, while also attempting to include only the most important ones. That said, there’s also value to simplicity in modeling, since the goal is always to model signal and not noise.