Reasons to not use PCA for feature selection

The only reason you need: PCA is not a feature selection algorithm<p>Is the author misunderstanding something very basic or are they deliberately writing this way for clicks and attention? I can see that they have great credentials so probably the latter? It&#x27;s a weird article.

Now that we&#x27;ve all said &quot;feature selection is not dimensionality reduction&quot; to our hearts&#x27; content, could we return to the point of the article?<p>Regardless of whether you&#x27;re doing feature selection or dimensionality reduction, the point remains that, if you&#x27;re doing supervised learning, PCA is just compressing your X space, without any regard to your y. It could be that the last principal component of X, containing only 0.1% of the variance, contains 100% of the correlation between X and y.<p>Using PCA for dimensionality reduction in a supervised learning context means throwing out an unknown amount of signal, which could be up to 100% of the signal.<p>Now for unsupervised, exploratory analysis, PCA is definitely a candidate, but there are plenty of often-better alternatives there too.

PCA can be construed as a loss compression of the data matrix. In fact, Eckart-Young theorem shows that this is an optimal compression (optimal w.r.t low rank, i.e. space to hold the values). In the language of OP, this shows the minimal energy loss for a given space constraint.<p>The key word is &quot;lossy&quot;. It may as well be that the loss&#x27;ed part had the signal for further classification down the pipeline. Or may be not. It depends on the case.

There was a recent discussion on PCA for classification that I had walked into, however, everyone had left the building when I joined. Since I run into this conceptual misunderstanding of PCA&#x27;s relevance to classification often, let me repeat what I said there.<p>The problem with using PCA as a preprocessing step for linear classification is that this dimensionality reduction step is being done without paying any heed to the end goal -- better linear separation of the classes. One can get lucky and get a low-d projection that separates well but that is pure luck. Let me see if I can draw an example<p><pre><code> ++++++ ++++++++++++++++ +++ --------- ----------------- ------- </code></pre> The &#x27;+&#x27; and &#x27;-&#x27; denote the data points of the two different classes. In this example the PCA direction will be the along the X axis, which would be the worst axis to project on to separate the classes. The best in this case would have been the Y axis.<p>A far better approach would be to use a dimensionality reduction technique that is aware of the end goal. One such example is Fisher discriminant analysis and its kernelized variant.

&gt; 1) Does linearly combining my features make any sense?<p>&gt; 2) Can I think of an explanation for why the linearly combined features could have as simple a relationship to the target as the original features?<p>the article provided a negative example where pca does not fit, but doesnt provide an example where it does or what pca is actually used for. I come away from this article thinking pca is useless.<p>what would be an example where 2) is true?<p>I cannot answer 2) without already having experience of what explanations there could possibly be. ( 2) is almost begging the question, at least pedagogically- pca is good when the features are good for pca)<p>When does linearly combining my features &quot;make sense&quot;? again, an example is not provided

Here&#x27;s a better idea, don&#x27;t use generalized statements but rather test your data first.<p>If your models fits well (and doesn&#x27;t overfit) after PCA, then go for it. If not, revisit.<p>PCA has its place, and as the other commenter said, sure, it&#x27;s not a feature selection algorithm. Or you can just feature select manually.

Much of the text is outside the viewport and cannot be scrolled to on mobile.

What would you recommend for feature selection in say, single-cell RNA seq studies (Typical dataset is ~10,000 x ~30000 (cells x genes) with &gt;90% of your table filled with 0s (which could be due to biological or technical noise)<p>PCA and UMAP are yes, dimensionality reduction methods, but are often seen as tools for feature selection.<p>See slide 61 Here: <a href="https:&#x2F;&#x2F;physiology.med.cornell.edu&#x2F;faculty&#x2F;skrabanek&#x2F;lab&#x2F;angsd&#x2F;lecture_notes&#x2F;10_lecture.pdf" rel="nofollow">https:&#x2F;&#x2F;physiology.med.cornell.edu&#x2F;faculty&#x2F;skrabanek&#x2F;lab&#x2F;ang...</a>

I followed the math but I don’t know ML… Do practitioners really use “energy” and “conservation of energy”? That just seems overly confusing.

PCA has a niche use-case. It&#x27;s more often harmful than not in my experience.

Hello,<p>I&#x27;m the author of the post. I&#x27;m slightly late to the party, but I&#x27;ll try to clarify a few misunderstandings.<p>First and foremost, the post deals with the following scenario too many data scientists find themselves in: &quot;I have (generated) a lot of features; let me do PCA and train my model using the top few principal components&quot;. This is a terrible idea and the post explains why.<p>Second, there seems to be a debate about &#x27;feature selection&#x27; vs. &#x27;feature construction&#x27; (or &#x27;feature generation&#x27;), and whether PCA is of the former or latter type. Here are the definitions I use in the whole blog.<p>Feature Construction is the process consisting of generating candidate features (i.e. transformations of the original inputs) that might have a simpler relationship with the target, one that models in our toolbox can reliably learn.<p>E.g. a linear model cannot learn a quadratic function. However, because a quadratic function of x is linear in [x, x^2], the feature transformation x -&gt; [x, x^2] is needed to make our quadratic function learnable by a linear model.<p>Check out this post for more details on what&#x27;s at stake during feature construction: <a href="https:&#x2F;&#x2F;blog.kxy.ai&#x2F;feature-construction&#x2F;" rel="nofollow">https:&#x2F;&#x2F;blog.kxy.ai&#x2F;feature-construction&#x2F;</a>.<p>Feature Selection is the process consisting of removing useless features in the set of candidates generated by feature construction. A feature is deemed useless when it is uninformative about the target or redundant.<p>Check out this post for a formulation of three key properties of a feature using Shapley values: feature importance, feature usefulness, and feature potential. <a href="https:&#x2F;&#x2F;blog.kxy.ai&#x2F;feature-engineering-with-game-theory-beyond-shap&#x2F;" rel="nofollow">https:&#x2F;&#x2F;blog.kxy.ai&#x2F;feature-engineering-with-game-theory-bey...</a><p>In the scenario the blog post deals with (i.e. &quot;I have (generated) a lot of features; let me do PCA and train my model using the top few principal components&quot;), data scientists do both feature construction (full PCA, i.e. projecting the original input onto eigenvectors to obtain as many principal components as the dimension of the original input) AND feature selection (only selecting the first few principal components with the highest eigenvalues).<p>When the goal is to predict y from x, using PCA for either feature construction OR feature selection is a bad idea!<p>For feature construction, there is nothing in PCA that will intrinsically guarantee that a linear combination of coordinates of x will have a simpler relationship to y than x itself. PCA does not even use y in this case! E.g. Imagine all coordinates of x but the first are pure noise (as far as predicting y is concerned). Any linear combination of x will just make your inputs noisy!<p>For feature selection, even assuming principal components make sense as features, principal components with the highest variances (i.e. corresponding to the highest eigenvalues) need not be the most useful for predicting y! High variance does not imply high signal.