> Feature engineering is a mechanism of creating new levels of abstraction in knowledge representation because each (non-trivial) feature extract and makes explicit some piece of knowledge hidden in the data. It is almost precisely what deep learning is intended for. In this sense, feature engineering does what hidden layers of a neural network do or what the convolutional layer of a neural network does<p>Very intriguing and thoughtful statement. I hadn't ever thought of it that way.
Lambdo is a workflow engine which simplifies data analysis by combining in one analysis pipeline<p>* Feature engineering and machine learning: Lambdo does not distinguish them and treats them as data transformations<p>* Model training and prediction: both feature definitions and ML models can be trained as part of one workflow<p>* Table population and column evaluation: workflow consists of nodes of these two types. This makes it similar to Bistro: <a href="https://github.com/asavinov/bistro" rel="nofollow">https://github.com/asavinov/bistro</a><p>Lambdo is intended for the following use cases:<p>* Numerous derived features with parameters derived from the data<p>* Regular re-training is required by using the same features as those to be used during prediction<p>* Time series analysis because it is where the quality of derived features is especially important<p>* Customization via user-defined Python functions
I definitely see the need for packages like this. So much of a data scientist's time is spent on feature engineering, but there are relatively few tools out there that are trying to improve that step in the process compared to tools for the modeling step.<p>I see this tool as a something that can help with the deployment piece of the feature engineering. As things stand, it's "easy" to package and deploy modeling code, but much harder to package up your feature engineering workflow. In part because there is no agreed upon standard for developing feature engineering pipelines.<p>I'd be curious how this could be combined with a library like Featuretools (<a href="http://github.com/featuretools/featuretools/" rel="nofollow">http://github.com/featuretools/featuretools/</a>) which helps automate the discovery of features, but currently has less functionality related to deployment.<p>(full disclosure: I work on Featuretools)
Things I always want when looking at something new:<p>- Where does it sit in relation to other data science components?<p>- What does it integrate with, and what is it agnostic to?<p>- Smallest self-contained use case? I see some examples in the repo readme but they are not self-contained so its harder for me to imagine its use