What you're asking for looks a lot like a kaggle challenge. You're asking if someone can use xgboost (or linear regression). It's the <i>last</i> ten per cent of a data science problem, if that.<p>The hard parts of data science include the following:<p><pre><code> - choosing the right input data (rather relying on regularisation)
- figuring out what the consequences are if you're wrong in a specific way, and avoiding the bad cases
- wrangling your data into a nice CSV format
- handling missing data
- spotting biases in your data collection methodology
</code></pre>
I'd expect a graduate to know about regression. For anyone else, this wouldn't help me assess their skills.
Working data scientist here. As many have said, this is, effectively a Kaggle challenge. I mean honestly at this point, I don't care, at all, how well someone can predict anything to be a data scientist - there is very little correlation between that and how good of a data scientist they are.<p>Tools to hire data scientists and going to continually fail until they realize that the interesting, hard part of being a data scientist is closer akin to a business lead (which can't really be tested in 60 minutes).<p>Concrete feedback:<p>- You ask for writing and descriptions on why a model was chosen, why features matters - are you grading this automatically? That would be a feat.<p>- The task is waaay to easy (even if you do believe there is a market for identifying people who can predict well).<p>- Python is overly limited. Why not SQL or R?
I am in the camp to think that this notebook judges data scientists in a way that soon will be obsolete.<p>If I'm given this clean dataset with all of the features properly set in columns and data types labeled, I could spin up Azure or Google Cloud's ML capabilities and have them run gridsearch and optimize my model.<p>To test data scientists, it seems like it's generally falling more into the buckets of people that can pull analytics and query databases to create the datasets and features OR people that can build the infrastructure to serve models, engineer pipelines, etc...<p>FYI though we're working on this now at <a href="https://www.interviewquery.com" rel="nofollow">https://www.interviewquery.com</a> to try to start creating suitable tests to assess data scientists without having them do 10+ take homes every month.
I'm a data scientist (for an MLB team that will win the WS this year!) and I love this. Of course this isn't a whole end to end evaluation platform. But we will get 300-500 applications for a position sometimes, and often folks have no business applying and this would be a great way to filter out some of the noise. Great job!
This was pretty cool. For fun, I tried to get the best possible score I could, using XGBoost, without any feature engineering and achieved a MAE of 0.042422154541399665.
I think this is good as a way to filter people out, but not as a way to rank people to find the best.<p>I would want to see a short script to clean and predict a dataset, plus a small description of why choices were made.<p>Wouldn’t care much about the performance of the model.
I think this is great as a self-assessment tool, especially for beginners.<p>It would work great with other learning tools, like MOOCs, datacamp, dataquest.io, as part of an overall data science learning process.<p>I'm more skeptical of its ability to help companies select candidates, but I could be very wrong about this and if I am then it's a huge win, so thanks for developing it.<p>I am super interested in seeing how you all develop this in the future,there's a lot of potential here. Is there a data science specific mailing list I can sign up for? I honestly, have zero interest in hiring for other roles so I am not going to sign up for a general mailing list.
I suppose I am in the minority but I thought it was a pretty good FizzBuzz challenge for DS.<p>In fact I'd say it is a bit aggressive for a 60 minute challenge.<p>Quite a bit of data wrangling is expected to complete modeling on all columns. Some regex knowledge would help here too (for example for wrangling internet_users column)<p>What was the idea behind asking for 20 most important features when we have 16 columns? Is it expected to do some feature engineering?<p>Disclaimer: I teach Python and basic Data Science to adults and I'd say most people would struggle to complete this in 60 minutes including myself.
Data scientists are fundamentally problem solvers.<p>The best way to assess technical problem solving is a structured hackathon. That is, to be given<p>* a problem with multiple subproblems and solution milestones<p>* with both objective and subjective criteria<p>* freedom of tools<p>* a "junkyard" of resources<p>* a fixed amount of time for each deliverable of the problem<p>And then you observe the process and the results.<p>For data science, the subproblems should be:<p>* requirements gathering / understanding the problem<p>* data acquisition, prep, and analysis<p>* refinement of requirements / communication<p>* feature engineering<p>* modeling<p>* presentation / storytelling / viz
Here are my high level thoughts after a quick look at the question and some clarification in thread below.<p>- a single question is difficult to evaluate. "Answering a business question" is at the very end, usually, of a bunch of exploratory steps<p>- 60 min is reasonable but not much time to evaluate real work. You either need to expand the time (also a problem, for interviewing) or allow scoring of "what I'd do next"<p>- tooling familiarity is going to be a huge factor with short time. Are you testing general knowledge or environment knowledge?<p>- too focused on models, too "kaggle-like". That covers about 20% of the skills and job.<p>Here are the sorts of things I look for. Do they understand:<p>1. How to verify & validate data, clean inputs, handle coding errors and ELT type issues<p>2. How to evaluate data set issues like bias, missing data and outliers, and account for that (and when you can't)<p>3. (situational) How their infrastructure works an what they need it to do (e.g. for distributed training, if appropriate). How to use it effectively.<p>4. How to control data and code throughout lifecycle, so you don't waste time and experiments<p>5. How to choose between approaches and models<p>6. How to evaluate performance rigorously<p>7. How to monitor performance over time<p>but here is the kicker<p>8. How do you know you are trying to solve the right problem?<p>For junior people, the emphasis will be on the earlier points. For senior people the last point is key.<p>Your question partially addresses some of the early points only.<p>Off the top of my head suggestions.<p>- Have separate stages. Cleanup & verification can have objective and subjective issues (missing & corrupt data? Outliers?)<p>- Don't focus too much on modeling, it's the least interesting part.<p>- Allow different toolsets possibly (e.g. R)<p>- Initial cleanup/eval stage on a CSV, but following stage pull from SQL?<p>- Possibly allow multiple inference choices from same or a few data sets. Give a short list of things the "business" is interested in, they pick and describe why<p>- good idea to focus a bit on producing one/two graphics/tables to communicate to a lay audience.<p>- more focus on verification<p>- add a validation discussion requirement. How are you going to know what you did is worth doing?<p>- add a "next things I would try/do"<p>The latter is going to be text heavy but no way to avoid this unless there is a follow on voice/personal interview.<p>There isn't any way you are going to auto score this stuff reliably, so that's probably ok. Consequence is your evaluators are going to actually have to be good at this.
Data scientists (in my opinion as one) should be spending most of our time listening and talking to our colleagues, our clients, our peers, teachers, and - last of all- to the data. Last because we dive into the data searching for things, answers to questions, people asked for and that other people inform our journey and our methods.<p>Counter-intuitive in regards to the phrase but good ds is people work first and data work like 9th.