If one were to use Hacker News as their only source of information, it would seem that machine learning is a very overrated topic. There is something related to it on HN's front page almost every day. This proliferation of courses, resources, books and startups would hint that machine learning is becoming more and more accessible to the average programmer and that the market is on track to getting saturated quickly. Is this the current trend? If yes, is it limited to the US? What about the machine learning scene in Europe? Maybe someone here could provide some perspective.
Speaking for NYC, but I imagine silicon valley is similar.<p>The supply-demand dynamics have changed a lot in the last couple years. I'd roughly break it out into two groups: people with work experience + strong software development skills, and those without. The first group is in higher demand than ever, and tend to add a lot of value to companies that really need it.<p>The second group has gotten extremely crowded, especially from STEM graduates - usually with a masters or phd - who have completed MOOCs or bootcamps. Supply keeps growing while demand is flat or shrinking (especially as executives get burned by "data scientists" who don't know how to help them build things of value). There's a huge crunch here; a lot of people I know in this group have been searching for jobs for months, eventually settling for a low quality job or giving up entirely :(
I hire machine learning engineers and data scientists. In my opinion there is a great shortage of truly qualified machine learning engineers. A lot of people are entering the market with a general knowledge of machine learning tools. These people should be considered analysts or product data scientists. When it comes to people that can build machine learning systems that work at scale, they are very rarely available for hire and often are the subject of bidding wars by multiple companies. The key difference is whether the candidate truly understands the mathematical and statistical basis of machine learning, has the programming skills to execute their ideas, and is able to write code that can be used in large scale production systems and can be leveraged by others.
I've been working in DS role for a few years now in NYC - and I definately feel the role is more valued on the east coast over SV. SV has a focus on consumer facing applications that are in many ways fancy CRUD. DS roles have thier place but aren't the core of the business. East coast has a b2b / infobroker focus where DS is the product. Media (especially adtech), finance, government consulting are over on this coast.<p>I think you also need to not confuse the growing ease of machine learning tools with the role becoming more accessible. There is a wide gap between tooling and knowledge to use those tools appropriately and creatively.<p>And may I never write another HN comment on my cell phone again.
Stack overflow salary calculator shows a significant 50% premium over Developer salaries, all other things remaining the same. [1] Even though in my opinion the tool is flawed and actually significantly underestimates (stackoverflow underpays) salaries in SV/NYC. It is still a good indicator.<p>The major issue is that Data Scientist is a very fuzzy term with it being applied to everyone from undergraduates with Stats degree and to those with PhDs and papers at KDD/ICML/NIPS/CVPR.<p>However rather than doing a Frontend or Mobile developer coding bootcamp, a data science bootcamp is likely to lead to more transferable skills in case you wish to get an MBA etc.<p>[1] <a href="http://stackoverflow.com/company/salary/calculator?p=7&e=1&s=2.5&l=1" rel="nofollow">http://stackoverflow.com/company/salary/calculator?p=7&e=1&s...</a>
Currently, Gartner analysts place ML at the "peak" of its Hype Cycle for Emerging Tech [0] with a runway of 2-5 years for mainstream adoption.<p>[0] <a href="http://www.gartner.com/newsroom/id/3412017" rel="nofollow">http://www.gartner.com/newsroom/id/3412017</a>
The startup I work at really favors their data scientists, though I am not one of them (I'm a frontend guy). The CEO and CTO pretty much keeps a personal eye on those guys' work.<p>Right now however the theme I've heard from the higher ups has been profitability, and this applies to all tech companies in general. Easy capital is gone and now companies are in the spotlight for not making profits.<p>So at least from my company's perspective, it's not that data science is saturated, it's that we're trying to not break the bank and hire too much.
I have only anecdotal experience (I live in Warsaw, but do contracts mostly for Poland, UK and US).<p>General data science is in need. I can get contracts easily, I know that people looking for competent people need to wait; especially as it is a skill much harder to pick than, say, front-end web dev (unless someone starts from a highly quantitive background like physics, modelling in biology, etc). My general impression are:<p>- ML (especially practical one, like logistic regression and random forest) is often integral parts of many data analyses (or at least a plus),<p>- there are not as many jobs solely focused on ML; and if so, often they require some specialistic expertise,<p>- and even less only for deep learning (also, for DL there is relatively high threshold for having skills at "hireable" level).<p>Some of my tips on how to learn data science: <a href="http://p.migdal.pl/2016/03/15/data-science-intro-for-math-phys-background.html" rel="nofollow">http://p.migdal.pl/2016/03/15/data-science-intro-for-math-ph...</a> (on purpose I put the emphasis on general data exploration/analysis before machine learning).
Like about everything on HN... You're either in the Silicon Valley or it doesn't apply to you.<p>In my opinion, you could start by defining what is a data science, a quant, or a machine learning job. Because that's not clearly defined. It means different jobs to a lot of people, jobs that are all hard to learn and absolutely NOT interchangeable.
We hire applied ML/AI specialists. For me it's not just an understanding of mathematical concepts, but also being able to apply new ideas to new problems.<p>This depends quite a bit on critical thinking, a good fundamental ability to analyze a problem and understand its parameters, then manage the logical operations required to deliver the feature and solve the problem.<p>As for why I think it's on HN every day: I also like to think of an innovation pipeline happening something like this:<p><pre><code> [---------explore------|----------exploit-----------]
,->developers -> engineers/scientists -> data scientists->--,
/----------<----------------<--------------------<----------/
</code></pre>
We're now in some sort of refinement cycle of innovation, where the current medium has been saturated on some level and there is a lot of push to mine value from the discoveries.
I'm an undergrad at a big university known for CS in Canada. The CS program here has several possible 'focuses'; 4 of 9 are related to ML/AI directly (computer vision, NLP, AI, scientific computing). 2 others require AI/ML/NN courses.<p>The bias might stem from the fact that we have some huge names in AI doing research here, but the data points seem clear (we say undergraduate education is slow to catch on, right?): the topic as a whole isn't overrated.<p>However, there seems to be a lack of understanding by people working in tech of the differences (in uses, theory, implementation) between ML, AI, NN, DL, etc. This might stem from a lack of understanding of the foundations of these topics (ex: statistics, vector calculus) or simply because we can abstract a lot of this away (ex: TensorFlow).
In my limited experience, there's a difference between a data scientist who can process data given data and a set of questions about it, and a data scientist who can figure out what data you need, and the questions that need to be answered.<p>I think making the transition from the first role to the second role comes with experience, both with the toolsets, and thinking about the problem as a whole.
I am the Chief Data Scientist of Dice.com. If you are interested in working as a junior Data Scientist, and are smart and hard working, please apply here: <a href="http://careeropportunities.dhigroupinc.com/" rel="nofollow">http://careeropportunities.dhigroupinc.com/</a>. The position is a telecommute role. We will absolutely consider people with no data science experience, so long as they demonstrate an aptitude for data science \ machine learning and can code.
I considered a graduate program in data science, but compared to average programmer salaries, it doesn't seem like data science pays all that much (excluding data science jobs for PHD's in silicon valley). It's more interesting that programming, but seems like a <i>much</i> tighter market with no discernible demand driving salaries up.
I'm a designer but work for a data science company (LMI specifically). All of our data work is done in D, which I never even knew existed until I started working here.<p>I can't speak to anything regarding ML, but for whatever it's worth in our segment of the market we have seen a lot of competition emerge in a big way the last few years. Former academic-type firms who specialized in bespoke economy analysis reports are starting to build software around all of the data that is out there since it's never been easier to collect and normalize it. I think it's a stretch to say the market is approaching saturation for us, though.
As many have said here, and as a working and apparently in-demand data scientist, I agree that the tricky part about data science is that being effective isn't a matter of just any one thing. You have to be a unicorn of sorts who is, above all things, capable of solving any problem which comes your way. You have to be exceptionally flexible and very scrappy.<p>There are a lot of people who know more about modeling, software engineering, statistics, machine learning, analytics, and so on than I do. But I excel at bringing everything together and solving difficult business problems. It's really difficult to train someone to be this way. It takes a lot of time, experience, skills, and a unique disposition to be an effective data scientist. At least to be the kind of data scientist I am. And I'm still early in my career.<p>Just my two cents. I suspect there will continue to be a glut of people who, on paper, have the data science skills, but lack all the intangibles. Who knows, maybe the various programs and boot camps will start doing business scenario learning: here's a tough real world problem where we don't tell you how to solve it, but we desperately need you to figure it out. Go!
Background: I currently lead a Data Science team at a big non-tech company. Previous to this I worked at a software company that had a Data Science team in their customer facing consulting group.<p>I'm going to speak primarily about applied data science. This means a data scientist who is solving a business need by doing ad-hoc analysis or building a reusable solution (e.g a R+Shiny dashboard) to a business problems.<p>Jobs: There are plenty of jobs out there, but you have to be careful. Many "Data Science" jobs are really BI, Business Analyst, or Sales Engineer types of jobs where some VP got it in their head that they need a Data Scientist. These jobs are great for people who are okay with Technology and Data Science being 10% of their job - and many people are like that. They don't care about engineering, coding, or tech and statistics beyond the minimum to do their jobs. But if you really want a job that involves solid tech and stats/ML skills you will be unsatisfied at these types of jobs.<p>Right now there are plenty of hard business problems that people want to turn into Data Science problems because they think it'll give them a competitive edge or something to market and show off. This results in more data science job openings. However, they are not really data science problems. As somebody else said, people will eventually realize they are not getting the value they need with data scientists doing these types of jobs. Then they'll replace that person with an MBA with some DS coursework (e.g. MBA who can use KNIME or SAS Enterprise Miner) or eliminate the position.<p>People: I interview people and I know people at other organizations who interview candidates for Data Science roles. MOOCs and many degree programs (including 2 year MS degrees) are pushing out people who have a very superficial overview of data science. Basically they teach them about every ML algorithm in the known universe and the functions to call them them in R/Python/SAS. The end result is a mediocre coder or non-coder who boils everything down to a confusion matrix or root mean squared error. But they cannot actually think through a business problem or see why a low error doesn't equal a good model (see <a href="http://www.tylervigen.com/spurious-correlations" rel="nofollow">http://www.tylervigen.com/spurious-correlations</a>)<p>Finding good people is hard and you have to be flexible to realize great people can come from different backgrounds.
I can speak for Spain, although I sometimes get calls from other European countries. Relative to the pathetic Spanish work market data science/machine learning is doing great. I think right now there is too much hype, which is going to stay for a few years. After that I suppose it won't be a hot thing but I don't think it's going to disappear. I hope I'm mistaken and we are really seeing some AI revolution, but after all my job is putting the trust on the data, and past data says fads come and go. If that happens I will keep with me the math, the statistics, any development skills I can learn meanwhile and of course the challenge of someday achieving true AI.
Worth a serious effort if you are going to use it originally in your own niche / industry, otherwise statistics will still help you more in any given market. So just learn statistics very very well and then ask again.
I am currently an MIS graduate student with 3 years of SAP functional experience. After this boring stint and hearing the hype around Data Science, I decided to give it a try (Decent statistics and engineering skills but no coding expertise. I also finished MOOCs and am currently working on some small projects during the holidays). Considering average pay as a prominent factor, what is a better option - Learning extra SAP skills (HANA etc) and try for a job in SAP or diving into Data Science completely and try to start as an entry level Data Analyst.
Is there a market for competent developers without professional/academic experience in data science or machine learning? Perhaps just a MOOC or some Kaggle projects?
If you're seeking work: If you want to be in demand, be the machine learning person for __________ , electric energy revenue protection, or healthcare payer fraud detection, investing, or supply chain. Pick a specialty.<p>If you're hiring: Get the above out of your pathetic small minds and start hiring the smartest people you can find. Look for successes in any industry. Your business isn't that unique. The best people can learn it much faster than you did.
I'm in Australia.<p>I'm hiring 6 people in a range of roles between "pure" data scientists to more data engineer/SWE roles. The exact mix depends on who we can get.<p>The ability to find good people is the biggest constraint on the work we do.<p>Our current team ranges from applied mathematicians (as in they are Math professors) to people with traditional SWE backgrounds. Basically we are a long long way from saturated.
People with acquired skills are plenty and not really up to scratch most of the time. So people who have these "acquired" skills have a high likely hood of being scrapped at the CV stage.<p>If you're serious about machine learning - build a blog or online repository of quality work and use that to get a job instead
From someone who is looking to transition to data science, the field is terrible if you haven't had an industry job before. I am ranked in the top 150 on Kaggle and can't even get phone interviews without someone in my network recommending me for a position.
There is certainly a big market for both data analysts and system builders here in Moscow. Most want a person with credentials, e.g. Yandex school of data analisys.
Field is rather on fire with big companies investing a lot of money in it
Off Topic: What will you advice to someone who writes code in Python(scraping, mining) and have a done of ML by wathing Udacity courses, how can I polish myself to get into position for a job?
This thread is really depressing as someone currently going through a bootcamp. (dataquest) Is it more realistic to aspire for data engineering/analyst roles?
I am curious about this as well. I think the difference between machine learning and software engineering is that companies may only need a few dozen machine learning engineers. They may need thousands of software engineers. There may be increasing demand, but the demand will never reach the demand of software engineering. Except at the premium ultra competitive level, where a data scientist who is globally known can have a massive impact on the companies bottom line. But we arent talking about those types of jobs.<p>I also believe that most traditional companies do have data scientists, but they havent really start incorporating machine learning into their products, they are analyzing information about their customers, but their products are not reliant on using data. Once that becomes more common, things will pick up.