Zillow lost money because they weren't willing to lose money

&gt; They thought they needed to build a machine learning model when they really needed to build an entirely new organization, one that possessed the technical and cultural mindset necessary to succeed in this space.<p>I totally agree. It&#x27;s not impossible to imagine their model working: why couldn&#x27;t you serve as a market-maker for homes at a large scale, especially with the unique insights Zillow could have based on their datasets.<p>However I think where the hubris lay is in how they thought they could leapfrog all the way to an automated solution before building a competency as a house-flipping company.<p>From what I understand, where they failed was partly in building a rich enough model to properly account for the less easily quantifiable elements which ultimately account for a property&#x27;s value. I.e. the price per square foot might make a property look like a steal, while something like a sewer main nearby, or problematic neighbor could radically change the value proposition to anyone standing at the site. That&#x27;s a non-trivial problem to solve for even the best ML and it&#x27;s not clear how you would automate this.<p>If you ask me, instead of focusing on building an automated price discovery system, they should have started by trying to build a quality home-flipping organization, and figuring out how to super-charge manual work using their datasets. Over time you might find ways to optimize the process and increase the level of automation to scale output relative to head-count.

Good riddance. If large-scale house flipping took off, we might actually end up in a scenario where housing was treated as a speculative asset, with empty houses getting flipped between investors looking to make a quick buck, further lowering the supply of actual places to live (because housing units remain empty while being flipped), driving up the cost for families who just want a place to live. Oh wait...

&quot;data is fungible?&quot; i think the author does not understand what fungibility means. if something is fungible, that means that any unit of it is exactly the same as any other unit of it. fungible data would be random data, which wouldn&#x27;t help you predict or model anything.<p>i think zillow also did not understand what fungibility means. real estate is not fungible. in fact, it&#x27;s anti-fungible- that&#x27;s why there are huge diligence processes that exist around most real estate transactions. maybe floors in an office building may be fungible, but residences are definitely not- with all their quirks, customizations and problems.<p>this whole argument that they failed because the ceo of zillow didn&#x27;t have big balls is pretty putrid. pair this with the word salad of misused words and twisting of quotes and i&#x27;d say this is probably one of the worst pieces i&#x27;ve ever seen posted here. i feel worse for having given it any time at all.<p>the simple fact is that the ceo of zillow didn&#x27;t know what they were doing, had a team that (supposedly) applied facebook&#x27;s infrastructure scaling prediction library to house prices and then attempted to apply a market making mindset to the real estate market at scale. not only is this probably something nobody should try to do, considering we contribute so much in tax dollars to first time homebuying incentives as it&#x27;s recognized that the housing market is where j q public can start to build wealth, it&#x27;s also probably something that nobody could do (well, at scale, with machines) given that housing is not fungible.<p>sure, financial instruments are fungible, maybe even late model cars, but definitely not houses. doesn&#x27;t take a scientist to spot that.

While no doubt Zillow made many of these mistakes, I think the reality is more sobering that the author of the article realizes. The more grim possibility, is that Zillow got out of the house buying business, not because they weren&#x27;t good enough at it, but because they _were_ good enough at it to realize that it was at the top.<p>If buyers want more now for their house, than it can be sold for in a few months time (which is necessary for renovations and other prep for sale), then there is no ML (and no non-ML) method to make money. Either you overpay and lose money, or you don&#x27;t overpay and you don&#x27;t buy any houses.<p>In that situation, the only smart play, is to get out of the market. Zillow is, no doubt, not perfect. But they have a lot of knowledge of the housing market, and they thought it was time to get out entirely. I think the author of the article either isn&#x27;t able, or doesn&#x27;t want, to consider that Zillow might have been exactly correct in doing so.

I really liked this quote, which is also true of machine learning organizations at large tech companies:<p><pre><code> The most valuable data is not social data, ... but your own data because every dataset that you’re looking at internally describes your own process, including your bugs, ... building models from your own data is the only way to build a really successful system. </code></pre> This is one thing that a lot of outsiders do not understand. Facebook&#x2F;Google&#x27;s data is basically worthless to anybody but Facebook&#x2F;Google. The data has value because it is derived from their own processes, which in this case are the requests and context of each product surface.

While the point the article makes is true — it costs money to acquire the real world data, the comparison to credit underwriting is misguided. Underwriting credit is fundamentally different than predicting house prices.<p>In particular when you’re auto-underwriting credit it’s not typically an origination-for-sale model. So the value of the loan is the present value of the future payments, less the future value of defaults, less the cost of acquiring the customer.<p>Historically those things can be modeled pretty accurately and the aspects that can’t be modeled accurately can often be hedged or eliminated by the law of large numbers. The innovation of the new ML underwriting with respect to accuracy is at the margins. The real disruption is the speed and cost. (Disclosure: I worked at a SMB fin tech and we reran multiple credit models for a million customers and past customers every night.)<p>If Zillow were getting into the rental business, in some ways it might have been easier for them. But they needed to model where they could sell an illiquid asset which is a much harder and much less well understood problem. And yes with enough capital to plow through and the appropriate risk attitude they could likely have gotten the handle on what their pipeline was really going to look like. But it’s hardly the same problem as credit underwriting.

I think the title is highly misleading. The main point here is that Zillow simply had no idea what it takes to be a market maker and their pool was picked off by savvy traders.<p>Good tweetstorms with technical explanations on how that happened:<p><a href="https:&#x2F;&#x2F;twitter.com&#x2F;macrocephalopod&#x2F;status&#x2F;1455887352371597312" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;macrocephalopod&#x2F;status&#x2F;14558873523715973...</a><p><a href="https:&#x2F;&#x2F;twitter.com&#x2F;0xdoug&#x2F;status&#x2F;1456032851477028870?s=21" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;0xdoug&#x2F;status&#x2F;1456032851477028870?s=21</a>

The article offers no new or inside information, just more armchair quarterbacking. I&#x27;m surprised that it is getting traction on HN. I think it says more about the zeitgeist than it does about the (lack of) insightful-ness of the content.

I think the article makes an interesting point about this being the first of many, but I disagree with the initial tone of the article. It seemed to paint Zillow as being afraid of loss. On the contrary, I viewed Zillow as demonstrating good common sense and an ability to make hard decisions. To me it shows that they aren&#x27;t committing the sunken cost fallacy, and are willing to cut an entire 25% of the company and take massive losses so they can redirect themselves towards better objectives.

I think a key point that is missed is the feedback cycle time. Real time bidding advertising has I believe a number of the listed concerns however the feedback time is maybe hours at most and might be milliseconds. So the risk is in general a lot smaller and worst case you just lose some of the money you spent that day&#x2F;week. With long term assets you could lose months worth of investments before your feedback loop fully kicks in.

In the original Foundation books by Asimov, the conceit of &quot;Psychohistory&quot; was similar to the concept of machine learning for pricing: The future can be predicted _if people aren&#x27;t aware of the prediction to change their behavior in relation to it_<p>This is similar to &#x27;adverse selection&#x27; in real life &amp; in Zillow&#x27;s model. The article makes a nod to this, but seems to imply that if you train your model on that adverse selection, you can come out ahead after paying to learn about it.<p>To me that kind of misses the point. Adverse Selection isn&#x27;t a static feature of the landscape you can identify and avoid, it is people understanding what you understand, adapting, and responding. Train your model with adversaries trying to beat it, then you&#x27;ll maybe counter the specific first round strategies they use, and they&#x27;ll learn new ones and beat your new model with their 2nd round strategies. It&#x27;s a continuous game. Your requirement to gather a corpus of training data will keep you in the 2nd turn of a game where the wins are biased to whoever has the 1st move.

Real estate is one of the few markets where non-experts can make money, where it’s not a hyper-liquid winner-take-all game. Coupled with this is the fact that housing is a necessity and owning a home leads people to invest in their communities more than if they were renting, I think it’s a good thing if Zillow (and OpenDoor, etc.) fail at pushing everyday people out of the business of real estate investing. Here’s hoping we see some regulation—the illiquidity of the home buying market is not a problem that needs to be solved.

Framing it as machine learning undersells the problem.<p>It&#x27;s a hybrid model trading in an adversarial, real-dollar environment. The leverage comes from having a small human team trade big volume, much more than they could possibly trade directly, by augmenting their human abilities with automation and a model. Or seen from the other side, it&#x27;s a model with human oversight.<p>Any system like that is high risk, high reward. All the successful ones started out by losing a lot of money. Paypal lost an incredible amount to fraud before they started breaking even. OpenDoor lost an incredible amount to mispricing, and took on a ton of balance sheet risk, before their business really started working.<p>&quot;To live, you must be willing to die&quot;<p>- poker legend Amir Vahedi

Let&#x27;s not forget there was also a huge public outcry on HN and in other places when it came out that they were buying and selling real estate. So if it was socially detrimental to the company&#x27;s image, and they didn&#x27;t have the corporate will to collect enough data for it to become profitable, it&#x27;s a no-brainer that they would get rid of it.<p>I see this as a victory for us calling out companies for immoral behavior.

Zillow lost money, because they were hit really hard during pandemic.<p>This article does not mention that. Instead, the rest of the article deals with Linkedin-wisdom and hard platitudes, such that it is not possible to build a good model on someone else&#x27;s data (as if Zillow even was).<p>Data scientists remarking on the Zillow fold, are like psychiatrists or engineers remarking on non-clients and bridges build by others. They know nothing about the business, about the constraints, about how the estimates are consumed. They end up silly, but without good information coming from Zillow, we assign value to their analysis, purely on Twitter-soundbite-ability and internet-authority.

“You cannot bootstrap off an existing dataset. Full stop. These datasets can contain implicit assumptions or associations that you are not aware of. This is the original sin of many a algorithmic risk underwriting startup”<p>False. You can definitely bootstrap and adjust the model as you either gather more data yourself or get more outside data. You can also build confidence intervals around the model predictions and decide how you want to proceed based on that. There is lots you can do with that initial model.

The essence of the article is that they underestimated how flawed their algorithms are and how hard it is to build a good lasting algorithm in a dynamic world.<p>Many seasoned wall street algorithms have suffered many times over 5 decades, and when they fail we call them black swan events.

I wonder why so few here question the basic assumption of injecting from above, machine-learning models to extract profit, into a vital part of the reproductive cycle of human families.

&gt; At a high level, the story of Zillow Offers is a story of our industry at its best.<p>Not in my book. All I see is the price of real estate being driven up by corporate greed and the individual home-buyer being shut out of the market.<p>Is it wrong of me to hate &quot;flippers&quot; (be they corporate or private)? Pure capitalists will tell me that every property sold went to the highest bidder — in the case of a flipper winning they were willing (able) to risk the capital to hopefully turn a profit on the flip.<p>I suspect if you dig deeper you might find sales going to flippers because they had 100% cash offers, because they are better at &quot;the game&quot;. I see no reason to punish prospective first-time home owners in this sort of market.<p>But I don&#x27;t know what the answer is either.

This is a good take, but<p>&gt; A machine learning organization thinks of risk entirely differently than an automated risk underwriting organization.<p>It&#x27;s possible and maybe even advisable to use machine learning in the automated risk underwriting business, but it <i>is</i> a different setup &#x2F; set of objectives.<p>As the author notes, IMO the adversarial and antifraud aspect of risk underwriting turns it less into a straight-up estimation problem and much more into a game theory type of problem. ML models can assist in evaluating risk, but you do indeed have to be preocuppied by your risk as a party to the transaction in the first place, and not just trying to predict prices as a third party observer (which by itself is pretty riskless).

&gt; Because when you have a hammer, everything tends to look like a nail and when you have TensorFlow, everything tends to look like an ML problem.<p>And if you have billions of dollars in cheap capital, everything looks like an investment problem.<p>Which is ultimately the suggestion of this article: &quot;Why aren&#x27;t you more like Wall Street?&quot;<p>The implications are exactly the opposite of Zillow being an innovative company. If they require billions of dollars in deep pockets (nbd) and a restructuring of their org to be more like old-school operators, all signs point to existing players as more fundamentally correct about the strategy required to succeed in the space.

If the assumption is that you&#x27;re going to lose half your money up front, then my plan would be to make sure &quot;my money&quot; is as little as possible: learn based on smaller bets. It sounds like Zillow built the Sea Dragon first, when they should have started with the Redstone and moved toward the Saturn V.<p>If Zillow thought they had all the data they needed, there would have been little harm starting with $100 million in properties -- if the loss there ended up being $5 million, they would have known immediately something was up and that they had work to do.

The amount of Monday morning quarterbacking of Zillow is just staggering.

All of this reads like a Dickensian nightmare, where corporations have bought up all the water and air.<p>This is ridiculous, we need much better regulation on this stuff.<p>I wonder if higher property taxes would help a bit? If you own a &#x27;home&#x27; then you&#x27;re going to be paying for the water, school, electricity infrastructure whether you use electricity, water, or not.<p>Of course, that would be gamed hard and would have to be strongly regulated as well.<p>But that, and vacant property taxes, limits on some other things, and some other adjustments might help.

Feels analogous to the history of the collateralized debt obligation debacle where the models used to value CDOs were trained on data that no longer resembled reality. At least Zillow can live to fight another day, where as Stan O&#x27;Neal put all of Merrill Lynch&#x27;s chips in with one of the biggest make-or-break gambles in the history of finance and the market turned against it, rendering Merrill to a fatally wounded company bailed out by Bank of America.

I think some reasons Zillow lost were that their pricing and risk processes were terribly underdeveloped in order to scale fast, their models were obviously inaccurate, and they didn&#x27;t understand the difference between an acquisition cohort and resale cohort, and specifically how much the tail sales of an acquisition cohort determines profitability.

This is the same folly as Long Term Capital Management.<p>You&#x27;re not going to be able to reliably model asset prices at the resolution and accuracy needed to front run the market for a long period of time. This case was worse because the &quot;Zestimate&quot; directly created a feedback loop that moved the underlying asset prices higher.

Zillow&#x27;s mistake is that they thought their AI could replace human buyers instead of augment them.<p>Most AIs today are for augmentation, not replacement. Vehicle autopilots are a perfect example. The ones that are commercially available aren&#x27;t capable of replacing the human, they just augment the human&#x27;s abilities.

Realtor and Mortgage Lending industry is very slow in tech adoption and adaptation of digital strategy. Unless there is a lift across the industry on the buy-sell-marketplace together, such mishaps will occur. This industry will fail if injected with viral nature of social media algorithms.

If Zillow had &quot;figured it all out&quot; on solving the magic pricing problem, they could have put the entire appraisal industry out of business. Well, guess what, they didn&#x27;t and they didn&#x27;t even come close.

SMEs are smarter than developers in their space.<p>Always has been that way. Always will be that way. AI is great for when you need to tame a firehose and make millisecond decisions. But there&#x27;s a 90 year old in Omaha who is better than the best AI.

Is it fair to call this the result of “AI thinking?” Meaning that urge to automate away human involvement, because —after all—-if people are involved in analyzing data and decision making, then clearly the AI isn’t finished let.

&gt; One of the things that happens for a brand-new launched credit card: done right, you lose about 50% of the dollar volume in the first several months<p>What does this mean? 50% of the money is held as debt? Or 50% of the money is lost to fraud?

Does anyone know where Zillow get its dataset from? I reckon it&#x27;s essentially sale price? Can a &quot;hobbyist&quot; investor do the same?

If you have a good business with high margins, why not grow that business instead of starting a new low margins business of flipping houses?

Buying assets using models, I’ve seen that in stocks, but people usually don’t go all in with how hard it is to predict the economy

Well, maybe they just exited because we&#x27;re going into a recession and it&#x27;s a good time to get out of house-flipping.

Reminds me of Boeing. You could replace Zillow with Boeing here and it would apply perfectly.

Good luck getting a public company to incentivize the people capable of pulling this off…

They lost money because they were gaming their own system for their own profit.

&gt;&gt;&gt; you should expect to lose 50% of your capital allocated towards underwriting.<p>How ?

I love this guy’s movies. Finding out he writes so articulately to boot? Wow

What does &#x27;bootstrap&#x27; mean in the context of this article?

There&#x27;s something really funny about white collar office worker businessmen talking about how it takes balls of steel to do what they do. Ok bro, sure. Trackballs of steel maybe

This is really well written. Thanks for sharing.

<a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=ajGX7odA87k&amp;t=833s" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=ajGX7odA87k&amp;t=833s</a>

Your data is not neutral, it is opinionated. Who is asking the question? What do they use the data for? What questions are they not asking?

I can&#x27;t agree with the article or many of the comments on it.<p>(A) Both Wall Street and Machine Learning Modelers struggle with tail risk. Hedge funds measure performance against<p><a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Sharpe_ratio" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Sharpe_ratio</a><p>which assumes risk is (i) normally distributed and (ii) a source of reward. For most people, however, risk looks like Theranos or the Fukushima accident or the Challenger distaster.<p>It&#x27;s unbelievable that a machine learning model trained to predict house prices based on experience would be accurate in the face of events like the COVID-19 pandemic or what will happen when the Fed raises interest rates. You can model risks like that, but to the extent that you&#x27;re working from experience you are working from a database from the 1929 Crash, South Sea Bubble, etc.<p>(B) Mark Levine wrote a good article about how you&#x27;d exploit such a predictive model. If you consistently gave people low offers, a few people would accept them. You would get a high rate of return but could invest little capital.<p>To invest more capital you have to make more offers that get accepted, that is, give better prices. Your rate of return goes down and if there is shrinkage from errors, accidents, etc. you could get a negative return.<p>It&#x27;s that &quot;tendency towards a declining rate of profit&quot; that Marx warned about.<p>(C) The analogy with stock market market makers doesn&#x27;t sound good when you consider the differing timescales.<p>Market makers are isolated from some risk because of the length of their holdings. Yet, they make profits by exploiting the stochastics of a stationary market (e.g. if you don&#x27;t like the price at time t1, you will usually get a better price at t2) but they lose money when markets move definitively in one direction or another.<p>That kind of trader heads for the bathroom when things go South and in the interest of being orderly markets impose sanctions on market makers who do the natural thing and press the &quot;STOP &amp; UNWIND ALL POSITIONS&quot; button when it gets tough.<p>In the case of Zillow I see holding times that go on for weeks or months and all kinds of real world risk like planning to do certain renovations but having to delay the work because out of 20 things you need from Home Depot they only have 16 of them.

They build an AI that perfectly emulated Wall St masters of the universe.

Buy low, sell high. You need “data science” to do this? $VNQ is up 33% since 2016. Do you realize how dumb and bad you have to be to lose money on real estate in this time? I imagine randomly picking homes off the MLS would have yielded better returns in the last five years than whatever Tableau-powered nonsense the biz ops analysts at Zillow used. The entire iBuying concept is a farce, completely divorced from basic fundamental analysis.