Lesson objective

Describe what published research records about automated valuation model error across neighbourhoods and what a large company said about forecasting home prices, and explain those findings to a client without overstating them.

Plain-language explanation

Why this matters now

Automated value estimates appear beside listings on consumer sites, inside agent tools and in lender systems. Sellers quote them at listing appointments and buyers quote them at offer time. They come from models trained on past sales, and they are the most visible AI-style output in residential real estate. Two kinds of evidence tell you how carefully to treat them: research on where their errors fall, and a large company's own account of forecasting home prices.

There is also a gap to be honest about. This course holds no independent accuracy figure for any consumer portal's estimate. Anyone who tells a seller that "those estimates are off by X percent" should be able to name the study, the model, the market and the period. If they cannot, the number should not be repeated.

A question to sit with

Two homes are each misvalued by an automated model. One is off by $40,000, the other by $30,000. Which estimate was worse, and what would you need to know before answering?

What goes wrong

Treating an estimate as a value. An automated estimate is a model output. It is not a licensee's comparative market analysis or broker price opinion, and it is never an appraisal. Module 9 covers those boundaries.

Quoting an accuracy figure nobody can source. Repeating a remembered or vendor-supplied error rate turns a guess into a representation to a client.

Overstating the research. The studies below found a pattern in the size of percentage error. They did not find that the models use race as an input, and they did not measure any named consumer portal.

Ignoring who bears the error. An estimate that is wrong by a larger share of a lower-priced home can matter more to that owner, and can flow into comparisons for neighbouring homes.

Assuming more data or better algorithms solve it. The later research tested exactly that.

Blaming "the algorithm" for a business failure. A company's public explanation should be quoted in its own words, not reinterpreted.

The practical answer

1. What the Urban Institute found. In a March 5, 2021 commentary, Why Automated Home Valuation Technology Errors Disproportionately Affect Majority-Black Neighborhoods, the Urban Institute compared automated valuation model (AVM) estimates with actual sale prices in the Atlanta, Memphis and Washington, DC metropolitan areas. It reported that:

  • absolute dollar error was larger in majority-white neighbourhoods, but the percentage magnitude of error was larger in majority-Black neighbourhoods after adjusting for lower sale prices;
  • AVM values in majority-Black neighbourhoods "did not exhibit systematic bias (by average dollar difference) over time";
  • after controlling for home values, property differences, neighbourhood conditions and turnover, neighbourhood majority race remained a statistically meaningful factor in percentage error;
  • inaccuracy "could make neighborhood-wide disparities even worse when these homes are used as comparable sales in the transaction of another property."

A May 18, 2022 Urban Institute brief, Revisiting Automated Valuation Model Disparities in Majority-Black Neighborhoods, found that even with property condition data and AI techniques, the percentage magnitude of AVM error remained greater in majority-Black neighbourhoods. The 2021 piece is a commentary on a think-tank study that was not peer reviewed, and it measured lender-grade AVM data from one provider, not consumer portal estimates.

2. What the Cityscape article found. Linna Zhu and Michael Neal (Urban Institute) and Caitlin Young (Yale Law School) set out the fuller method in Racial Disparities in Automated Valuation Models: New Evidence Using Property Condition and Machine Learning, published in HUD's Cityscape journal (Volume 26, Number 1, 2024). They paired AVM estimates with arm's-length sale prices for properties that traded in 2018 in selected counties of the Atlanta and Memphis areas, using data from a major property records provider, combined with 2014-2018 American Community Survey tract racial composition and property condition ratings.

Finding What the article reports
Direction of error Over- versus under-valuation did not systematically differ by neighbourhood racial composition
Size of percentage error Roughly twice as large in majority-Black as in majority-White neighbourhoods; for 2009 in Atlanta, 64 percent versus 24 percent
After controlling for condition Location in a majority-Black neighbourhood still raised the percentage magnitude of error by 3.499 percentage points, about $4,549 on a $130,000 home
Property condition A poor condition rating raised percentage error by 4.35 percentage points and a fair rating by 2.72, relative to a good rating
Conclusion "despite data improvement and ML, evidence still shows that the percentage magnitude of AVM error is greater in majority-Black neighborhoods"

Two limits must travel with those findings. The study covers two metro areas and one AVM data provider, so it is not a measurement of any consumer portal estimate. And the higher percentage error is partly attributable to lower average home values in majority-Black neighbourhoods; the study does not show that AVMs use race as an input.

3. Why percentage error matters. A fictional illustration, not study data, shows the arithmetic. An estimate of $210,000 on a home that sells for $240,000 is $30,000 off, which is 12.5 percent of the sale price. An estimate of $620,000 on a home that sells for $660,000 is $40,000 off, about 6.1 percent. The larger dollar error is the smaller share of the home's price. For the owner whose equity, listing price or offer rests on the estimate, the share is what bites.

4. Forecasting home prices at scale. Zillow Group's Form 8-K and Exhibit 99.1, filed November 2, 2021, record that its board decided to wind down Zillow Offers, its iBuying service in which Zillow bought and sold homes directly. The filing attributes the decision to "home pricing unpredictability, capacity constraints and other operational challenges" made worse by an unprecedented housing market, a pandemic and labour and supply chain conditions. Chief executive Rich Barton said: "We've determined the unpredictability in forecasting home prices far exceeds what we anticipated and continuing to scale Zillow Offers would result in too much earnings and balance-sheet volatility." Zillow recorded a $304.4 million inventory write-down in the third quarter of 2021 "as a result of purchasing homes during the third quarter at higher prices than Zillow Group's current estimates of future selling prices after selling costs," expected $240 million to $265 million of additional fourth-quarter charges on homes under contract, and expected the wind-down to reduce its workforce by approximately 25 percent.

Quote the filing, not a story about it. It does not say an algorithm or AI "failed"; it names forecasting unpredictability together with capacity and operational constraints. It concerns a company buying homes as a principal, not agents' use of AI, and it says nothing about consumer-facing value estimates. What it does document, in a company's own words, is that forecasting home prices at scale proved less predictable than a well-resourced company expected.

5. How to talk about an estimate with a client.

  1. Say what it is: a number produced by a model from past data, not a licensee's opinion and not an appraisal.
  2. Say what you know about its accuracy, and only that; do not invent an error rate.
  3. Explain that research in several metro areas found error sizes can differ by neighbourhood, with the study's limits.
  4. Build your own analysis from verified comparable sales, and show your working.
  5. Record the estimate you discussed, its source and date, and what you told the client.

Remember this

  • An automated estimate is a model output, never an appraisal and never your value opinion.
  • The Urban Institute and Cityscape research found larger percentage error in majority-Black neighbourhoods in the metro areas studied, with one data provider, and no systematic direction of error.
  • The research does not show models use race as an input, and it does not measure any consumer portal.
  • Zillow's 2021 filing cites forecasting unpredictability with capacity and operational constraints; it does not blame an algorithm.
  • Do not quote an accuracy figure you cannot source.

Real estate example

Copperline Properties is a fictional brokerage. Agent Nadia Brookfield meets the owners of 27 Linnet Street, a fictional three-bedroom home, for a listing appointment. The sellers bring two printouts: a consumer website estimate of $214,000 and a relative's claim that "those sites are always 20 percent too low around here." Nadia does not repeat or dispute the 20 percent figure; she explains she has no source for it and none for the website's accuracy in their area. She says the estimate is a number produced by a model from past data, not her opinion and not an appraisal. She mentions that published research in Atlanta, Memphis and Washington, DC found that the percentage size of automated valuation error was larger in majority-Black neighbourhoods, without a consistent direction, and that those studies used one lender-grade data provider and did not test consumer sites. She then presents her own analysis of four verified closed sales from the fictional Tri-County MLS within the past six months: $226,000, $238,500, $231,000 and $219,500, which average $228,750. She adjusts for the sellers' older roof and suggests a list range of $225,000 to $232,000, showing her working. If a buyer's side later relied on the $214,000 estimate, it would be $14,750 below her comparable average, about 6.4 percent of $228,750. The sellers ask whether the estimate can be hidden next to their listing on Copperline's website. Nadia checks the brokerage's MLS rules, which follow the model rule on IDX displays, and arranges for the feature to be disabled for their listing. She records the estimate, its source and date, what she told the sellers and her analysis in the listing file.

Responsible-use warning

Do not describe an automated value estimate as a value, an appraisal or your opinion, and do not repeat any accuracy figure you cannot trace to a named study, model, market and period. When you explain the research on neighbourhood disparities, keep its limits: metro areas studied, one data provider, a pattern in percentage error, no finding that race is a model input, and no measurement of consumer portal estimates. Overstating a study is still a misrepresentation.

Human-review requirement

The licensee owns every statement to a client about an automated estimate and every comparative market analysis, and builds pricing advice from verified comparable sales rather than from an estimate. The sponsoring or designated broker, under the supervision duty their state defines, checks that estimates are never presented as appraisals or licensee opinions, that research is described with its limits, and that seller requests about estimate displays are handled under the local MLS rules. An appraisal belongs only to a licensed or certified appraiser, and a lender owns its own valuation decisions.

Practical exercise

Prepare a client explanation of an automated estimate using fictional figures.

  1. Invent a property, an automated estimate for it and four fictional comparable sales.
  2. Compute the average of your comparable sales, the dollar difference from the estimate, and that difference as a percentage of your average. Show your arithmetic.
  3. Invent a second property at about three times the price, with a larger dollar error but a smaller percentage error. Show both calculations.
  4. Write three sentences a client could understand explaining what an automated estimate is and is not.
  5. Write two sentences summarising the Urban Institute and Cityscape findings, including the metro areas, the data provider limit and what the studies did not show.
  6. Write one sentence on Zillow's 2021 wind-down that quotes the filing's own stated reasons and claims nothing more.
  7. Find your local MLS's rule on automated estimates shown in IDX displays and note how it compares with model Section 18.2.8.
  8. Write the file note you would keep after the conversation, including the estimate's source and date and the words you used to describe it.
  9. Read your client explanation once more and remove any sentence that calls the estimate a value, an appraisal or your opinion, or that states an accuracy figure you cannot source.

Knowledge check

  1. What did the Cityscape article by Zhu, Neal and Young find about the direction and size of AVM error?

    Answer guidance: Over- versus under-valuation did not systematically differ by neighbourhood racial composition, but the percentage magnitude of error was roughly twice as large in majority-Black as in majority-White neighbourhoods, and remained higher after controlling for property condition.

  2. Name two limits that must accompany any summary of the AVM disparity research.

    Answer guidance: Any two of: it covers a small number of metro areas and one AVM data provider; it is not a measurement of any consumer portal estimate; the higher percentage error is partly attributable to lower average home values; and it does not show that AVMs use race as an input.

  3. What reasons did Zillow Group's November 2, 2021 Form 8-K give for winding down Zillow Offers?

    Answer guidance: Home pricing unpredictability, capacity constraints and other operational challenges, made worse by an unprecedented housing market, a pandemic and labour and supply chain conditions. It does not say an algorithm or AI failed.

  4. Under NAR's model MLS rule 18.2.8, what can a seller ask for?

    Answer guidance: That a participant's IDX display showing an automated estimate of the listing's market value in immediate conjunction with the listing disable that feature for the seller's listings. The local MLS's rule governs.

Sources and further reading