Wealthy Americans are pretty optimistic about their long-term investment returns, expecting to earn average annual returns of 17.5% above inflation from their portfolios.

That's according to a new survey from Natixis that surveyed households that have over $100,000 in investable assets in March and April of 2021. Those same investors report they expect to earn 17.3% above inflation in 2021, which, while high, may be understandable. The S&P 500 price index returned 15.76% last year and the market was up 5.24% already when the survey was fielded.

But longer term, earning an annual return that averages out to 17.5% above inflation year after year is an "exceptionally high" expectation, says Dave Goodsell, executive director of the Natixis Center for Investor Insight. These estimates are much higher than historical averages. For instance, U.S. stocks average 10-year returns of 9.2%, according to Goldman Sachs data.

Financial advisors' estimates are also much lower. Advisors surveyed by Natixis are calling for more realistic expectations of 6.7% average annual returns above inflation. In fact, the gap between investors' expectations and advisors predictions has been widening dramatically over the years.

Goodsell attributes the high expectations to recency bias, saying that investors are looking at their rate of returns for 2020 (which were pretty impressive) and thinking that they'll do better in the future based on their most recent yearly gains.

It's like saying to yourself, 'The market was good last year and the factors haven't changed greatly, so I should do the same or better this year.' But Goodsell says that extrapolating those returns into "your long term expectations of 17% above inflation is pretty, pretty optimistic."

Investors also need to be "emotionally equipped" to deal with higher levels of risk that can come with gaining outsized returns like this, Goodsell says.

"When we ask people abstractly, 'How do you feel about things,' they're pretty comfortable saying, 'I'll take risks to get ahead,'" Goodsell says. But, "when you start to think about what risk means, what it entails, people become a little more conservative."

Generally, investors are not big fans of volatile markets and want their investments to be safe. About 77% of investors surveyed by Natixis, including 79% of millennial investors, say they would choose safety over investment performance.

Investors need to realize that if they only earn 10% returns instead of the 17% they were hoping for, that's still a really good return rate, Goodsell says. When it comes to investing, it's about keeping everything in perspective, he adds.

Additionally, Goodsell recommends that investors not get too caught up in the hourly and daily market moves. "One of the pitfalls to having instant access to this stuff is, do you have enough time to think and interpret it? Sometimes it's probably good to step back and think about what to do," he says.

Sign up now: Get smarter about your money and career with our weekly newsletter

Don't miss: 58% of men were able to continue saving for retirement during the pandemic—but only 41% of women were