The worst of inflation might be in the rear-view mirror, but the share of Americans who say they're "doing OK financially" has hit a four-year low.

Among all U.S. adults, 72% say they were "doing OK" in 2023 — the lowest percentage since April 2020, according to an annual Federal Reserve survey released Tuesday. The sentiment has been trending down since 2021, when it was 78%.

Notably, the share of parents with kids who say they are doing OK dropped from 69% in 2022 to 64% in 2023.

A lot of that has to do with inflation, as 35% of the 11,400 survey respondents say rising prices were the "main financial challenge" in 2023, the highest of all self-reported answers, including retirement savings and debt.

Inflation is taking its toll, despite wage growth

Wage growth has been outpacing inflation since early 2023, but the survey suggests that the cumulative effect of high inflation over the past few years has been a strain on household finances.

Inflation subsided after reaching a year-over-year peak of 9.1% in June 2022, but it's been hovering around 3% for nearly a year, according to the consumer price index, which tracks the price of commonly bought goods and services. Over the past three years, CPI inflation has increased by 18%.

Consumer spending has been resilient despite rising inflation, but the cracks are starting to show. In March, spending grew by 0.8% while income only grew by 0.5%, which suggests that Americans are spending beyond their means. This follows a monthly trend of overspending since late 2023.

These findings are reflected in the Fed's survey: While 34% of respondents say their income increased in the previous year, 38% of all respondents say they are spending more, too.

People are also saving less. The total share of income that consumers save has been declining over the past few months. The rate is 3.2% as of March, which is about half of what it was before the pandemic.

As savings have dwindled, personal debt has continued to increase, with credit card and auto loan delinquencies rising past pre-pandemic levels. These delinquencies have revealed "worsening financial distress among some households," writes a New York Fed official in a blog post about the trend.

In a recent speech, Federal Reserve chair Jerome Powell expressed cautious optimism that inflation will continue to decelerate down to the Fed's target of 2%.

"We've made a lot of progress," he said. "And we have a ways to go. We've got work left to do, but we're not looking at the very high inflation rates that we were seeing two years ago."

Want to be a successful, confident communicator? Take CNBC's new online course Become an Effective Communicator: Master Public Speaking. We'll teach you how to speak clearly and confidently, calm your nerves, what to say and not say, and body language techniques to make a great first impression. Sign up today and use code EARLYBIRD for an introductory discount of 30% off through July 10, 2024.

Plus, sign up for CNBC Make It's newsletter to get tips and tricks for success at work, with money and in life.