Now, the cereal company is taking a hit.
Kellogg's announced earlier this month that it would not renew its endorsement contract with the gold medal swimmer, and its reputation has apparently suffered as a result.
A report from Vanno, which monitors the reputations of 5,600 companies, showed that Kellogg's ranked No. 9 in January and fell to No. 16 when it announced peanut butter recalls.
But when Kellogg's gave Phelps the boot on Feb. 5, their ranking plummeted to No. 68.
Kellogg's, now ranked 83rd, also experienced sinking stocks in the wake of its decision to drop Phelps.
While the economic factors most likely played a role in the company's ranking, many criticized Kellogg's in the wake of its decision and vowed to boycott the cereal maker.
Advocates for the legalization of marijuana urged a boycott of Kellogg's because of the company's decision to boot Phelps and called the action "hypocritical and disgusting."
The company announced in early February that Phelps' conduct was "not consistent with the image of Kellogg."
Phelps, who also endorses Subway, was able to keep his deal with the sandwich chain.