In the sometimes heated discussion regarding the greatest men's tennis player of all time, it was the only serious asterisk.
And now, three months into this young season, Federer has taken some positive steps toward erasing that qualifier. Three big, swift strides, to be precise.
For the third time in three months, Federer defeated Nadal in a meaningful match. This time it was the Miami Open, and Sunday's score was a forceful 6-3, 6-4.
That followed his straight-sets win in Indian Wells and his five-set victory in the Australian Open back in January.
Suddenly, 23-14 doesn't seem so one-sided. And the GOAT argument doesn't seem quite as complicated. Especially the way things have gone this season.
"The dream continues," Federer said during the trophy presentation. "What a start to the year. Can't believe it."
And, believe it or not, he really can't.
After surprising himself in Melbourne and Indian Wells, Federer figured the semifinals in Key Biscayne might be a stretch.
Who says you can't teach an old dog new tricks?
We would remind you that Federer is 35 years old -- so far beyond the sweet spot of an elite player's prime that it is, frankly, difficult to process. The last time he won this tournament was 11 years ago, and he beat his coach, Ivan Ljubicic, in the final.
Federer, reinventing himself to a thrilling degree, has now:
• Won 19 of 20 matches this year and is far and away the leader in the ATP World Tour's Race to London.
• Already locked down a major for the first time since 2012 and secured two Masters titles for the first time since 2014.
• Beaten Nadal four straight times for the first time ever, going back to 2015 in Basel, and won five consecutive sets for the first time.
"Congrats to Roger," Nadal said. "Amazing start for the season, one of the best comebacks ever. Well done. Very happy for you."
If just a tad sad for himself. As Nadal explained to the crowd, this was his fifth time holding the smaller loser's trophy in the Miami final.
Federer now has a 10-9 edge in hard-court matches over Nadal, and this one underlined how he has been able to turn around this once one-sided matchup.
Nadal would typically overpower Federer's one-handed backhand with his lefty forehand and wear him down with an endless reservoir of energy.
Federer has a little more pop after switching to a larger racket, but it was his commitment to be more aggressive that has made the bigger difference. Federer made the conscious decision to crowd the baseline against Nadal and hit a significant number of shots from inside it, virtual half-volleys.
He finished with 17 forehand winners, 11 more than Nadal. Federer served flawlessly again, fending off all four break points. He converted two of the nine offered by Nadal.
And don't forget: The best match of the tournament -- and perhaps this 2017 season -- was Federer's three-hour, 11-minute semifinal win over Nick Kyrgios in three tiebreakers.
There's a lot to savor for fans of Federer and, with the clay-court season nearly upon us, you might have to hang onto those memories until Wimbledon comes around in July.
"Congrats to Rafa on a great comeback yourself," Federer told him on the court. "I'm happy we're both here together. Over the years, mostly we had some great battles over the years.
"The clay courts around the corner. I'm sure you're going to tear me to pieces over there."
Federer will play a more limited clay schedule, possibly Rome or Madrid and the French Open.
Even if Nadal manages to come away with his 10th French Open title in Paris, Federer should be the favorite at the All England Club. Imagine if Roger and Rafa came away with the first three majors of the season, to run their Grand Slam singles totals to 19 and 15, respectively.
That would make them, certifiably, the greatest and the second-greatest of all time. No argument, no asterisk.
And for now, there is no GOAT argument.