The Spaniard surrendered a 5-3 lead in the first set and then spurned two set points in the tiebreak, which Mayer, ranked No. 63 in the world, edged thanks to an impressive crosscourt forehand winner.
Nadal, who has won the Barcelona Open a record 11 times, had not lost a set in the tournament since a round of 16 game against Fabio Fognini in 2015, winning 30 sets in a row until he met Mayer.
Nadal was in danger of losing two consecutive matches on clay for the first time in his career following defeat in the Monte Carlo Masters semifinal to Fognini, but he bounced back immediately by breaking his opponent in the second-set opener.
Nadal, who was playing on a court named after him, broke the Argentine twice more in the decider to set up an all-Spanish third round tie with David Ferrer.