LA Clippers owner Donald Sterling has sent a letter to the National Basketball Association telling the league he won't pay his $2.5 million fine and rejecting his lifetime ban, a source close to the situation said Friday.

The letter also threatens the NBA with a lawsuit if Sterling's punishment for making racist remarks, handed down by NBA Commissioner Adam Silver in April, is not rescinded, the source said.

"We reject your demand for payment," the letter says, according to

In an exclusive interview this week, Sterling repeatedly denied he is a racist despite remarks made in a private conversation with his friend V. Stiviano that were released online.

In the recording, which drew widespread condemnation from fans, players and the league after it appeared on TMZ, Sterling chastises Stiviano for posting pictures online of her posing with African-Americans, including NBA legend Earvin "Magic" Johnson. He tells her not to bring Johnson to Clippers games.

Sterling said he was baited into saying those remarks.

Silver and the league moved quickly, announcing the punishment three days after the recordings were released.

"I'm apologizing, and I'm asking for forgiveness," Sterling told Cooper. "Am I entitled to one mistake? After 35 years. I mean, I love my league, I love my partners. ... It's a terrible mistake, and I'll never do it again."

The league's 29 other owners have taken the initial steps to force Sterling to sell the franchise he has owned for 33 years, with an advisory committee meeting three times so far.