Well, that's not true. Apparently he does; in fact, George W. Bush calls Kanye West's criticism of him after Hurricane Katrina the "worst moment" of his presidency:
MATT LAUER: This from the book. "Five years later I can barely write those words without feeling disgust." You go on. "I faced a lot of criticism as President. I didn't like hearing people claim that I lied about Iraq's weapons of mass destruction or cut taxes to benefit the rich. But the suggestion that I was racist because of the response to Katrina represented an all time low."
PRESIDENT GEORGE W. BUSH: Yeah. I still feel that way as you read those words. I felt 'em when I heard 'em, felt 'em when I wrote 'em and I felt 'em when I'm listening to 'em.
MATT LAUER: You say you told Laura at the time it was the worst moment of your Presidency?
PRESIDENT GEORGE W. BUSH: Yes. My record was strong I felt when it came to race relations and giving people a chance. And -- it was a disgusting moment.
Ta-Nehisi Coates drolly notes that it's "typical of George Bush that implication that he's a racist is worse then the implication that he sent thousands of people to their deaths on lie."
I just want to stand up for Bush for a second and say that this isn't really a Bush thing. Many white Americans have settled on the idea that calling someone a racist is actually the worst thing you can ever say about someone. There's an important distinction to be made here, though -- as a result it's worse to call someone a racist than it is to actually act on one's prejudice. As a result, a lot of people are more focused on minimizing accusations of racism than dealing with actual racism. This is a rather extreme manifestation of this -- the left has accused Bush of being a war criminal for "legalizing" torture but he finds the implication that he's indifferent to the suffering of African Americans far more offensive.
That said, West's criticism of Bush was unfair. The response to Hurricane Katrina was a reflection of basic administrative incompetence and cronyism, not active racial animus. Bush made an active effort to court black voters, marginalized (mostly) Islamophobes, he appointed a diverse Cabinet, and, defying the nativists in his own party, he brokered an immigration compromise that policy-wise, was better than anything we're going to see for a very long time. The spate of noose-themed hate crimes during 2006-2007 seemed to make him genuinely angry.
Bush had a lot of flaws, but racism wasn't really one of them, at least, no more than the average human being. His big problem is that he was a member of a party for whom the most effective hammer against the American welfare state has always been race, even when a devil's bargain between liberals and white supremacists gave Democrats an enduring majority.