I think we need to be a bit more careful when setting immigrants, legal or illegal, in opposition to unemployed, young black men. While there's certainly some occupational overlap, and immigrants are slackening a labor market black males would like to see tightened, the correlation isn't 1:1. When you're talking ghetto poverty, you're talking urban poverty, and much of that is spatial in nature. Immigrants who head over without a waiting family or community have an enormous amount of geographical mobility, so they go to inland California to work in agriculture, or Southern Virginia to do day laboring, or wherever. Urban blacks, by contrast, are rooted in a particular space, a space that has largely been vacated by good jobs and employers (due to the shift towards suburbs, the decline of manufacturing, the effects of various ghetto pathologies, and so forth). The argument can be made that breaking up these ghettoes is critical -- that was one of the rationales of welfare reform, and I agree with it -- but it's tough to do, and efforts to accelerate it have largely failed. That is not the fault of illegal immigrants, they've merely picked up the existing slack.
The quick riposte to that is that if they weren't there, said slack would tighten, become better-paying, and attract urban blacks. Well, maybe. But it's also worth noting that immigrants have actually kept certain industries in the US. It's easy to say that we should just pay folks more money to pick lettuce, but then our lettuce becomes decidedly noncompetitive in the globalized marketplace. There's a weird circularity to all this, where we've used the combination of cheap immigrant labor, heavy subsidization, and more advanced technology to out-compete Mexican's agricultural sector, which we partially atone for by hiring Mexicans to work in ours at higher wages. If we sent the Mexicans home and raised picker salaries to $14 an hour, our agricultural industry would likely disintegrate.