It’s the origins of it.

When it started being used, it was used derogatorily, so the connotation stuck despite it not being inherently racist to shorten a word.

Tbh, it doesn’t have to be a slur still. But it is still used by racists as a slur because of the origins, so it would take a cohesive effort to reclaim and reframe it.