Being an ally is not the same thing as being a friend. This may seem obvious, or it may seem baffling, and hopefully this can be relevant reading either way.
You can be someone’s friend while also marginalizing them, either through microaggressions (such as being a cis person telling a trans person that they don’t look trans) or outright aggressions (such as being a white person calling a person of color a racist slur). They may or may not perceive what you’ve done as marginalizing. They may have gotten really good at finding internal ways to manage that marginalization (such as apologizing for your behavior for you, or agreeing with your behavior and its implications about them) in the interest of maintaining the friendship, or maintaining jobs, family, housing, whatever. They’ve likely lost any or all of those things before when advocating for themself about that thing. Yes, your friendship with someone can matter enough that they would rather put up with being repeatedly hurt by than risk losing yet another damn thing in their life.