I think the fact Europeans enslaved people have very much to do with the OP's claim that replacing direct enslavement with debt peonage at a later date more than made up for not only the original enslavement but later colonialism.
Blaming an oppressed group for their own oppression is not a good look.