As I native, I would assume that "мне по хрену"/"мне похрен" is an euphemism of the more rough variant "мне похуй" (it is up to my dick, same height as my dick).