[swores]

Maybe it's an accent thing (mine is typical British) but the 'm' in 'rhythm' doesn't act like a vowel in my mind, not the way 'y' does in 'sky'.

The only kind-of vowel sound in the second syllable of rhythm is what you get from saying "th", but actually it just sounds like 3 consonants together without a vowel.

People say that 'y' is sometimes a vowel because it sometimes sounds like one, not because it sometimes fits in a syllable.

[thaumasiotes]

Y /j/ always sounds like a vowel. As I mentioned, phonetically it is one. It's just that a shortened version can act as a consonant for phonological purposes. Y as represented in yell and pretty are the same sound; it's just shorter in yell.