But I wonder if there is some kind of minimum amount of mining required to verify X number of transactions. If there are many transactions, but few miners, then would transactions begin taking too long to become verified?
Yes, there is a byte limit to the size of a block. If blocks become infrequent enough due to decreased mining power, it's conceivable that a backlog of transactions would build up. At the next difficulty adjustment, though, speed would pick back up again.