There are rumors of DNS resolvers deciding that some TTLs are “too low” to be valid, and applying their own default TTL value instead, thereby negating any benefit a low TTL would have had.
If the DNS server is out for more than 1 minute you'll get an outage. You also increase latency by preventing anybody from caching the response for more than 1 minute.