Maybe the time is hashing your password. This is designed to take as long as it can while still being reasonable. 1 second on a fast machine isn't unheard of.
Perhaps it could be a two-level thing then, where you first decrypt e.g. the list of keys and then read the one you want and get a "start/end" offset to read/decrypt just the value for that key from another file. So keys and values are still decrypted, but you don't have to decrypt the whole bundle in one step to get the value.