the table is so small you don't actually need to decode varints. any serious application needing speed would build a lookup table for all the common values -- in fact we will. size does matter. speed does matter.
Complexity matters as well. Is it really worth the pain (and attack surface) of the extra code for a varint data type in return for 0.002% reduction in size?