I never thought that my wireless headphones had enough juice to fill my house with smoke, to burn for 15 minutes, or result in chemical burns to the user.
They're made of thermoplastic, practically everything plastic burns except for the few things around which are regulated to be impregnated with fire retardants (children's clothes, furniture, etc). Unless what you're holding is made entirely of metal or ceramic, you can be pretty sure that it is some degree of fire risk, if it contains a battery or not.
The energy densities of modern batteries are a couple factors lower than that of gasoline, and much lower than TNT. Gasoline is incredibly energy-dense, but its energy density is comparable to butter. Do you worry about sticks of butter exploding on you? No, you don't. Gasoline is dangerous because it really likes to evaporate and the mixture of gasoline fumes and air ignites at the slightest provocation. Batteries are dangerous because they can go into thermal runaway from an internal short if the barrier between the electrodes fails. The energy density is not what matters here.