Anything that falls in that 'personal data' segment above that belongs to me has to be obtained with my prior, explicit consent. And these bits of data with my name or other details in them must be included if I send Google a "right to be forgotten" or "show me all the data you've got about me" request. That's GDPR in a nutshell.
This might be a grey area for now, as both GDPR and listening devices are both quite new. But Google, Amazon & co aren't super popular with EU regulators and governments, so they might side with users' rights on this one.