Normally that's correct except that the USB-C pinout is well designed to not put GND and VBUS near each other so it's pretty difficult to actually short anything. Still it's better to use something non-conductive.
I cleaned my phone usb connector like this last week, after the charging cable wouldn't go in, because something was lodged in there. Just used a pin I found on the wall at work to scrape it out. No problem ¯\_(ツ)_/¯
I personally use the toothpick end of flossing picks. They're usually very thin and the plastic material doesn't cause any hard abrasion.