[MrEldritch]

Given that you'll never see a lone quark (they're physically forbidden - a naked quark is so energetically-unfavorable that attempting to create one would inherently require enough energy to create a second quark to go along with it), and two-quark mesons are both incredibly short-lived (a few nanoseconds at the longest) and act more like force carriers than anything else, then the three-quark baryons - of which protons and neutrons are the only ones that last long enough to care about from an engineering perspective - are the smallest (quark-based) units that could really be called "matter."

[pharrington]

Quarks are charged, and thus interact with photons. If reacting with photons is not what you meant by "Given that you'll never see a lone quark", but instead you mean something like, for example, `a single one can be formed as a first order byproduct of a physical interaction`, there is no coherent definition of matter that I am aware of where atoms are the smallest unit, but photons or electrons are not.

edited: updated for precision