If you weren't that wealthy, you might have part-time servants. If you were wealthy, your servants would eat in their own little dining room in your manor, and shop for food for you and the servants in the same shopping trip
In many cases the correct term is slave. Servants are different and generally treated better. Although slave also comes with a very large spectrum of treatment (it mostly wasn't as bad as your high school history makes it out - bad yes, but not quite that bad)