I am in a similar situation, I am full time employed (as a developer) by 1 person GmbH. I work with a tax agency that has not said anything about this.
The thing is, that just b/c someone at the tax agency said something, doesn't mean that it is right nor respected when they audit you. A lot of things in this area are not clearly defined and thus open to interpretation leading to all kinds of weird decisions.
Unfortunately your tax agency is wrong. Finanzamt does not care much. It is the Rentenversicherung that you should care about. They might end up calling Scheinselbstständig. Better get at least one more client to work with.
The following article (German) is a good read on that: http://www.schwerd.info/sozialversicherungsrecht/scheinselbs...