Fluorine on an aromatic ring is not labile: other examples of substances with fluorinated aromatic rings are fluoxetine(prozac), citalopram (celexa) and paroxetine (paxil), which are antidepressants many people are taking daily without fluorine-related toxicity
Flourine/halogenated alkaloids in general like these may be toxic, but it's not because of the flourine. The halogens are used because they form very strong bonds in those positions (among other reasons)