The joke is that the tackle box isn't used for fishing, isn't it? The father doesn't go fishing to catch fish, he goes fishing to relax, having baits and the such isn't a requirement for the latter.
And if the teenage girl knows that, perhaps she goes with him, somedays, to also relax away from people? And her invitation can mean that her family (dad?) taught her about being alone and relaxing?
(who, in order to avoid being put in an accurate but damning box for his habits by society, pretends to inhabit a still problematic —but much, much more socially acceptable— box?)