That's a good idea. Travel time may be an issue (I'm looking to network with developers outside my company). Also I feel that the type of people who won't take a detour on the way home from work will also eat lunch at their desk so that they can leave a few minutes sooner.
People with kids have very particular timetables they have to hit. Usually their lunch time would not be so regimented. Truthfully, though the best thing is to try it out and see the kind of people you get. I made it a point to engage with every attendee I did not immediately recognize and find why were they there and if this was their first meetup what got them there.