I would be curious if the event or the organizer had any connection to the venue. It wouldn't keep me from going, and it might make me notice it more just for the unexpected juxtaposition.
As the organiser in question. I don't have any religious affiliation. I'm just aware of many wonderful buildings around my citys who's primary purpose is that of worship but design see's them very well suited to a conference.