So you can do that if you want, you would probably need some USB hubs to accomplish, and probably a logical way to map out your serial connections, but it should be doable.
Or in the case of how I did it, i setup a multi-port KVM, that allowed me to manage x number of devices (with video) from a single unit. In this instance, a serial connection was used from the pi to the actual KVM which allowed me to switch inputs etc, from the gui.