Hacker News new | ask | show | jobs
by p1esk 2171 days ago
2x 2080ti would be faster than titan rtx, provide the same amount of memory, and would be cheaper.
4 comments

Unfortunately multi-GPU training doesn't scale linearly yet [1] so it's often a better call to get a larger card then two smaller ones, at least for the single-model case.

[1] https://github.com/keras-team/keras/issues/9204

(non-TF) Keras has notoriously bad multi-GPU support though (and was generally not well optimized. Case in point, the latest version just re-exports/forwards to tf.keras).

Looking at something like https://lambdalabs.com/deep-learning/gpu-benchmarks or https://github.com/tensorpack/benchmarks/tree/master/other-w..., multi-gpu scaling on 2080tis seems pretty darn close to linear. Plus, there are benefits to having more than one accelerator handy on a local workstation. For one, it's much easier to have multiple experiments running simultaneously or to run parallel training (e.g. hyperparameter search or RL episodes). Given that only the uber-expensive enterprise cards have proper virtualization/time sharing, trying this workflow on a Titan RTX will most likely be suboptimal unless you always run models that can make use of most of the memory and compute (no RNNs, no Neural ODEs, etc.)

If NVIDIA gave me a Titan RTX for free, I would use it too.
I'd even buy my own waterblock in that case.
Lots of drawbacks with this approach: - More heat - More power consumption - More noise - The GPU memory isn't addressable as a single unit
> provide the same amount of memory

Are you sure? The last time I checked the situation with NVLink memory pooling with 2080ti cards was very unclear.