torchfoo.distributed.all_concat#

torchfoo.distributed.all_concat(x)[source]#

Concatenate a Tensor distributed on multiple GPUs. Backprop-able.

Parameters:: x (Tensor) – a Tensor on this device
Returns:: Concatenated tensor from all processes along dim 0.
Return type:: Tensor