Hangs on dist init process group in distribute py

[JRMeyer]

>>> alchemi5t
[September 5, 2019, 7:16am]

I've been trying to use multiple gpus for training but the training
hangs(Responsive, but indefinitely stays here without aby output or
error) at initializing process group. Any insights on how to fix this?

Tried pytorch 0.4.1 and 1+.

'distributed':{
'backend': 'nccl',
'url': 'tcp: slash / slash /localhost:23456'
},

[This is an archived TTS discussion thread from discourse.mozilla.org/t/hangs-on-dist-init-process-group-in-distribute-py]

[JRMeyer]

>>> erogol
[September 5, 2019, 10:50am]

I don't have something to replicate your problem hence hard to guess.
But it might be about something your machine's setup, or pytorch version
or dataloader etc.

[Archived Post]

[JRMeyer]

>>> alchemi5t
[September 19, 2019, 4:54pm]

Fixed it by installing apex. Not sure why i needed it though.

[Archived Post]