Horovod hangs -- include/shm.h:42 NCCL WARN Cuda failure 'invalid argument' · Issue #893 · horovod/horovod
horovod-0.16.0/examples$ mpirun --prefix /opt/openmpi/ -np 8 -H heinzel135:4,heinzel136:4 -bind-to none -map-by slot -x NCCL_DEBUG=INFO -x LD_LIBRARY_PATH -x PATH -mca pml ob1 -mca btl ^openib python tensorflow_mnist.py # ... INFO:tensorflow:Create CheckpointSaverHook. INFO:tensorflow:Graph was finalized. INFO:tensorflow:Graph was finalized. INFO:tensorflow:Graph was finalized. INFO:tensorflow:Gra
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
Horovod with oneCCL failed · Issue #2321 · horovod/horovod
horovod/horovod/runner/mpi_run.py at master · horovod/horovod
Add in-place broadcast for TensorFlow by maxhgerlach · Pull Request #3128 · horovod/horovod
`ncclCommInitRank failed: unhandled system error` with 3 local nodes (HOROVOD_GPU_OPERATIONS=NCCL, 0.20.3), hang with 4 nodes (HOROVOD_GPU_OPERATIONS=NCCL 0.20.3, HOROVOD_GPU_BROADCAST=NCCL 0.19.5) · Issue #2395 · horovod/horovod
Segmentation fault with `hvd.barrier` · Issue #3308 · horovod/horovod
horovod/horovod/torch/mpi_ops.py at master · horovod/horovod
fix barrier seg fault and added test to mix it with multiple collectives by Tixxx · Pull Request #3313 · horovod/horovod
horovod/docs/mpi.rst at master · horovod/horovod
horovod run with pytorch1.6 torch.cuda.amp will cause nan · Issue #2175 · horovod/horovod
What's the reasonable NEGOTIATE_ALLREDUCE duration? · Issue #871 · horovod/horovod
Really need help! Try horovodrun performs slower! · Issue #1403 · horovod/horovod
AWS Benchmarks sanity check · Issue #396 · horovod/horovod
can horovod run distributed in muti containers on the same nodes · Issue #451 · horovod/horovod
horovod/docs/troubleshooting.rst at master · horovod/horovod
FORCE-TERMINATE AT Data unpack would read past end of buffer:-26 - error grpcomm_direct.c · Issue #1040 · horovod/horovod
Support .step(synchronize=False) for gradient clipping & NVIDIA AMP by alsrgv · Pull Request #1104 · horovod/horovod
how to Deploy and Run a Horovod framework with Mellanox Network（ROCE and Lossless queue） · Issue #654 · horovod/horovod
Horovod docker with Cuda-10.0 support · Issue #882 · horovod/horovod
Add RDMA instructions to Dockerfile by alsrgv · Pull Request #1150 · horovod/horovod