markd
November 27, 2023, 3:26pm
2
The simple answer is you are running distrubuted, and parent process is telling you that one of the child processes failed. It is not clear for which reason, but it could be:
In sufficient resources for the child process (GPU, GPU memory, CPU, memory)
Perhaps if this is a remote host it could be different python script, data, libraries, etc. If this is distributed across nodes. (or the Anaconda environment you are running on for the ‘parent process’)
It is mentioned in with similar issues:
opened 08:42AM - 19 May 23 UTC
when i run this command:
`torchrun --nproc-per-node 1 --master_port 25641 train… .py --cfg-path train_configs/minigpt4_stage2_finetune.yaml`
this error occurs, how can i fix it?
```
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 81571) of binary: /home/tiger/miniconda3/envs/minigpt4/bin/python
Traceback (most recent call last):
File "/home/tiger/miniconda3/envs/minigpt4/bin/torchrun", line 8, in <module>
sys.exit(main())
File "/home/tiger/miniconda3/envs/minigpt4/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
return f(*args, **kwargs)
File "/home/tiger/miniconda3/envs/minigpt4/lib/python3.9/site-packages/torch/distributed/run.py", line 794, in main
run(args)
File "/home/tiger/miniconda3/envs/minigpt4/lib/python3.9/site-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/home/tiger/miniconda3/envs/minigpt4/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/tiger/miniconda3/envs/minigpt4/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
train.py FAILED
------------------------------------------------------------
Failures:
<NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2023-05-19_16:43:27
host : n136-117-136.byted.org
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 81571)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
========================================================================================================================
```
And of course enable trackback on the child process/worker:https://pytorch.org/docs/stable/elastic/errors.html