One thing that I notice is that you are running out of memory (you can see that from the CUDA_ERROR_OUT_OF_MEMORY errors that you’re getting:
2018-04-14 08:01:57.171981: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] failed to allocate 152.62M (160038912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-04-14 08:01:57.172873: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] failed to allocate 137.36M (144035072 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
Can you try to re-run your job with a smaller batch size?