VRAM-hungry LSTM monster

Technical Help
3 
wsadmin@AIML1001:/usr/lib/python3/dist-packages$ find tensorflow*/ libcudnn_ops_train.so.8
find: ‘libcudnn_ops_train.so.8’: No such file or directory

Hello, thanks for the detailed response. What is your LD_LIBRARY_PATH? Mine was not set before running export LD_LIBRARY_PATH=/usr/lib/python3/dist-packages/tensorflow:$LD_LIBRARY_PATH so it became /usr/lib/python3/dist-packages/tensorflow:
And here is me running the workaround:

wsadmin@AIML1001:/home/mher/projects/Untitled Folder$ printenv | grep LD_
LD_LIBRARY_PATH=/usr/lib/python3/dist-packages/tensorflow:
wsadmin@AIML1001:/home/mher/projects/Untitled Folder$ sudo TF_CPP_MIN_LOG_LEVEL=3 python3 LSTM-hell.py
No protocol specified
Traceback (most recent call last):
  File "LSTM-hell.py", line 52, in <module>
    regressor.fit(x, y_labels, batch_size=1)
  File "/usr/lib/python3/dist-packages/keras/utils/traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/usr/lib/python3/dist-packages/tensorflow/python/eager/execute.py", line 54, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InternalError: Graph execution error:

Failed to call ThenRnnForward with model config: [rnn_mode, rnn_input_mode, rnn_direction_mode]: 2, 0, 0 , [num_layers, input_size, num_units, dir_count, max_seq_length, batch_size, cell_num_units]: [1, 1, 1, 1, 2, 1, 1]
         [[{{node CudnnRNN}}]]
         [[sequential/lstm/PartitionedCall]] [Op:__inference_train_function_2337]

Here is the LSTM_hell.py file content:

wsadmin@AIML1001:/home/mher/projects/Untitled Folder$ cat LSTM-hell.py
#!/usr/bin/env python
# coding: utf-8

# In[1]:


import tensorflow as tf
import numpy as np
import pandas as pd
import tensorflow.keras as k
from tensorflow.keras.layers import LSTM, Dense, Reshape, RepeatVector
from tensorflow.keras.models import Sequential


# In[2]:


y_posterior = lambda x: 2.71 ** x + 5 * x + 1.2

x = np.random.normal(0, 1, (1, 2))
y_labels = y_posterior(x)[:, 0]


# In[3]:


y_labels


# In[4]:


#get_ipython().system('nvidia-smi')


# In[5]:


regressor = Sequential()

# regressor.add(Dense(1, input_shape=(1,)))
regressor.add(LSTM(1,
    batch_input_shape=(1, 2, 1)))
# regressor.add(Dense(1))

regressor.compile(optimizer = 'sgd', loss = 'mean_squared_error')


# In[6]:


regressor.fit(x, y_labels, batch_size=1)


# In[7]:


x.shape


# In[8]:


y_labels.shape


# In[9]:


x


# In[10]:


#get_ipython().system('nvidia-smi')


# In[ ]:


import sys
sys.version