Predicting the stay time of private cars has various applications in location-based services and traffic management. Due to the associated randomness and uncertainty, achieving the promising performance of stay time prediction is a challenge. We propose an RNN-based encoder model to solve this problem, which consists of three components, i.e., an encoder module, an exception module, and an MLP dropout. First, we encode the stay behaviour into hidden vectors at a specific time to avoid the effect of time sparsity. The encoder module utilizes a multilayer perceptron (MLP) to learn spatiotemporal features from the historical trajectory data, such as the inherent relationship between the stop points and corresponding stay time. We proved a linear relationship problem that cannot be ignored in the stay time prediction problem. In particular, we have added basic arithmetic logic units to the network framework to find linear relationships. By reconstructing the basic arithmetic and logical relations of the network, we have improved the ability of the neural network to handle linear relations and the extrapolation ability of the neural network. Our method can remember the number patterns seen in the training set very well and infer this representation reasonably. Moreover, we utilize the dropout technique to prevent the prediction model from overfitting. We perform extensive experiments based on a large-scale real-world private car trajectory dataset. The experimental results demonstrate that our method achieves an RMSE of 0.1429 and a MAPE of 55.8533%. Furthermore, the results verify the effectiveness and advantages of the proposed model when compared with the benchmarks.