Improving a Convolutional Neural Network for simulating a fabric

At first glance, I'm not seeing any batch normalization (eg nn.BatchNorm2d) between convolution layers which would help mitigate the exploding/losing stability of your predicted points. Also, from my experience with position-based prediction models, the noise being added into the input data tends to be very finicky (at least compared to other types of models like image segmentation for example), in that no/too little noise and the model gets stuck in suboptimal positions while adding in a bit too much noise prevents the model from learning (ie your added noise could make it or break it, try changing the range and scale.. a lot).