Is there really no padding=same option for PyTorch's Conv2d?

Question

I'm currently working on building a convolutional neural network (CNN) that will work on time series data.

More specifically, the data is financial data of shape (100, 40), with 100 meaning 100 time stamps and 40 meaning 40 features.

The CNN that I'm using uses asymmetric kernel sizes (i.e. 1 x 2 and 4 x 1) and also asymmetric strides (i.e. 1 x 2 for the 1 x 2 layers and 1 x 1 for the 4 x 1 layers).

In order to maintain the height dimension to stay 100, I needed to apply some padding to the data. I was looking into ways to do this and noticed that people who use TensorFlow or Keras simply do padding='same', but this option is apparently unavailable in PyTorch according to many resources I've found including this thread on Github.

What I've found is that according to some answers in this question and also this answer on the PyTorch discussion forum, I can manually calculate how I need to pad my data and can use torch.nn.ZeroPad2d to solve my problem, as it seems that normal torch.nn.Conv2d layers don't support asymmetric padding (I believe that the total padding I need is 3 in height and 0 in width).

The experimental code that I've written to test this out is like this:

import torch
import torch.nn as nn

conv = nn.Conv2d(1, 1, kernel_size=(4, 1))
pad = nn.ZeroPad2d((0, 0, 2, 1)) # Add 2 to top and 1 to bottom.

x = torch.randint(low=0, high=9, size=(100, 40))
x = x.unsqueeze(0).unsqueeze(0)

y = pad(x)

x.shape # (1, 1, 100, 40)
y.shape # (1, 1, 103, 40)

print(conv(x.float()).shape)
print(conv(y.float()).shape)

# Output
# x -> (1, 1, 97, 40)
# y -> (1, 1, 100, 40)

As you can see, it does work in the sense that the size of dimensions is remaining the same. However, I've been wondering is there really no padding='same' option out there? Also, how do we know whether to apply the padding 2 to the top or to the bottom?

Thank you.

---

Edit

This is a little late, but in case anyone's curious how I solved this issue I basically manually added padding in order to emulate the padding=same option.

Answer 1

I had the same issue some time ago, so I implemented it myself using a ZeroPad2d layer as you are trying to do. Here is the right formula:

from functools import reduce
from operator import __add__

kernel_sizes = (4, 1)

# Internal parameters used to reproduce Tensorflow "Same" padding.
# For some reasons, padding dimensions are reversed wrt kernel sizes,
# first comes width then height in the 2D case.
conv_padding = reduce(__add__, 
    [(k // 2 + (k - 2 * (k // 2)) - 1, k // 2) for k in kernel_sizes[::-1]])

pad = nn.ZeroPad2d(conv_padding)
conv = nn.Conv2d(1, 1, kernel_size=kernel_sizes)

print(x.shape) # (1, 1, 103, 40)
print(conv(y.float()).shape) # (1, 1, 103, 40)

Also, as mentioned by @akshayk07 and @Separius, I can confirm that it is the dynamic nature of pytorch that makes it hard. Here is a post about this point from a Pytorch developper.

Answer 2

It looks like there is now, in pytorch 1.9.1, according to the docs.

padding='valid' is the same as no padding. padding='same' pads the input so the output has the shape as the input. However, this mode doesn't support any stride values other than 1.

Answer 3

padding='same' and padding='valid' is possible in Pytorch 1.10.0+. However, 'same' and 'valid' for padding is not possible for when stride > 1.