We develop an efficient approach to impose filter orthogonality on a convolutional layer. Our proposed orthogonal convolution requires no additional parameters and little computational overhead, and consistently outperforms the kernel orthogonality alternative on a wide range of tasks such as image classification and inpainting under supervised, semi-supervised and unsupervised settings. Project page: http://pwang.pw/ocnn.html