Style transfer using neural networks
Introduction
Style transfer (also known as color transfer) is a technique that transfers style from one image (or multiple images) to another while maintaining its structure. So if we have a black-and-white photograph of a building and another, already colorful photograph (or many photographs) of similar buildings, then we can colour the first photograph using the learned style.
Algorithm
The key is to separate the structure of images from their style and then move the extracted style from one image to another while maintaining its unchangeable structure. As in the case of most image tasks, convolutional neural networks have been used here as well. They are used to build a hierarchical representation of both images - from simple patterns (style related) to complex structural elements. Then, the Neareast-Neighbor Field Search algorithm is used to build relations between the two images, so that it is possible to know what part of one image will be used for coloring a fragment of the other image. The final step is color transfer based on information from previous steps. The algorithm used here proved to be much more effective than the previous solutions.
Performance
The algorithm (for the sample images shown here) was launched on the Intel E5 2.5GHz processor and NVIDIA Tesla K40c GPU. The execution time was about 80s for 700x500 resolution images.
Applications:
- coloring of black and white photographs
- support for architectural design (generating different styles for the same building)
- generating objects, characters in games
- styling, retouching photos