current position:Home>It is AI's "hair generating" artifact that makes you say goodbye to baldness by generating bangs and adjusting hairline

It is AI's "hair generating" artifact that makes you say goodbye to baldness by generating bangs and adjusting hairline

2021-08-31 20:37:34 Heart of machine

When the gradually backward hairline and increasingly sparse bangs become the source of anxiety , In order to meet the users' yearning for thick hair , Years of deep cultivation Artificial intelligence Mido's technology brain in the field —— Metu Imaging Laboratory (MT Lab) Based on Deep learning Technical advantages accumulated in the field , Landing a number of hair generation projects and realizing high-definition real hair texture generation , At present, it has taken the lead in Meitu's core products Meitu XiuXiu and overseas products AirBrush Online bangs are generated 、 Hairline adjustment and sparse area reissue , Meet the diversified needs of users for hairstyles .

among , The fringe generation function can be based on a custom generation area , Generate different styles of bangs ( Pictured 1.1-1.3).

chart 1.1: Bangs generation ( Left : Original picture , Right : Effect drawing of full bangs generation )

chart 1.2: Bangs generation ( Left : Original picture , Right : Effect drawing of full bangs generation )

chart 1.3: Multiple bangs generation renderings

The hairline adjustment function keeps the original hairline style , Different heights of hairline can be adjusted ( Pictured 2.1-2.2):

chart 2.1: Comparison before and after hairline adjustment

chart 2.2: Comparison chart of hairline adjustment

The challenge of hair generation

Hair editing as a general generation task , In the process of landing practice, it still faces the following key technical bottlenecks to be broken through :

  • The first is the acquisition of generated data . Take the bangs generation task as an example , When generating a specific style of bangs , Whether a person has bangs or not is the most ideal paired data , But the possibility of obtaining this type of real data is very low . meanwhile , If a specific style of fringe data is collected , In a way that forms an unpaired dataset of specific attributes , Then obtaining high-quality and multi style data requires a high cost , Basically no operability ;

  • The second is the generation of high-definition image details . Because the hair has complex texture details , adopt CNN It is difficult to produce real and ideal hair . among , With paired data , Although you can design something similar Pixel2PixelHD[1]、U2-Net[2] Wait for the Internet to go ahead Supervised learning , However, the definition of the image generated by this method is still very limited ; In the case of unpaired data , Generally through similar HiSD[3]、StarGAN[4]、CycleGAN[5] Attribute conversion is carried out to generate , The pictures generated in this way not only have poor clarity , There is also instability in the generation of target effects 、 Problems such as unreal generation effect .

In view of the above situation ,MT Lab Based on huge data resources and outstanding model design ability , With the help of StyleGAN[6] It solves the two core problems of pairing data generation and HD image details faced by hair generation task .StyleGAN As the main direction of the current generation field —GAN( Generative adversary network ) stay Image generation The main representative in the application , It is an unsupervised high-definition video based on style input Image generation Model .StyleGAN Can be based on 7 Thousands of copies 1024*1024 High definition face image training data FFHQ, Through sophisticated network design and training skills to generate clear and realistic image effects . Besides ,StyleGAN It can also have the ability of attribute editing based on style input , adopt Hidden variables The editor of , Realize the modification of image semantic content .

chart 3: be based on StyleGAN Generated image

Metu based StyleGAN New hair editing scheme
1. Paired data generation

StyleGAN The most direct way to generate pairing data is in w + Directly edit the implicit vector of related attributes in space , Generate related properties , The implicit vector editing methods include GanSpace[7]、InterFaceGAN[8] And StyleSpace[9] wait . however , such Image generation The method usually implies that the attribute vector is not decoupled , That is, the generation of target attributes is often accompanied by other attributes ( Background and face information ) Make a difference .

therefore ,MT Lab combination StyleGAN Projector[6]、PULSE[10] And Mask-Guided Discovery[11] And other iterative reconstruction methods to solve the problem of generating hair pairing data . The main idea of this scheme is to edit the original picture briefly , Obtain a rough target attribute reference image , Take it and the original image as the reference image , Re pass StyleGAN Perform iterative reconstruction .

Take light hair color as an example , You need to dye the hair area in the original picture with a uniform light color block , Through Downsampling Obtain a rough edit sketch as the target attribute reference image , stay StyleGAN In the iterative reconstruction process , The generated image is monitored for similarity with the original image at high resolution scale , To ensure that the original information outside the hair area does not change .

On the other hand , Generate pictures through Downsampling Supervise with the target attribute reference image , To ensure that the generated light hair color area is consistent with the hair area of the original image , The two iterations generate the desired image under supervised balance , At the same time, we also obtained the matching data of a person with or without light hair ( Refer to the following figure for the complete process 4).

It is worth emphasizing that , During the implementation of the scheme, it is necessary to ensure that the target attribute of the generated image is consistent with the reference image , Also ensure that the generated image is consistent with the original image information outside the target attribute area ; It is also necessary to ensure that the implicit vector of the generated image is in StyleGAN In the implicit vector distribution of , To ensure that the final generated image is a high-definition image .

chart 4 : Dye light hair StyleGAN Schematic diagram of iterative reconstruction

Besides , Based on the idea of this scheme , In the field of hair generation, the matching data of hairline adjustment can also be obtained ( Here's the picture 5)、 Matching data generated by bangs ( Here's the picture 6) And matching data with fluffy hair ( Here's the picture 7).

chart 5: Hairline pairing data

chart 6: Bangs matching data

chart 7: Fluffy hair matching data

2. Paired data gain

Based on iterative reconstruction , The matching data corresponding to the matching data can also be obtained StyleGAN Hidden vectors , Through the hidden vector interpolation It can also achieve data gain , Then a sufficient number of paired data can be obtained .

Take the pairing data of hairline adjustment as an example , Here's the picture 8 Shown ,(a) and (g) It's a set of paired data ,(c) and (i) It's a set of paired data , Between each set of paired data , Can pass interpolation The pairing data with different degrees of adjustment of hairline . Such as (d) and (f) Namely (a) and (g)、(c) and (i) Between interpolation .

similarly , Implicit vectors can also be used between two sets of paired data interpolation Get more pairing data . Such as (b) and (h) Namely (a) and (c)、(g) and (i) adopt interpolation Obtained pairing data . Besides , adopt interpolation The obtained pairing data can also generate new pairing data , Such as (e) yes (b) and (h) Paired data obtained by difference , Based on this, the demand for ideal hairline adjustment pairing data can be met .

chart 8: Paired data gain

3. image-to-image Generate

be based on StyleGan After the paired data are obtained by iterative reconstruction , You can go through pixel2piexlHD The model carries out supervised learning and training , such image-to-image The method is relatively stable and robust , But the clarity of the generated image can not achieve the ideal effect , So choose through image-to-image The model adopts StyleGAN Pre training model to help improve the generation details . Conventional StyleGAN Realization image-to-image The way is through encoder The network obtains the image hidden vector of the input graph , Then edit the hidden vector directly , Finally, the target attribute Image generation , However, the similarity between the image generated in this way and the original image is often low , Unable to meet the requirements of editing based on the original image .

therefore MT Lab The method of implicit vector editing is improved , On the one hand, the original image encode The implicit vector to the target attribute , Omit the steps of intermediate hidden vector editing ; On the other hand, it will encoder Characteristics and characteristics of network StyleGAN The characteristics of the network are fused , Finally, the target attribute image is generated by the fused features , To maximize the similarity between the generated image and the original image , Overall network structure and GLEAN[12] The model is very similar , This method takes into account two main problems: image high-definition detail generation and original image similarity restoration , This also completes the whole process of hair generation with high definition and real detail texture , The details are as follows: 9:

chart 9: Hair generation network structure

be based on StyleGAN Expansion of edit generation scheme

be based on StyleGAN Editing the generation scheme can reduce the difficulty of the design of the generation task scheme , Improve the R & D efficiency of generating tasks , Achieve a significant improvement in the generation effect , At the same time, it also has high scalability . among , combination StyleGAN The method of generating ideal hair pairing data greatly reduces the difficulty of image editing task , For example, expand the attributes concerned by the scheme beyond the hair , You can get more attribute pairing data , For example, pairing data of facial features replacement ( Here's the picture 10), Thus, we can try to practice any face attribute editing task .

Besides , With the help of StyleGAN Pre training model implementation image-to-image The method can ensure the clarity of the generated image , Therefore, it can also be extended to such as Image restoration Image denoising 、 Image super-resolution and other more general generation tasks .

chart 10: Pairing data of facial feature replacement : Original picture ( Left ), Refer to the figure ( in ), Result chart ( Right )

at present , MT Lab Already in Image generation New technological breakthroughs have been made in the field , It realizes the generation of high-definition portrait and achieves fine control . In addition to landing hair generation ,MT Lab It not only realizes tooth shaping 、 Eyelid formation 、 Face attribute editing functions such as makeup migration , It also provides AI Face changing 、 Grow old 、 Become a child 、 Change gender 、 Generate smiles and other new ways of playing popular social networks , A series of cool playing methods bring more fun to users 、 Better use experience , It also shows the strong technical support and R & D investment behind it .

future , Deep learning It will still be MT Lab One of the key research areas , It will also continue to deepen the research on cutting-edge technologies , Continuously deepen the technological innovation and breakthrough of the industry .

reference :
[1] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao,Jan Kautz, and Bryan Catanzaro. High-resolution image syn-thesis and semantic manipulation with conditional gans. In CVPR, 2018.
[2] Xuebin Qin, Zichen Zhang, Chenyang Huang, Masood Dehghan, Osmar R Zaiane, and MartinJagersand. U2-net: Going deeper with nested u-structure for salient object detection. Pattern Recognition, 2020.
[3] Xinyang Li, Shengchuan Zhang, Jie Hu, Liujuan Cao, Xiaopeng Hong, Xudong Mao, Feiyue Huang, Yongjian Wu, Rongrong Ji. Image-to-image Translation via Hierarchical Style Disentanglement. InProc. In CVPR, 2021.
[4] Choi, Y., Choi, M., Kim, M., Ha, J.W., Kim, S., Choo, J.: Stargan: Unified genera-tive adversarial networks for multi-domain image-to-image translation. In CVPR, 2018.
[5] Choi, Y., Choi, M., Kim, M., Ha, J.W., Kim, S., Choo, J.: Stargan: Unified genera-tive adversarial networks for multi-domain image-to-image translation. In CVPR, 2018.
[6] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten,Jaakko Lehtinen, and Timo Aila. Analyzing and improvingthe image quality of StyleGAN. InProc. In CVPR, 2020.
[7] Erik H ̈ark ̈onen, Aaron Hertzmann, Jaakko Lehtinen, andSylvain Paris. Ganspace: Discovering interpretable gancontrols. In NIPS, 2020.
[8] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Inter-preting the latent space of gans for semantic face editing. In CVPR, 2020.
[9] Zongze Wu, Dani Lischinski, and Eli Shecht-man. StyleSpace analysis: Disentangled controlsfor StyleGAN image generation. In arXiv, 2020.
[10] Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi,and Cynthia Rudin. Pulse: Self-supervised photo upsam-pling via latent space exploration of generative models. In CVPR, 2020.
[11] Mengyu Yang, David Rokeby, Xavier Snelgrove. Mask-Guided Discovery of Semantic Manifolds in Generative Models. In NIPS Workshop, 2020.
[12] K. C. Chan, X. Wang, X. Xu, J. Gu, and C. C. Loy, Glean: Generative latent bank for large-factor image super-resolution, In CVPR, 2021.

copyright notice
author[Heart of machine],Please bring the original link to reprint, thank you.

Random recommended