Long time no see ,AR How's it going

Three years and three years , Has the popular virtual technology disappeared ？ This article will lead readers to understand the current AR Technology development status , And a brief introduction to AR The key technology , Finally, show me how to run a AR Demo. If you are helpful, please don't hesitate to praise 、 Comment on 、 Leaving a message. .

One 、 Introduction to virtual reality technology

1.1 AR/VR/MR Conceptual differentiation

AR(Argumented Reality)： An enhancement of reality , Device recognition of reality ( shape 、 Location 、 action 、 edge ), So as to superimpose virtual information on reality .

VR(Virtual Reality)： Immersive virtual reality , Provide users with a complete virtual world , You don't have to think about the real world .

MR(Mixed Reality)： Hybrid virtual technology , Combine real and virtual environments .MR Will enhance vision ( Modify real objects ), For example, in medicine MR Technology to see through the human body , So that doctors can see the blood vessels in the body visually , More accurate completion of surgery . A lot of people think MR Just a new concept hype , In essence AR.

The article was written as 2021 year 12 month , here VR Technology and equipment have taken shape , Such as a variety of VR game , And with consumer maturity OculusQuest 2 equipment , Based on the above concept of ecological meta universe, it has been heated .

comparison VR, To achieve a mature consumer experience ,AR There are still many technical difficulties . This article mainly discusses AR Development context and current situation of ,VR Technology is not in the main discussion .

1.2 AR Historical development and current situation

• AR The beginning of history

1992 Boeing researchers proposed AR This phrase of , A few years later AR The concept of is slowly becoming hot , But it's always in the laboratory .

• first AR SDK

To 1999 year ,Kato Hirokazu Dominated AR Toolkit project , You can do it 2D Object recognition , And let the virtual object follow the real object .2000 year ,ARToolkit project Open source ,2005 year , It supports Symbian system , Become the world's first mobile open source AR-SDK.2008 Year begins , It has supported IPhone and Android. By the end of 2021 year 12 month ,ARToolkit Support most popular platforms in the world , And can be integrated into Unity platform , Cooperate with each other to develop AR game .

• first AR Body feeling game

2003 Sony in Playstation Go public EyeToy, Detect the player's actions through the external camera , To interact with the game . A sense of body for a time AR Games are popular .

• tipping AR Conceptual Google Glass

although AR Technology has been developing steadily , But it will be AR The concept is brought to the general public Google Glass.Google Glass Using the principle of optical reflection projection （HUD）, A miniature projector projects light onto a reflective screen , Then it is refracted to the human eye through a convex lens , Implement the so-called “ Primary amplification ”, Form a large enough virtual screen in front of people , It can display simple text information and various data

although Google Glasss More of a “ A cell phone on your face ”, quite a lot AR Technology involved in the field , Such as environmental testing , User interaction technology does not involve . But it involves AR display technique , The light information is projected onto the eyeball . Therefore, its emergence has aroused widespread discussion , One is wearable devices , One is AR technology .

• AR The first generation of black technology in the world ——HoloLens

2015 In, Microsoft released the AR Headgear HoloLens, It can detect the external environment , Integrate the virtual object with the display environment , And it completes the interaction by tracking the user's gestures . This is the first relatively mature... In a real sense AR equipment , It's small , Have independent computing and storage capacity , When wearing, it does not affect the user to view the external environment .2019 year , Microsoft released HoloLens 2 It solves the problem of narrow vision of the first generation equipment , And greatly improve the accuracy of gesture tracking .

HoloLens2 Although relatively mature , But the cost is high , As of the completion of this article , Cost as much as 3500 dollar . in addition , Although it is relative to others AR The equipment is small , But still heavy 500g,iPhone 12 heavy 152g, It's hard for ordinary users to accept the weight of wearing three mobile phones on their heads . The good news is , Microsoft revealed HoloLens3 The weight will be less than 90 g .

• AR Other players in the world

Widely questioned by the public as ”PPT Swindlers Company “ Of Magic Leap 2017 First generation products released in Magic Leap One. Besides , Both apple and Samsung are rumored to be in 2022 Released its AR glasses . Domestic enterprises have cool Xview series 、 Bright vision 、 Ali nail NrealLightAR Launched AR Product equipment .

• AR The world's iconic game

Niantic The company in 2017 Released Pokémon Go game , This one is based on LBS The game is popular all over the world , The first month of the release is more than 2 Billion dollars , list 3 After years, the total water flow will reach 26.5 Billion dollars .Pokémon Go The success of quickly detonated AR Game market . thereafter AR Games continue to emerge , Such as construction 、 Chess, etc .

• Mobile AR The popularity of SDK

At present, mobile terminals AR SDK Fragmentation is serious , There are dozens of differences at home and abroad sdk, Here are some influential sdk.

• Apple ARKit：2017 Released so far , Considered the most powerful , The most commercial potential sdk. coordination SceneKit 、SpriteKit、RealityKit Wait until Apple's framework library can develop a very cool app, The advantage is that almost all iOS The device can be used seamlessly （iOS 11.0+,iPhone 6s+）.
• Google ARCore： Support at the same time Android and iOS, Because there is no control over the hardware , Comprehensive measures can't compare with ARKit, But there is no significant difference for consumers .
• With a long history ARToolkit： As mentioned earlier , The open source community is not very active .
• commercial sdk Vuforia： Very good performance , Support Android、iOS、UWP, If you can, it will use ARKit or ARCore, If the service conditions are not met , It will use its own AR Engine capability .
• Influential in China AR company ： Shangtang company SenceAR, Too empty AR、 Lao Zi Yun et al .

Most of the SDK All support becoming Unity There is a development plug-in , meanwhile Unity Officials are trying to ease the developer's sdk Learning cost , Will include ARkit、ARCore Multiple in sdk Again encapsulated into ARFoundation SDK, Of course, this SDK It's only suitable for Unity Develop in the editor .

1.3 Summary and Prospect of the current situation

at present AR Technology still has a lot of pain points , Mainly in the following aspects ：

1. The calculation of detection environment is very large , Therefore, the real-time performance is not good , Once many devices move quickly , The fusion of virtual object and environment will produce drift error .
2. Electricity consumption and heating , The frequent use of computing and cameras makes AR The endurance of the equipment is low , Most of them are lower than 5 Hours .
3. Wearable ar Display technology is immature , Most devices are in clarity 、 View 、 There are some deficiencies in experience , Even a certain amount of hair “ Motion sickness ”.
4. The volume is not miniaturized , Even the rendering of Apple glasses , It's also much heavier than ordinary glasses .
5. Software ecology is not mature .

For the above reasons ,AR Compared with the overall user scale of the ecosystem VR smaller ,AR Mainly used in glasses B In the field of end-to-end enterprise customization , For example, medical treatment 、 military 、 Remote assistance, etc . Mobile AR Content applications mainly focus on tools 、 There are two categories of games , And most of them are only suitable for “ Slow moving speed ”、“ The map environment is narrow ( indoor )” Application scenarios of .

5G The development of technology is regarded as AR A major boost to technology , Based on the concept of Cloud Computing ,5G The high transmission speed can make AR The equipment architecture has changed ： That is, abandon the computing structure , Cloud resources are responsible for computing tasks ,AR The device is only responsible for ingestion and display . thus ,AR The device will be able to overcome computing performance 、 Power consumption 、 fever 、 Disadvantages such as large volume .

Two 、AR Introduction to key technologies in the field （ Learning point principle ）

2.1 Technology to explore the environment （ Self orientation 、 testing 3D object 、 Environment modeling ）

AR The first unique technology of technology is how the equipment goes “ understand ” The external environment , Most of the AR Technology is to record the external environment as “ Lattice cloud ”, For example, a 3D object , The device will recognize its edge 、 Size 、 And some key feature points . These points plus depth information ( Or distance information ) It becomes a three-dimensional coordinate , Multiple points are formed “ Lattice cloud ”. Generally speaking, the process of establishing the modeling of the whole environment map is as follows ：

1. Arrive at a new site , Turn on camera .
2. The equipment determines its own height according to sensors such as gyroscope 、 toward , And take this as the origin of the spatial coordinate system .
3. Identify user camera input , Conduct “ Lattice cloud ” modeling .
4. Users move around , Until the whole map is scanned and modeled .

The above process has a professional term , namely SLAM（Simultaneous Localization And Mapping, Time positioning and map building ）, It has been widely used in the field of robot , For example, a sweeping robot 、 Food delivery robot, etc .

SLAM There is one of the key technologies , That is, how to obtain the depth information of a point ( Distance information ). There are several schemes in the industry ：

• Binocular matching ( Two cameras )

This is a mode of learning depth information from human eyes , Because people have two eyes , Compare the images seen through two eyes , You can calculate the distance between the object and the human eye . So binocular matching is taking pictures with two cameras , Use the difference between the abscissa of the object in two pictures to calculate the distance , The following figure is a conceptual diagram , The truth is not that simple .

• Monocular ranging

People can also get the distance information of objects by closing one eye , How is this done ？ Mainly through the accumulation of life experience, and then assist some eye muscle memory . This “ Experience ” Corresponding to the program is the trained algorithm model . Apple has a deep accumulation in monocular ranging , Many old machines have only one camera , Apple can also recognize the distance of objects through image algorithms . This scheme is the most cost-effective one , Seamless adaptation to most existing equipment .

• Depth camera

Using lidar or structured light (RGB-D) The reflection on the object surface obtains the depth point cloud ; Irradiate an object with infrared rays , measurement ToF Time and so on . Most sweeping robots use depth cameras to detect the terrain .iPhone12 Pro The series has one more under the camera LiDar Laser radar , It uses lasers to determine distance and depth

SLAM Technology is widely used in all walks of life , It's not limited to AR field , Like autonomous driving 、3D modeling 、 robot , Drones are SLAM Application fields of .

2.2 display technique （HMD\ Light field imaging \ Holographic projection ）

AR Display technology is another key technology , It is mainly divided into three categories ：

• Visual difference display technology

The principle and 3D Like a movie , Use two eyes to see different images to form poor vision . Use two different displays or ray projectors , An image that enables the left and right eyes to see the same object from different perspectives , The human brain combines the two images to get “ Three dimensional sense ”.

• Holographic projection technology

Holographic projection technology can be achieved by projecting light onto air or a special medium （ Like glass 、 Holographic film ） Present on 3D image . People can see the image from any angle , Get exactly the same visual effect as in the real world . At present, many stages use holographic projection technology , For example, the Chinese Spring Festival Gala 、 Chuyin future concert, etc .

• Light field imaging technology

Do not take advantage of poor vision , The device projects a... Directly into the human eye 3D Object light field , Simulate the most realistic reflection of light in nature . This concept comes from Magic Leap company 2015 Promotional video for , Of course, it was later confirmed that the video is post synthesis .Magic Leap With this technical concept, it has won the favor of countless investment institutions , Market valuation up to 45 Billion dollars , Investment institutions include Google 、 Alibaba 、 Qualcomm and other well-known enterprises and institutions . By the end of 2021 year 12 month ,Magic Still haven't come up with any light field imaging technology equipment , Therefore, it has been widely questioned and criticized by the industry .

Display technology is about AR The most direct experience , Currently on the market AR The equipment didn't do “ Natural grade ” Experience . Long term use will cause discomfort , If the light field imaging technology can mature , The man observed AR The convergence distance and focus distance of the object will be the same , To relieve effectively “ Motion sickness ” problem .

2.3 Interaction technology

So far, , In addition to the traditional touch , Key 、 Voice technology ,AR Roughly 3 Virtual interaction technology in two directions ：

• Motion capture ： Mainly hand motion capture , at present Hololens2 In this area, the technology accumulation is the strongest , No peripherals are required , You can control the device smoothly through gestures .

• Eye tracking ： Use the camera to capture images of human eyes or faces , Then the algorithm is used to detect face and human eyes 、 Location and tracking , So as to estimate the user's line of sight change . At present, there are many mature eye movement instruments on the market , It is more convenient to play with eye tracker 3D game , For example, you can move the lens by turning your line of sight , Mouse movement is no longer required . Eye tracking has many application scenarios , Like a lie detector 、 Concentration test, etc . Many distance education companies are going to use eye tracker to judge whether students are listening to the teacher attentively .

• Brain-machine interface ： Manipulate devices by recognizing electrical signals of brain activity , Musk's Neuralink By implanting nerve wires one tenth the thickness of hair into the brain , To detect and transmit brain signals . This scheme is more sensitive and accurate than the non implantation scheme . The picture below is Neuralink The company shows a video of monkeys playing video games through brain computer interface , The portal is here .

These interaction technologies look very exciting , Not only can it be applied to AR,VR The same applies to . The author thinks , We are on the eve of a great era , The next decade will produce far more drastic changes than the popularity of mobile Internet （2021 year 10 month 28 Japan Facebook Officially renamed Meta）.

3、 ... and 、 Introduction to developer technology

3.1 Technology selection

At present, head mounted devices are not popular , Each manufacturer plays its own game , Microsoft Hololens2 It is the strongest headwear at present AR equipment , Its development method is Unity+MRTK plug-in unit , at present Hololens It is mainly used in enterprise level scenarios , If you want to start a business on this platform , It is suggested to dig deep into an application scenario , For example, military industry 、 Remote assistance, etc . Other AR Headsets can be abandoned for the time being , Sit and wait for the big players to enter ( Apple 、 Samsung, etc ).

Mobile AR Application developers suggest holding on to each other Google And Apple's thighs , according to 1.2 In this section , You can study separately ARCore and ARKit Two SDK. these two items. SDK In theory, it will be used in the future AR Used by head mounted equipment , So admission at this time is not in vain .

Cross platform developer suggestions Unity platform , Use Unity Officially sealed ARFoundation SDK, Shield platform differences .

3.2 AR SDK Universal functions

The following table comes from Unity Official website ( Make a brief text translation and comb ), A good summary of most AR SDK Function with , Developers can get a first glimpse of AR How it works .

function	describe
Device tracking	Track the position and direction of the current device in physical space .
Ray projection	It is usually used to determine the display position of virtual content , A virtual line is projected , To determine the relative position of the virtual object and the real object .
Plane detection	Detect horizontal and vertical surfaces （ For example, coffee table 、 The wall ） The size and location of . These surfaces are called “ Plane ”.
Reference point ( Characteristic point )	Track the position of the plane and feature points over time .
Point cloud detection	SLAM An important mechanism in , See above 2.1 section
gesture	According to people's hands , Recognize gestures as input events .
face recognition	Face detection , For example, start when a face appears AR The effect is attached to the face .
2D Image tracking	Detect specific in the environment 2D Images . For example, when a specific QR code is recognized, it starts AR effect
3D Object tracking	The same goes for
Environmental detection	Detect illumination and color information in specific areas of the environment , This helps to make AR The content is better integrated with the environment , For example, produce shadows .
grid	Generate triangular meshes corresponding to physical space , Therefore, it is more able to interact and interact with the representation of the physical environment / Or visually cover the details .
2D and 3D Personal tracking	There is no need to say more , Experience on your own .
Distance and occlusion	Applies the distance from the object in the physical world to the rendered object 3D Content , So as to realize the realistic mixing of physical objects and virtual objects .
Multi device tracking	In the same AR conversation ( game ) Share and show others in AR Location of equipment .

3.2 ARFoundation SDK Taste fresh （Mac+iOS）

The author is Android developer , But there is no support on hand AR The equipment , So choose iOS Come and have a taste AR, Download the necessary tools first ：

• Unity： Need to register a Unity ID, similar Apple ID The role of , If you just learn to use , Can download Unity Personal version （ It is recommended to use UnityHub To download and manage Unity）.
• Xcode：Unity The compilation process will become a Xcode engineering , Then, the user will Xcode Configure parameters in to compile the final app.
• Visual Studio Code And its supporting plug-ins ：Unity No code editing ability , Clicking the script will jump to the editor set by the user by default to write code .

Start creating a new project ：

1. adopt UnityHub To build a new project , There is one by default AR Engineering formwork .

2. Unity After creating a new one, you can see that a new one has been created in the upper left corner by default Scene, It contains AR Session Origin and AR Session.

3. After the completion of the new project , stay Unity File->Build Setting Switch the target compilation platform to iOS.

4. Continue to click on the above figure Player Settings.., Choose to install at startup AR Kit .

5. By reading demo Code (C#), You can find this demo Its function is to detect the plane , And the user can click on a plane to place an object .

6. here demo In fact, the function is not perfect , Click to place the object without creating and setting , stay Unity Create a Cube The precast body , And drag to AnchorCreator.cs On the member variable of the script （ Don't panic , This is Unity The mechanism of ）.

7. Finally, you can start compiling and running , Click on Unity in File->Build AndRun, It will generate a... By default Xcode engineering , And automatically open Xcode, stay Xcode Login inside AppleId For real machine debugging , And configure signature and other information .

8. The operation effect is as follows , First, the carpet is detected to be a flat surface , It's covered with white dots . Click the carpet to create a Cube On the ground , Click again to create another .

Limited to space and energy ( lazy ？),demo Relevant contents will not be further expanded . Readers can try the following by themselves ：

• Put this shit yellow Cube Replace with a more beautiful AR object , Like a little dinosaur 、 A tank 、 A milk dog .
• by AR Add some simple interactive events to the object , For example, follow your device to move randomly （ Walk the dog ？）.
• Let's have some game interaction , For example, click on the tank , It fires bullets .

If readers have any needs, they can leave a message in the comment area , According to the demand, the author will publish another article to introduce in detail how to achieve more AR effect .

summary

at present AR The technology is already quite mature , The author is looking forward to 2022 Apple and Samsung will go public in AR glasses , At that time, everything may be officially opened AR Era . Mobile Internet tuyere you didn't catch , It's better to AR perhaps VR Take a share in the tuyere . Some people will worry about whether they will do useless work in entering at this time ( Cannon fodder )？ I don't think so , developer 、 Interaction designers can start on the mobile side AR/VR Accumulate experience in .AR/VR The way and technology of interaction are significantly different , These accumulated experience and technology can be copied to AR On the glasses , Glasses can be regarded as a mobile phone worn on the face , isn't it? ？

Now? , Whether it's Apple's technology or Google All technologies have entered a stable stage , and AR Glasses will continue to use existing SDK technology , So it's a good time for developers to enter now .

So from the perspective of ordinary developers , What technologies should we stock up on ？

• Unity Technology stack (C#)： Whether it's game development or not ,Unity It's a must , Perfect for building 3D Content .
• ARKit SDK Technology stack ( Apple ecology is not much to say )： The biggest technology stack in the consumer market , If you want to make money, you can't leave it .
• Hololens Technology stack (C#)： Optional , The surface is the strongest AR Wearable devices , You can't ignore the possibility of its sudden outbreak , And the track is new （ A chance for the back wave to beat the front wave ？）
• ARCore SDK(JAVA+kOTLIN)： Optional , Learn from you Android For the same reason , Occupy more equipment , Low threshold , It has a wide range of applications , The technology stack with the most employment potential , Maybe you will develop robots in the future AR The program may , It is not ？

At present, the author prefers Unity+ARKit ecology ( To be straitened for money ), Therefore, we will continue to plough deeply in this regard , If necessary 、 If you have any questions, please leave a message in the comment area .

Reference documents

web And virtual reality webAR：kstack.corp.kuaishou.com/article/433…

Kwai Fu AR：mp.weixin.qq.com/s/0LcJwlJor…

AR and MR The difference between ：www.zhihu.com/question/39…

Apple AR The history of ：zhuanlan.zhihu.com/p/420474476

AR Introduction to development ：zhuanlan.zhihu.com/p/87017830

ARToolkit brief introduction ：www.cnblogs.com/polobymulbe…

Monocular depth domain research ：www.zhihu.com/question/38…

Monocular vision ranging previous life and present life ：zhuanlan.zhihu.com/p/56263560

Magic Leap Exploring the secrets of light field technology ：www.zhihu.com/question/42…

Eye tracking technology ：zhuanlan.zhihu.com/p/101479231

Unity Chinese community class ：learn.u3d.cn/

Unity Official documents ：docs.unity3d.com/cn/current/…

Unity B Stand by the official ：space.bilibili.com/386224375/c…

hi, I am the Kwai Tai electric supplier.

Kwai Kong Electric wireless technology team is recruiting talents. ! We are the core business line of the company , There are many experts here , It is also full of opportunities and challenges . With the rapid development of business , The team is also expanding rapidly . Welcome to join us , Create world-class e-commerce products together ~

Hot jobs : Android/iOS Advanced development , Android/iOS Experts , Java Architects , The product manager ( E-commerce background ), Test Development ... A lot of HC I'm waiting for you ~

For internal recommendation, please send your resume to >>> Our mailbox : [email protected] <<<, Note: my flower name has a higher success rate ~