current position:Home>Long time no see. How's ar developing?

Long time no see. How's ar developing?

2022-02-03 00:40:09 Quick Kwai electric wireless team

Long time no see ,AR How's it going

Three years and three years , Has the popular virtual technology disappeared ? This article will lead readers to understand the current AR Technology development status , And a brief introduction to AR The key technology , Finally, show me how to run a AR Demo. If you are helpful, please don't hesitate to praise 、 Comment on 、 Leaving a message. .

One 、 Introduction to virtual reality technology

1.1 AR/VR/MR Conceptual differentiation

AR(Argumented Reality): An enhancement of reality , Device recognition of reality ( shape 、 Location 、 action 、 edge ), So as to superimpose virtual information on reality .

playground AR game ( The picture is from the Apple App Store )

VR(Virtual Reality): Immersive virtual reality , Provide users with a complete virtual world , You don't have to think about the real world .

world of diving game ( Picture from the Internet )

MR(Mixed Reality): Hybrid virtual technology , Combine real and virtual environments .MR Will enhance vision ( Modify real objects ), For example, in medicine MR Technology to see through the human body , So that doctors can see the blood vessels in the body visually , More accurate completion of surgery . A lot of people think MR Just a new concept hype , In essence AR.

Look at the blood vessels on the body ( Picture from the Internet )

The article was written as 2021 year 12 month , here VR Technology and equipment have taken shape , Such as a variety of VR game , And with consumer maturity OculusQuest 2 equipment , Based on the above concept of ecological meta universe, it has been heated .

comparison VR, To achieve a mature consumer experience ,AR There are still many technical difficulties . This article mainly discusses AR Development context and current situation of ,VR Technology is not in the main discussion .

1.2 AR Historical development and current situation

  • AR The beginning of history

1992 Boeing researchers proposed AR This phrase of , A few years later AR The concept of is slowly becoming hot , But it's always in the laboratory .

  • first AR SDK

To 1999 year ,Kato Hirokazu Dominated AR Toolkit project , You can do it 2D Object recognition , And let the virtual object follow the real object .2000 year ,ARToolkit project Open source ,2005 year , It supports Symbian system , Become the world's first mobile open source AR-SDK.2008 Year begins , It has supported IPhone and Android. By the end of 2021 year 12 month ,ARToolkit Support most popular platforms in the world , And can be integrated into Unity platform , Cooperate with each other to develop AR game .

  • first AR Body feeling game

2003 Sony in Playstation Go public EyeToy, Detect the player's actions through the external camera , To interact with the game . A sense of body for a time AR Games are popular .

  • tipping AR Conceptual Google Glass

although AR Technology has been developing steadily , But it will be AR The concept is brought to the general public Google Glass.Google Glass Using the principle of optical reflection projection (HUD), A miniature projector projects light onto a reflective screen , Then it is refracted to the human eye through a convex lens , Implement the so-called “ Primary amplification ”, Form a large enough virtual screen in front of people , It can display simple text information and various data

I don't think this picture will infringe

although Google Glasss More of a “ A cell phone on your face ”, quite a lot AR Technology involved in the field , Such as environmental testing , User interaction technology does not involve . But it involves AR display technique , The light information is projected onto the eyeball . Therefore, its emergence has aroused widespread discussion , One is wearable devices , One is AR technology .

  • AR The first generation of black technology in the world ——HoloLens

2015 In, Microsoft released the AR Headgear HoloLens, It can detect the external environment , Integrate the virtual object with the display environment , And it completes the interaction by tracking the user's gestures . This is the first relatively mature... In a real sense AR equipment , It's small , Have independent computing and storage capacity , When wearing, it does not affect the user to view the external environment .2019 year , Microsoft released HoloLens 2 It solves the problem of narrow vision of the first generation equipment , And greatly improve the accuracy of gesture tracking .

Assist in industrial design ( Picture from the Internet )

HoloLens2 Although relatively mature , But the cost is high , As of the completion of this article , Cost as much as 3500 dollar . in addition , Although it is relative to others AR The equipment is small , But still heavy 500g,iPhone 12 heavy 152g, It's hard for ordinary users to accept the weight of wearing three mobile phones on their heads . The good news is , Microsoft revealed HoloLens3 The weight will be less than 90 g .

  • AR Other players in the world

Widely questioned by the public as ”PPT Swindlers Company “ Of Magic Leap 2017 First generation products released in Magic Leap One. Besides , Both apple and Samsung are rumored to be in 2022 Released its AR glasses . Domestic enterprises have cool Xview series 、 Bright vision 、 Ali nail NrealLightAR Launched AR Product equipment .

Apple smart glasses concept map ( Picture from the Internet )

  • AR The world's iconic game

Niantic The company in 2017 Released Pokémon Go game , This one is based on LBS The game is popular all over the world , The first month of the release is more than 2 Billion dollars , list 3 After years, the total water flow will reach 26.5 Billion dollars .Pokémon Go The success of quickly detonated AR Game market . thereafter AR Games continue to emerge , Such as construction 、 Chess, etc .

pockmon go and My Country Game interface ( Picture from the Internet )

  • Mobile AR The popularity of SDK

At present, mobile terminals AR SDK Fragmentation is serious , There are dozens of differences at home and abroad sdk, Here are some influential sdk.

  • Apple ARKit:2017 Released so far , Considered the most powerful , The most commercial potential sdk. coordination SceneKit 、SpriteKit、RealityKit Wait until Apple's framework library can develop a very cool app, The advantage is that almost all iOS The device can be used seamlessly (iOS 11.0+,iPhone 6s+).
  • Google ARCore: Support at the same time Android and iOS, Because there is no control over the hardware , Comprehensive measures can't compare with ARKit, But there is no significant difference for consumers .
  • With a long history ARToolkit: As mentioned earlier , The open source community is not very active .
  • commercial sdk Vuforia: Very good performance , Support Android、iOS、UWP, If you can, it will use ARKit or ARCore, If the service conditions are not met , It will use its own AR Engine capability .
  • Influential in China AR company : Shangtang company SenceAR, Too empty AR、 Lao Zi Yun et al .

Most of the SDK All support becoming Unity There is a development plug-in , meanwhile Unity Officials are trying to ease the developer's sdk Learning cost , Will include ARkit、ARCore Multiple in sdk Again encapsulated into ARFoundation SDK, Of course, this SDK It's only suitable for Unity Develop in the editor .

1.3 Summary and Prospect of the current situation

at present AR Technology still has a lot of pain points , Mainly in the following aspects :

  1. The calculation of detection environment is very large , Therefore, the real-time performance is not good , Once many devices move quickly , The fusion of virtual object and environment will produce drift error .
  2. Electricity consumption and heating , The frequent use of computing and cameras makes AR The endurance of the equipment is low , Most of them are lower than 5 Hours .
  3. Wearable ar Display technology is immature , Most devices are in clarity 、 View 、 There are some deficiencies in experience , Even a certain amount of hair “ Motion sickness ”.
  4. The volume is not miniaturized , Even the rendering of Apple glasses , It's also much heavier than ordinary glasses .
  5. Software ecology is not mature .

For the above reasons ,AR Compared with the overall user scale of the ecosystem VR smaller ,AR Mainly used in glasses B In the field of end-to-end enterprise customization , For example, medical treatment 、 military 、 Remote assistance, etc . Mobile AR Content applications mainly focus on tools 、 There are two categories of games , And most of them are only suitable for “ Slow moving speed ”、“ The map environment is narrow ( indoor )” Application scenarios of .

5G The development of technology is regarded as AR A major boost to technology , Based on the concept of Cloud Computing ,5G The high transmission speed can make AR The equipment architecture has changed : That is, abandon the computing structure , Cloud resources are responsible for computing tasks ,AR The device is only responsible for ingestion and display . thus ,AR The device will be able to overcome computing performance 、 Power consumption 、 fever 、 Disadvantages such as large volume .

Two 、AR Introduction to key technologies in the field ( Learning point principle )

2.1 Technology to explore the environment ( Self orientation 、 testing 3D object 、 Environment modeling )

AR The first unique technology of technology is how the equipment goes “ understand ” The external environment , Most of the AR Technology is to record the external environment as “ Lattice cloud ”, For example, a 3D object , The device will recognize its edge 、 Size 、 And some key feature points . These points plus depth information ( Or distance information ) It becomes a three-dimensional coordinate , Multiple points are formed “ Lattice cloud ”. Generally speaking, the process of establishing the modeling of the whole environment map is as follows :

  1. Arrive at a new site , Turn on camera .
  2. The equipment determines its own height according to sensors such as gyroscope 、 toward , And take this as the origin of the spatial coordinate system .
  3. Identify user camera input , Conduct “ Lattice cloud ” modeling .
  4. Users move around , Until the whole map is scanned and modeled .

The above process has a professional term , namely SLAM(Simultaneous Localization And Mapping, Time positioning and map building ), It has been widely used in the field of robot , For example, a sweeping robot 、 Food delivery robot, etc .

SLAM There is one of the key technologies , That is, how to obtain the depth information of a point ( Distance information ). There are several schemes in the industry :

  • Binocular matching ( Two cameras )

This is a mode of learning depth information from human eyes , Because people have two eyes , Compare the images seen through two eyes , You can calculate the distance between the object and the human eye . So binocular matching is taking pictures with two cameras , Use the difference between the abscissa of the object in two pictures to calculate the distance , The following figure is a conceptual diagram , The truth is not that simple .

Conceptual diagram of binocular matching distance calculation

  • Monocular ranging

People can also get the distance information of objects by closing one eye , How is this done ? Mainly through the accumulation of life experience, and then assist some eye muscle memory . This “ Experience ” Corresponding to the program is the trained algorithm model . Apple has a deep accumulation in monocular ranging , Many old machines have only one camera , Apple can also recognize the distance of objects through image algorithms . This scheme is the most cost-effective one , Seamless adaptation to most existing equipment .

  • Depth camera

Using lidar or structured light (RGB-D) The reflection on the object surface obtains the depth point cloud ; Irradiate an object with infrared rays , measurement ToF Time and so on . Most sweeping robots use depth cameras to detect the terrain .iPhone12 Pro The series has one more under the camera LiDar Laser radar , It uses lasers to determine distance and depth

The sweeping robot uses a depth camera to detect the terrain

SLAM Technology is widely used in all walks of life , It's not limited to AR field , Like autonomous driving 、3D modeling 、 robot , Drones are SLAM Application fields of .

2.2 display technique (HMD\ Light field imaging \ Holographic projection )

AR Display technology is another key technology , It is mainly divided into three categories :

  • Visual difference display technology

The principle and 3D Like a movie , Use two eyes to see different images to form poor vision . Use two different displays or ray projectors , An image that enables the left and right eyes to see the same object from different perspectives , The human brain combines the two images to get “ Three dimensional sense ”.

  • Holographic projection technology

Holographic projection technology can be achieved by projecting light onto air or a special medium ( Like glass 、 Holographic film ) Present on 3D image . People can see the image from any angle , Get exactly the same visual effect as in the real world . At present, many stages use holographic projection technology , For example, the Chinese Spring Festival Gala 、 Chuyin future concert, etc .

Chuyin future concert

  • Light field imaging technology

Do not take advantage of poor vision , The device projects a... Directly into the human eye 3D Object light field , Simulate the most realistic reflection of light in nature . This concept comes from Magic Leap company 2015 Promotional video for , Of course, it was later confirmed that the video is post synthesis .Magic Leap With this technical concept, it has won the favor of countless investment institutions , Market valuation up to 45 Billion dollars , Investment institutions include Google 、 Alibaba 、 Qualcomm and other well-known enterprises and institutions . By the end of 2021 year 12 month ,Magic Still haven't come up with any light field imaging technology equipment , Therefore, it has been widely questioned and criticized by the industry .

Magic Leap With naked eyes AR Concept video

Display technology is about AR The most direct experience , Currently on the market AR The equipment didn't do “ Natural grade ” Experience . Long term use will cause discomfort , If the light field imaging technology can mature , The man observed AR The convergence distance and focus distance of the object will be the same , To relieve effectively “ Motion sickness ” problem .

2.3 Interaction technology

So far, , In addition to the traditional touch , Key 、 Voice technology ,AR Roughly 3 Virtual interaction technology in two directions :

  • Motion capture : Mainly hand motion capture , at present Hololens2 In this area, the technology accumulation is the strongest , No peripherals are required , You can control the device smoothly through gestures .

Hololens 2 Gesture tracking gesture

  • Eye tracking : Use the camera to capture images of human eyes or faces , Then the algorithm is used to detect face and human eyes 、 Location and tracking , So as to estimate the user's line of sight change . At present, there are many mature eye movement instruments on the market , It is more convenient to play with eye tracker 3D game , For example, you can move the lens by turning your line of sight , Mouse movement is no longer required . Eye tracking has many application scenarios , Like a lie detector 、 Concentration test, etc . Many distance education companies are going to use eye tracker to judge whether students are listening to the teacher attentively .

Eye tracking - Locate a person's pupil

  • Brain-machine interface : Manipulate devices by recognizing electrical signals of brain activity , Musk's Neuralink By implanting nerve wires one tenth the thickness of hair into the brain , To detect and transmit brain signals . This scheme is more sensitive and accurate than the non implantation scheme . The picture below is Neuralink The company shows a video of monkeys playing video games through brain computer interface , The portal is here .

Youtube video - Monkeys play games through brain computer interface

These interaction technologies look very exciting , Not only can it be applied to AR,VR The same applies to . The author thinks , We are on the eve of a great era , The next decade will produce far more drastic changes than the popularity of mobile Internet (2021 year 10 month 28 Japan Facebook Officially renamed Meta).

3、 ... and 、 Introduction to developer technology

3.1 Technology selection

At present, head mounted devices are not popular , Each manufacturer plays its own game , Microsoft Hololens2 It is the strongest headwear at present AR equipment , Its development method is Unity+MRTK plug-in unit , at present Hololens It is mainly used in enterprise level scenarios , If you want to start a business on this platform , It is suggested to dig deep into an application scenario , For example, military industry 、 Remote assistance, etc . Other AR Headsets can be abandoned for the time being , Sit and wait for the big players to enter ( Apple 、 Samsung, etc ).

Mobile AR Application developers suggest holding on to each other Google And Apple's thighs , according to 1.2 In this section , You can study separately ARCore and ARKit Two SDK. these two items. SDK In theory, it will be used in the future AR Used by head mounted equipment , So admission at this time is not in vain .

Cross platform developer suggestions Unity platform , Use Unity Officially sealed ARFoundation SDK, Shield platform differences .

3.2 AR SDK Universal functions

The following table comes from Unity Official website ( Make a brief text translation and comb ), A good summary of most AR SDK Function with , Developers can get a first glimpse of AR How it works .

function describe
Device tracking Track the position and direction of the current device in physical space .
Ray projection It is usually used to determine the display position of virtual content , A virtual line is projected , To determine the relative position of the virtual object and the real object .
Plane detection Detect horizontal and vertical surfaces ( For example, coffee table 、 The wall ) The size and location of . These surfaces are called “ Plane ”.
Reference point ( Characteristic point ) Track the position of the plane and feature points over time .
Point cloud detection SLAM An important mechanism in , See above 2.1 section
gesture According to people's hands , Recognize gestures as input events .
face recognition Face detection , For example, start when a face appears AR The effect is attached to the face .
2D Image tracking Detect specific in the environment 2D Images . For example, when a specific QR code is recognized, it starts AR effect
3D Object tracking The same goes for
Environmental detection Detect illumination and color information in specific areas of the environment , This helps to make AR The content is better integrated with the environment , For example, produce shadows .
grid Generate triangular meshes corresponding to physical space , Therefore, it is more able to interact and interact with the representation of the physical environment / Or visually cover the details .
2D and 3D Personal tracking There is no need to say more , Experience on your own .
Distance and occlusion Applies the distance from the object in the physical world to the rendered object 3D Content , So as to realize the realistic mixing of physical objects and virtual objects .
Multi device tracking In the same AR conversation ( game ) Share and show others in AR Location of equipment .

3.2 ARFoundation SDK Taste fresh (Mac+iOS)

The author is Android developer , But there is no support on hand AR The equipment , So choose iOS Come and have a taste AR, Download the necessary tools first :

  • Unity: Need to register a Unity ID, similar Apple ID The role of , If you just learn to use , Can download Unity Personal version ( It is recommended to use UnityHub To download and manage Unity).
  • Xcode:Unity The compilation process will become a Xcode engineering , Then, the user will Xcode Configure parameters in to compile the final app.
  • Visual Studio Code And its supporting plug-ins :Unity No code editing ability , Clicking the script will jump to the editor set by the user by default to write code .

Start creating a new project :

  1. adopt UnityHub To build a new project , There is one by default AR Engineering formwork .

  1. Unity After creating a new one, you can see that a new one has been created in the upper left corner by default Scene, It contains AR Session Origin and AR Session.

  1. After the completion of the new project , stay Unity File->Build Setting Switch the target compilation platform to iOS.
  1. Continue to click on the above figure Player Settings.., Choose to install at startup AR Kit .

  1. By reading demo Code (C#), You can find this demo Its function is to detect the plane , And the user can click on a plane to place an object .

  1. here demo In fact, the function is not perfect , Click to place the object without creating and setting , stay Unity Create a Cube The precast body , And drag to AnchorCreator.cs On the member variable of the script ( Don't panic , This is Unity The mechanism of ).

  1. Finally, you can start compiling and running , Click on Unity in File->Build AndRun, It will generate a... By default Xcode engineering , And automatically open Xcode, stay Xcode Login inside AppleId For real machine debugging , And configure signature and other information .

  1. The operation effect is as follows , First, the carpet is detected to be a flat surface , It's covered with white dots . Click the carpet to create a Cube On the ground , Click again to create another .

Limited to space and energy ( lazy ?),demo Relevant contents will not be further expanded . Readers can try the following by themselves :

  • Put this shit yellow Cube Replace with a more beautiful AR object , Like a little dinosaur 、 A tank 、 A milk dog .
  • by AR Add some simple interactive events to the object , For example, follow your device to move randomly ( Walk the dog ?).
  • Let's have some game interaction , For example, click on the tank , It fires bullets .

If readers have any needs, they can leave a message in the comment area , According to the demand, the author will publish another article to introduce in detail how to achieve more AR effect .


at present AR The technology is already quite mature , The author is looking forward to 2022 Apple and Samsung will go public in AR glasses , At that time, everything may be officially opened AR Era . Mobile Internet tuyere you didn't catch , It's better to AR perhaps VR Take a share in the tuyere . Some people will worry about whether they will do useless work in entering at this time ( Cannon fodder )? I don't think so , developer 、 Interaction designers can start on the mobile side AR/VR Accumulate experience in .AR/VR The way and technology of interaction are significantly different , These accumulated experience and technology can be copied to AR On the glasses , Glasses can be regarded as a mobile phone worn on the face , isn't it? ?

Now? , Whether it's Apple's technology or Google All technologies have entered a stable stage , and AR Glasses will continue to use existing SDK technology , So it's a good time for developers to enter now .

So from the perspective of ordinary developers , What technologies should we stock up on ?

  • Unity Technology stack (C#): Whether it's game development or not ,Unity It's a must , Perfect for building 3D Content .
  • ARKit SDK Technology stack ( Apple ecology is not much to say ): The biggest technology stack in the consumer market , If you want to make money, you can't leave it .
  • Hololens Technology stack (C#): Optional , The surface is the strongest AR Wearable devices , You can't ignore the possibility of its sudden outbreak , And the track is new ( A chance for the back wave to beat the front wave ?)
  • ARCore SDK(JAVA+kOTLIN): Optional , Learn from you Android For the same reason , Occupy more equipment , Low threshold , It has a wide range of applications , The technology stack with the most employment potential , Maybe you will develop robots in the future AR The program may , It is not ?

At present, the author prefers Unity+ARKit ecology ( To be straitened for money ), Therefore, we will continue to plough deeply in this regard , If necessary 、 If you have any questions, please leave a message in the comment area .

Reference documents

web And virtual reality…

Kwai Fu…

AR and MR The difference between…

Apple AR The history of

AR Introduction to development

ARToolkit brief introduction…

Monocular depth domain research…

Monocular vision ranging previous life and present life

Magic Leap Exploring the secrets of light field technology…

Eye tracking technology

Unity Chinese community class

Unity Official documents…

Unity B Stand by the official…

hi, I am the Kwai Tai electric supplier.

Kwai Kong Electric wireless technology team is recruiting talents. ! We are the core business line of the company , There are many experts here , It is also full of opportunities and challenges . With the rapid development of business , The team is also expanding rapidly . Welcome to join us , Create world-class e-commerce products together ~

Hot jobs : Android/iOS Advanced development , Android/iOS Experts , Java Architects , The product manager ( E-commerce background ), Test Development ... A lot of HC I'm waiting for you ~

For internal recommendation, please send your resume to >>> Our mailbox : [email protected] <<<, Note: my flower name has a higher success rate ~

copyright notice
author[Quick Kwai electric wireless team],Please bring the original link to reprint, thank you.

Random recommended