computer vision computer vision is an interdisciplinary field that deals with how computers can be made to gain highlevel understanding from digital images or videos from the perspective of engineering it seeks to automate tasks that the human visual system can do computer vision tasks include methods for acquiring processing analyzing and understanding digital images and extraction of highdimensional data from the real world in order to produce numerical or symbolic information eg in the forms of decisions as a scientific discipline computer vision is concerned with the theory behind artificial systems that extract information from images the image data can take many forms such as video sequences views from multiple cameras or multidimensional data from a medical scanner as a technological discipline computer vision seeks to apply its theories and models for the construction of computer vision systems subdomains of computer vision include scene reconstruction event detection video tracking object recognition 3d pose estimation learning indexing motion estimation and image restoration definition computer vision is an interdisciplinary field that deals with how computers can be made to gain highlevel understanding from digital images or videos from the perspective of engineering it seeks to automate tasks that the human visual system can do as a technological discipline computer vision seeks to apply its theories and models for the construction of computer vision systems history in the late 1960s computer vision began at universities that were pioneering artificial intelligence it was meant to mimic the human visual system as a stepping stone to endowing robots with intelligent behavior what distinguished computer vision from the prevalent field of digital image processing at that time was a desire to extract threedimensional structure from images with the goal of achieving full scene understanding studies in the 1970s formed the early foundations for many of the computer vision algorithms that exist today including extraction of edges from images labeling of lines nonpolyhedral and polyhedral modeling representation of objects as interconnections of smaller structures optical flow and motion estimation the next decade saw studies based on more rigorous mathematical analysis and quantitative aspects of computer vision these include the concept of scalespace the inference of shape from various cues such as shading texture and focus and contour models known as snakes researchers also realized that many of these mathematical concepts could be treated within the same optimization framework as regularization and markov random fields by the 1990s some of the previous research topics became more active than the others research in projective 3d reconstructions led to better understanding of camera calibration with the advent of optimization methods for camera calibration it was realized that a lot of the ideas were already explored in bundle adjustment theory from the field of photogrammetry this led to methods for sparse 3d reconstructions of scenes from multiple images progress was made on the dense stereo correspondence problem and further multiview stereo techniques at the same time variations of graph cut were used to solve image segmentation this decade also marked the first time statistical learning techniques were used in practice to recognize faces in images see eigenface toward the end of the 1990s a significant change came about with the increased interaction between the fields of computer graphics and computer vision this included imagebased rendering image morphing view interpolation panoramic image stitching and early lightfield rendering recent work has seen the resurgence of featurebased methods used in conjunction with machine learning techniques and complex optimization frameworks related fields artificial intelligence areas of artificial intelligence deal with autonomous planning or deliberation for robotical systems to navigate through an environment a detailed understanding of these environments is required to navigate through them information about the environment could be provided by a computer vision system acting as a vision sensor and providing highlevel information about the environment and the robot artificial intelligence and computer vision share other topics such as pattern recognition and learning techniques consequently computer vision is sometimes seen as a part of the artificial intelligence field or the computer science field in general information engineering computer vision is often considered to be part of information engineering solidstate physics solidstate physics is another field that is closely related to computer vision most computer vision systems rely on image sensors which detect electromagnetic radiation which is typically in the form of either visible or infrared light the sensors are designed using quantum physics the process by which light interacts with surfaces is explained using physics physics explains the behavior of optics which are a core part of most imaging systems sophisticated image sensors even require quantum mechanics to provide a complete understanding of the image formation process also various measurement problems in physics can be addressed using computer vision for example motion in fluids neurobiology a third field which plays an important role is neurobiology specifically the study of the biological vision system over the last century there has been an extensive study of eyes neurons and the brain structures devoted to processing of visual stimuli in both humans and various animals this has led to a coarse yet complicated description of how real vision systems operate in order to solve certain vision related tasks these results have led to a subfield within computer vision where artificial systems are designed to mimic the processing and behavior of biological systems at different levels of complexity also some of the learningbased methods developed within computer vision eg neural net and deep learning based image and feature analysis and classification have their background in biology some strands of computer vision research are closely related to the study of biological vision indeed just as many strands of ai research are closely tied with research into human consciousness and the use of stored knowledge to interpret integrate and utilize visual information the field of biological vision studies and models the physiological processes behind visual perception in humans and other animals computer vision on the other hand studies and describes the processes implemented in software and hardware behind artificial vision systems interdisciplinary exchange between biological and computer vision has proven fruitful for both fields signal processing yet another field related to computer vision is signal processing many methods for processing of onevariable signals typically temporal signals can be extended in a natural way to processing of twovariable signals or multivariable signals in computer vision however because of the specific nature of images there are many methods developed within computer vision which have no counterpart in processing of onevariable signals together with the multidimensionality of the signal this defines a subfield in signal processing as a part of computer vision other fields beside the abovementioned views on computer vision many of the related research topics can also be studied from a purely mathematical point of view for example many methods in computer vision are based on statistics optimization or geometry finally a significant part of the field is devoted to the implementation aspect of computer vision how existing methods can be realized in various combinations of software and hardware or how these methods can be modified in order to gain processing speed without losing too much performance distinctions the fields most closely related to computer vision are image processing image analysis and machine vision there is a significant overlap in the range of techniques and applications that these cover this implies that the basic techniques that are used and developed in these fields are similar something which can be interpreted as there is only one field with different names on the other hand it appears to be necessary for research groups scientific journals conferences and companies to present or market themselves as belonging specifically to one of these fields and hence various characterizations which distinguish each of the fields from the others have been presented computer graphics produces image data from 3d models computer vision often produces 3d models from image data there is also a trend towards a combination of the two disciplines eg as explored in augmented reality the following characterizations appear relevant but should not be taken as universally accepted image processing and image analysis tend to focus on 2d images how to transform one image to another eg by pixelwise operations such as contrast enhancement local operations such as edge extraction or noise removal or geometrical transformations such as rotating the image this characterization implies that image processinganalysis neither require assumptions nor produce interpretations about the image content computer vision includes 3d analysis from 2d images this analyzes the 3d scene projected onto one or several images eg how to reconstruct structure or other information about the 3d scene from one or several images computer vision often relies on more or less complex assumptions about the scene depicted in an image machine vision is the process of applying a range of technologies methods to provide imagingbased automatic inspection process control and robot guidance this implies that image sensor technologies and control theory often are integrated with the processing of image data to control a robot and that realtime processing is emphasised by means of efficient implementations in hardware and software it also implies that the external conditions such as lighting can be and are often more controlled in machine vision than they are in general computer vision which can enable the use of different algorithms there is also a field called imaging which primarily focus on the process of producing images but sometimes also deals with processing and analysis of images for example medical imaging includes substantial work on the analysis of image data in medical applications finally pattern recognition is a field which uses various methods to extract information from signals in general mainly based on statistical approaches and artificial neural networks a significant part of this field is devoted to applying these methods to image data photogrammetry also overlaps with computer vision eg stereophotogrammetry vs computer stereo vision applications applications range from tasks such as industrial machine vision systems which say inspect bottles speeding by on a production line to research into artificial intelligence and computers or robots that can comprehend the world around them the computer vision and machine vision fields have significant overlap computer vision covers the core technology of automated image analysis which is used in many fields machine vision usually refers to a process of combining automated image analysis with other methods and technologies to provide automated inspection and robot guidance in industrial applications in many computer vision applications the computers are preprogrammed to solve a particular task but methods based on learning are now becoming increasingly common examples of applications of computer vision include systems for learning 3d shapes has been a challenging task in computer vision recent advances in deep learning has enabled researchers to build models that are able to generate and reconstruct 3d shapes from single or multiview depth maps or silhouettes seamlessly and efficiently automatic inspection eg in manufacturing applications assisting humans in identification tasks eg a species identification system controlling processes eg an industrial robot detecting events eg for visual surveillance or people counting interaction eg as the input to a device for computerhuman interaction modeling objects or environments eg medical image analysis or topographical modeling navigation eg by an autonomous vehicle or mobile robot and organizing information eg for indexing databases of images and image sequences play media darpas visual media reasoning concept video one of the most prominent application fields is medical computer vision or medical image processing characterized by the extraction of information from image data to diagnose a patient an example of this is detection of tumours arteriosclerosis or other malign changes measurements of organ dimensions blood flow etc are another example it also supports medical research by providing new information eg about the structure of the brain or about the quality of medical treatments applications of computer vision in the medical area also includes enhancement of images interpreted by humansultrasonic images or xray images for exampleto reduce the influence of noise a second application area in computer vision is in industry sometimes called machine vision where information is extracted for the purpose of supporting a manufacturing process one example is quality control where details or final products are being automatically inspected in order to find defects another example is measurement of position and orientation of details to be picked up by a robot arm machine vision is also heavily used in agricultural process to remove undesirable food stuff from bulk material a process called optical sorting military applications are probably one of the largest areas for computer vision the obvious examples are detection of enemy soldiers or vehicles and missile guidance more advanced systems for missile guidance send the missile to an area rather than a specific target and target selection is made when the missile reaches the area based on locally acquired image data modern military concepts such as battlefield awareness imply that various sensors including image sensors provide a rich set of information about a combat scene which can be used to support strategic decisions in this case automatic processing of the data is used to reduce complexity and to fuse information from multiple sensors to increase reliability artists concept of a mars exploration rover an example of an unmanned landbased vehicle notice the stereo cameras mounted on top of the rover one of the newer application areas is autonomous vehicles which include submersibles landbased vehicles small robots with wheels cars or trucks aerial vehicles and unmanned aerial vehicles uav the level of autonomy ranges from fully autonomous unmanned vehicles to vehicles where computer vision based systems support a driver or a pilot in various situations fully autonomous vehicles typically use computer vision for navigation ie for knowing where it is or for producing a map of its environment slam and for detecting obstacles it can also be used for detecting certain task specific events eg a uav looking for forest fires examples of supporting systems are obstacle warning systems in cars and systems for autonomous landing of aircraft several car manufacturers have demonstrated systems for autonomous driving of cars but this technology has still not reached a level where it can be put on the market there are ample examples of military autonomous vehicles ranging from advanced missiles to uavs for recon missions or missile guidance space exploration is already being made with autonomous vehicles using computer vision eg nasas mars exploration rover and esas exomars rover other application areas include support of visual effects creation for cinema and broadcast eg camera tracking matchmoving surveillance tracking and counting organisms in the biological sciences typical tasks each of the application areas described above employ a range of computer vision tasks more or less welldefined measurement problems or processing problems which can be solved using a variety of methods some examples of typical computer vision tasks are presented below computer vision tasks include methods for acquiring processing analyzing and understanding digital images and extraction of highdimensional data from the real world in order to produce numerical or symbolic information eg in the forms of decisions recognition the classical problem in computer vision image processing and machine vision is that of determining whether or not the image data contains some specific object feature or activity different varieties of the recognition problem are described in the literature object recognition also called object classification one or several prespecified or learned objects or object classes can be recognized usually together with their 2d positions in the image or 3d poses in the scene blippar google goggles and likethat provide standalone programs that illustrate this functionality identification an individual instance of an object is recognized examples include identification of a specific persons face or fingerprint identification of handwritten digits or identification of a specific vehicle detection the image data are scanned for a specific condition examples include detection of possible abnormal cells or tissues in medical images or detection of a vehicle in an automatic road toll system detection based on relatively simple and fast computations is sometimes used for finding smaller regions of interesting image data which can be further analyzed by more computationally demanding techniques to produce a correct interpretation currently the best algorithms for such tasks are based on convolutional neural networks an illustration of their capabilities is given by the imagenet large scale visual recognition challenge this is a benchmark in object classification and detection with millions of images and hundreds of object classes performance of convolutional neural networks on the imagenet tests is now close to that of humans the best algorithms still struggle with objects that are small or thin such as a small ant on a stem of a flower or a person holding a quill in their hand they also have trouble with images that have been distorted with filters an increasingly common phenomenon with modern digital cameras by contrast those kinds of images rarely trouble humans humans however tend to have trouble with other issues for example they are not good at classifying objects into finegrained classes such as the particular breed of dog or species of bird whereas convolutional neural networks handle this with ease several specialized tasks based on recognition exist such as contentbased image retrieval finding all images in a larger set of images which have a specific content the content can be specified in different ways for example in terms of similarity relative a target image give me all images similar to image x or in terms of highlevel search criteria given as text input give me all images which contains many houses are taken during winter and have no cars in them computer vision for people counter purposes in public places malls shopping centres pose estimation estimating the position or orientation of a specific object relative to the camera an example application for this technique would be assisting a robot arm in retrieving objects from a conveyor belt in an assembly line situation or picking parts from a bin optical character recognition ocr identifying characters in images of printed or handwritten text usually with a view to encoding the text in a format more amenable to editing or indexing eg ascii 2d code reading reading of 2d codes such as data matrix and qr codes facial recognition shape recognition technology srt in people counter systems differentiating human beings head and shoulder patterns from objects motion analysis several tasks relate to motion estimation where an image sequence is processed to produce an estimate of the velocity either at each points in the image or in the 3d scene or even of the camera that produces the images examples of such tasks are egomotion determining the 3d rigid motion rotation and translation of the camera from an image sequence produced by the camera tracking following the movements of a usually smaller set of interest points or objects eg vehicles humans or other organisms in the image sequence optical flow to determine for each point in the image how that point is moving relative to the image plane ie its apparent motion this motion is a result both of how the corresponding 3d point is moving in the scene and how the camera is moving relative to the scene scene reconstruction given one or typically more images of a scene or a video scene reconstruction aims at computing a 3d model of the scene in the simplest case the model can be a set of 3d points more sophisticated methods produce a complete 3d surface model the advent of 3d imaging not requiring motion or scanning and related processing algorithms is enabling rapid advances in this field gridbased 3d sensing can be used to acquire 3d images from multiple angles algorithms are now available to stitch multiple 3d images together into point clouds and 3d models image restoration the aim of image restoration is the removal of noise sensor noise motion blur etc from images the simplest possible approach for noise removal is various types of filters such as lowpass filters or median filters more sophisticated methods assume a model of how the local image structures look like a model which distinguishes them from the noise by first analysing the image data in terms of the local image structures such as lines or edges and then controlling the filtering based on local information from the analysis step a better level of noise removal is usually obtained compared to the simpler approaches an example in this field is inpainting system methods the organization of a computer vision system is highly application dependent some systems are standalone applications which solve a specific measurement or detection problem while others constitute a subsystem of a larger design which for example also contains subsystems for control of mechanical actuators planning information databases manmachine interfaces etc the specific implementation of a computer vision system also depends on if its functionality is prespecified or if some part of it can be learned or modified during operation many functions are unique to the application there are however typical functions which are found in many computer vision systems image acquisition a digital image is produced by one or several image sensors which besides various types of lightsensitive cameras include range sensors tomography devices radar ultrasonic cameras etc depending on the type of sensor the resulting image data is an ordinary 2d image a 3d volume or an image sequence the pixel values typically correspond to light intensity in one or several spectral bands gray images or colour images but can also be related to various physical measures such as depth absorption or reflectance of sonic or electromagnetic waves or nuclear magnetic resonance preprocessing before a computer vision method can be applied to image data in order to extract some specific piece of information it is usually necessary to process the data in order to assure that it satisfies certain assumptions implied by the method examples are resampling in order to assure that the image coordinate system is correct noise reduction in order to assure that sensor noise does not introduce false information contrast enhancement to assure that relevant information can be detected scale space representation to enhance image structures at locally appropriate scales feature extraction image features at various levels of complexity are extracted from the image data typical examples of such features are lines edges and ridges localized interest points such as corners blobs or points more complex features may be related to texture shape or motion detectionsegmentation at some point in the processing a decision is made about which image points or regions of the image are relevant for further processing examples are selection of a specific set of interest points segmentation of one or multiple image regions which contain a specific object of interest segmentation of image into nested scene architecture comprised foreground object groups single objects or salient object parts also referred to as spatialtaxon scene hierarchy while the visual salience is often implemented as spatial and temporal attention segmentation or cosegmentation of one or multiple videos into a series of perframe foreground masks while maintaining its temporal semantic continuity highlevel processing at this step the input is typically a small set of data for example a set of points or an image region which is assumed to contain a specific object the remaining processing deals with for example verification that the data satisfy modelbased and application specific assumptions estimation of application specific parameters such as object pose or object size image recognition classifying a detected object into different categories image registration comparing and combining two different views of the same object decision making making the final decision required for the application for example passfail on automatic inspection applications match nomatch in recognition applications flag for further human review in medical military security and recognition applications imageunderstanding systems imageunderstanding systems ius include three levels of abstraction as follows low level includes image primitives such as edges texture elements or regions intermediate level includes boundaries surfaces and volumes and high level includes objects scenes or events many of these requirements are really topics for further research the representational requirements in the designing of ius for these levels are representation of prototypical concepts concept organization spatial knowledge temporal knowledge scaling and description by comparison and differentiation while inference refers to the process of deriving new not explicitly represented facts from currently known facts control refers to the process that selects which of the many inference search and matching techniques should be applied at a particular stage of processing inference and control requirements for ius are search and hypothesis activation matching and hypothesis testing generation and use of expectations change and focus of attention certainty and strength of belief inference and goal satisfaction hardware there are many kinds of computer vision systems nevertheless all of them contain these basic elements a power source at least one image acquisition device ie camera ccd etc a processor as well as control and communication cables or some kind of wireless interconnection mechanism in addition a practical vision system contains software as well as a display in order to monitor the system vision systems for inner spaces as most industrial ones contain an illumination system and may be placed in a controlled environment furthermore a completed system includes many accessories like camera supports cables and connectors most computer vision systems use visiblelight cameras passively viewing a scene at frame rates of at most 60 frames per second usually far slower a few computer vision systems use image acquisition hardware with active illumination or something other than visible light or both for example a structuredlight 3d scanner a thermographic camera a hyperspectral imager radar imaging a lidar scanner a magnetic resonance image a sidescan sonar a synthetic aperture sonar or etc such hardware captures images that are then processed often using the same computer vision algorithms used to process visiblelight images while traditional broadcast and consumer video systems operate at a rate of 30 frames per second advances in digital signal processing and consumer graphics hardware has made highspeed image acquisition processing and display possible for realtime systems on the order of hundreds to thousands of frames per second for applications in robotics fast realtime video systems are critically important and often can simplify the processing needed for certain algorithms when combined with a highspeed projector fast image acquisition allows 3d measurement and feature tracking to be realised egocentric vision systems are composed of a wearable camera that automatically take pictures from a firstperson perspective as of 2016 vision processing units are emerging as a new class of processor to complement cpus and graphics processing units gpus in this role computational imaging computational photography machine vision glossary space mapping teknomofernandez algorithm visual system visual perception vision science egocentric vision visual agnosia lists list of computer vision topics list of emerging technologies outline of artificial intelligence david marr 1982 vision w h freeman and company isbn0716712849 azriel rosenfeld avinash kak 1982 digital picture processing academic press isbn0125973012 barghout lauren lawrence w lee 2003 perceptual information processing system us patent application 10618543 isbn0262081598 berthold klaus paul horn 1986 robot vision mit press isbn0262081598 michael cfairhurst 1988 computer vision for robotic systems prentice hall isbn0131669192 olivier faugeras 1993 threedimensional computer vision a geometric viewpoint mit press isbn0262061589 tony lindeberg 1994 scalespace theory in computer vision springer isbn0792394186 james l crowley and henrik i christensen eds 1995 vision as process springerverlag isbn354058143xcs1 maint extra text authors list link gsta h granlund hans knutsson 1995 signal processing for computer vision kluwer academic publisher isbn0792395301 reinhard klette karsten schluens andreas koschan 1998 computer vision threedimensional data from images springer singapore isbn9813083719 emanuele trucco alessandro verri 1998 introductory techniques for 3d computer vision prentice hall isbn0132611082 bernd jhne 2002 digital image processing springer isbn3540677542 richard hartley and andrew zisserman 2003 multiple view geometry in computer vision cambridge university press isbn0521540518 grard medioni sing bing kang 2004 emerging topics in computer vision prentice hall isbn0131013661 r fisher k dawsonhowe a fitzgibbon c robertson e trucco 2005 dictionary of computer vision and image processing john wiley isbn0470015268 nikos paragios and yunmei chen and olivier faugeras 2005 handbook of mathematical models in computer vision springer isbn0387263713 wilhelm burger mark j burge 2007 digital image processing an algorithmic approach using java springer isbn1846283795 pedram azad tilo gockel rdiger dillmann 2008 computer vision principles and practice elektor international media bv isbn0905705718 richard szeliski 2010 computer vision algorithms and applications springerverlag isbn9781848829343 j r parker 2011 algorithms for image processing and computer vision 2nd ed wiley isbn9780470643853 richard j radke 2013 computer vision for visual effects cambridge university press isbn9780521766876 usc iris computer vision conference list computer vision papers on the web a complete list of papers of the most relevant computer vision conferences computer vision online news source code datasets and job offers related to computer vision keith prices annotated computer vision bibliography cvonline bob fishers compendium of computer vision british machine vision association supporting computer vision research within the uk via the bmvc and miua conferences annals of the bmva opensource journal bmva summer school and oneday meetings vtecomputer visioncategories datasets digital geometry commercial systems feature detection geometry image sensor technology learning morphology motion analysis noise reduction techniques recognition and categorization research infrastructure researchers segmentation software technologies computer stereo vision 3d reconstruction motion capture applications autonomous vehicles face recognition image search optical character recognition remote sensing robots main category vtemixed and virtual realityconcepts virtuality virtual cinematography augmented reality augmented virtuality real life projection augmented model realityvirtuality continuum artificial reality simulated reality ubiquitous computing virtual world persistent multimodal interaction telepresence immersion technology haptic suit wearable computer omnidirectional treadmill display headmounted display optical headup display virtual retinal display virtual reality headset stereoscopy computer vision stereo camera camera resectioning omnidirectional camera software compositing imagebased modeling and rendering realtime computer graphics visual hull chroma key hidden surface determination photography free viewpoint television 360degree video vr photography tracking motion capture tracking system types optical inertial devices wired glove gametrak google glass microsoft hololens playstation move leap motion kinect sixense truemotion immersivedevicespersonal merge vr ar daydream google cardboard htc vive oculus rift samsung gear vr playstation vr pimax osvr magic leap rooms allosphere cave treadport history sensorama virtual boy famicom 3d system sword of damocles sega vr virtuality applications pervasive game artoolkit arcore interactive art virtual graffiti simulated reality in fiction 