Image Signal Processor (ISP) has traditionally been an important chip pillar in the multimedia market. From digital cameras to smartphones, ISPs play an important role in everything that needs to capture images. With the further increase in the intelligence of social life, ISPs are entering many new market applications, and these new intelligent scenarios are putting new demands on ISPs.

The first is the field of autonomous driving/assisted driving. Machine vision is the pillar technology of autonomous driving/assisted driving, without machine vision, it is difficult for these new intelligent driving technologies to succeed. In the field of autonomous/assisted driving, ISPs play an important role by taking the raw image (RAW image) and processing it accordingly to make it better able to be processed by machine vision models. In this process, to ensure the reliability of autonomous/assisted driving technology, it is necessary to ensure that the machine vision model works correctly in low light conditions, for example, which requires the ISP to be able to do night vision and noise reduction on the image. Traditionally, the impact of noise can be reduced by increasing exposure time, but for latency-sensitive scenarios such as autonomous/assisted driving, increasing exposure time is not a useful option, which requires ISPs to be able to use next-generation noise reduction and night vision technologies to meet the corresponding needs.

Outside of autonomous driving/assisted driving, security is also a new intelligent scenario with new demands on ISP technology. Security scenarios also need to deal with various low-light and other scenarios, and to ensure that the image quality is not compromised, ISPs are also needed to be able to handle these scenarios well, thus ensuring that security surveillance can truly provide protection at all times and in all scenarios.

Finally, in multimedia scenes (such as cell phones, etc.), shooting is also becoming increasingly intelligent, and the user's demand for quality imaging is gradually increasing, in addition to the aforementioned similar night photography, there are various intelligent needs, including automatic scene recognition (so that the best sensor settings can be automatically called according to the scene), automatic face detection (so as to ensure the accuracy of algorithms such as automatic exposure and focus ), super-resolution (so as to achieve better long-distance shooting), and so on. All of these either require improvements to the original ISP functionality or require ISPs to introduce new features to ensure user experience.

Artificial intelligence can solve the technical challenges of ISPs

In the aforementioned intelligent applications, the fact that the use of a new generation of artificial intelligence neural networks is often the best solution for night vision, scene recognition, face detection, super-resolution, etc.

In terms of night vision noise reduction, the current optimal solution is to use neural networks to accomplish noise reduction. Since noise is a random process, it is difficult to have an analytical formula to complete noise reduction; instead, a neural network can be trained by collecting a large number of low-light/high-light photo pairs of the same scene to ensure that the neural network can fit low-light photos to high-light photos on the training data set, so that when the training data volume is large enough, the neural network can have good generalization ability The network is capable of night vision noise reduction in all scenes. As shown in the figure below, the solution using AI neural network on the right is much better than the traditional ISP night vision noise reduction solution.

In addition to night vision noise reduction, super-resolution is another scenario where AI has significant advantages. Like night vision noise reduction, it is difficult to model the whole problem mathematically, but by learning from a large amount of data, neural networks can achieve very good super-resolution results, which are far better than ordinary digital zooms, thus enabling a good user experience at a low hardware cost (no need for high magnification lenses).

Finally, for scene recognition, face detection and other functions, artificial intelligence is already widely used in such scenes, so it is a natural choice to use the relevant models to ISP-related fields and let ISPs make relevant shooting parameter adjustments based on the model output. It is worth noting that the accuracy of scene recognition and face detection using neural networks will be much higher than using other traditional methods (the probability of error in judgment can even be an order of magnitude lower), so the future use of artificial intelligence is also a natural direction.

" Integration of artificial intelligence and ISP

Back to the chip design point of view, in order to add the element of artificial intelligence in the ISP, it is necessary to make appropriate adjustments to the ISP, to do integration with artificial intelligence.

As we all know, an important feature of this generation of neural network-based AI is the large number of neural network parameters and the large amount of computation, so in order to meet the relevant needs of ISP, there needs to be a corresponding module (AI engine) that can efficiently handle AI computation to meet the needs of ISP. It is also important to note that since ISPs have requirements for both latency and power consumption, the AI engine must also take into account these two aspects.

For this reason, there are two different ISP integration AI solutions. The first solution is to integrate the ISP and the AI engine together, inside the same IP module. The advantage of this is that the ISP and AI engine are more coupled and therefore able to achieve higher latency and power consumption through the co-optimization of the ISP and AI engine. For example, ISPs tend to be a pipeline and process pixels in batches (rather than always waiting until all pixels are passed); on the other hand, convolutional neural network-based AI models can take advantage of similar pipeline characteristics to process pixels in batches. By deeply integrating ISPs and AI engines together, such pipelines can be designed collaboratively to achieve better latency. In addition, both neural networks and ISPs in fact have high demand for DRAM accesses, and it would greatly reduce the pressure on DRAM bandwidth if ISPs and AI engines could be deeply integrated to collaboratively optimize the scheduling of memory accesses, thus ensuring that both do not access memory heavily at the same time. However, the problem with this design is that the AI engine can only be called by the ISP, so if the ISP is not working, the AI engine will not in fact be turned on, which leads to the problem of dark silicon (wasted chip area); in addition, the co-design of the ISP and AI engine often assumes that the AI engine will run a few fixed models, so if you want to update the model, similar If you want to update the model, then a similar design solidified on the hardware is no longer optimal.

In addition to deep integration, another design idea is to separate the ISP and AI engine, but ensure that the ISP can have a smooth data path between the AI engine, while ensuring that the ISP has a high priority to call the AI engine. The advantage of this is that the AI engine will not be wasted, that is, when the ISP does not start can also be called to other applications; in addition, you can flexibly configure the model used in the AI engine, so that the neural network model used in the ISP can use software control. Of course, because the ISP and AI engine are less coupled, it is also more difficult to deeply optimize the ISP and AI engine together, so there will be a certain cost in terms of latency and energy efficiency ratio.

In our opinion, if the target product itself is an ISP chip, then obviously the first integration mode is the most reasonable, because the primary goal of the ISP chip is high performance and energy efficiency, and its AI engine will not be called by other modules in the system. On the other hand, if the target product is an ISP IP, then both integration methods are reasonable. For high-end ISP IP that pursues performance, we think it is more likely to integrate a more powerful AI engine into the ISP to ensure high performance and energy efficiency; for mid-range ISP IP, the future direction may be to integrate a more basic AI engine into the ISP IP to For the mid-range ISP IP, the future direction may be to integrate a more basic AI engine into the ISP IP to ensure that basic related models can run, and also to leave enough interfaces on the ISP to enable the ISP to access other AI engines on the SoC through on-chip interconnects, so that if larger models need to be run, other AI engines can be used to implement them.

AI ISPs are becoming mainstream in the chip industry

AI ISPs have in fact gained the attention of the semiconductor industry, and related products have been gradually introduced.

At the end of last year, Hysis announced the launch of the next generation of the VC ISP chip for placement scenes, whose main highlight is the efficient processing of night vision noise reduction through deep integration with the AI engine. Also last year, Oppo released its self-researched Mariana ISP chip, whose main feature is also to realize real-time night vision noise reduction for 4K images by integrating an AI engine with up to 18TOPS arithmetic power, thus bringing a new user experience to cell phone users.

More and more manufacturers have launched AI ISPs this year. earlier this year, Amba released AISP at CES, which leverages Amba's accumulation in the image and AI fields to achieve an efficient next-generation ISP; last month, VeriSilicon also launched AI-ISP IP, also for night vision noise reduction scenarios; and AcuityWise's AcuityWise AI-ISP was also officially released VeriSilicon's AI-ISP is also officially released, which achieves the best results for the entire AI ISP by taking away several key hardware modules from the ISP and replacing them with AI algorithms.

As mentioned earlier, with the further intelligence of autonomous driving/assisted driving, security, and consumer electronics, the corresponding demand for ISPs is pushing ISPs and AI to do integration, and the new products of the above companies also happen to target these important application scenarios. We believe that as intelligence deepens further, AI will become an increasingly important part of ISPs, and further integration of AI engines will be seen in ISP chips and IP in the future.