enables developers to incorporate proprietary
functions, the company also supplies a reference design that includes a camera and micro-electro-mechanical (MEM) sensors (Figure 1).
While individual processors can be incorporated into embedded systems, there are other
options to accelerate these tasks. These include
embedding either embedded “soft-core” processors or IP into the FPGAs used in such systems.
Such products can prove useful for those developing such products as smart cameras, drones
and FPGA-based frame grabbers.
One example of such soft-core IP processors is the VectorBlox MXP Matrix Processor
from CEVA (Mountain View, CA; USA; www.
ceva-dsp.com). As a scalable soft-core processor designed for FPGAs, the IP can be implemented in an FPGA as a plug-in IP block that
implements parallel vector processor algorithms on the 2D and 3D matrices commonly found in image processing. According to CEVA, the soft-core processor core can
enhance the performance of standard MicroBlaze processors from Xilinx (San Jose, CA,
USA; www.xilinx.com) by a minimum of 1-2
orders of magnitude. Taking advantage of this,
Altek (Hsinchu, Taiwan; www.altek.com.tw)
recently licensed CEVA’s IP to perform object
detection and tracking and 3D depth sensing
in the company’s drones and smart cameras.
Just as developers of cameras and frame grabbers can leverage the power of soft-core processors, they can also take advantage of FPGA
IP libraries to perform dedicated image processing tasks. These include camera interfacing, image pre-processing functions such as
Bayer interpolation, image compression, stereo
vision, face detection and motion detection.
For the developer of machine vision sys-
tems that incorporates standard camera inter-
faces such as GigE Vision and CameraLink and
CoaXPress, FPGA vendors and third-party sup-
pliers offer IP to perform these tasks. Xilinx, for
example, has developed its GigEVCore1.2, an
FPGA core that maps the GigE Vision control
and message channels to a soft-core embedded
processor such as the MicroBlaze. Supporting
both Xilinx and FPGAs from Altera (San Jose,
CA, USA; www.altera.com), Sensor to Image
(Schongau, Germany; www.s2i.org) has devel-
oped FPGA IP to allow designers to imple-
ment Camera Link, GigE Vision and CoaX-
Press (CXP) interfaces on FPGAs.
While making camera interfacing easier,
the advent of high-density FPGAs also allows
many image pre-processing functions to be
incorporated. While pre-processing algo-
rithms such as Bayer interpolation, bad
pixel correction and color balancing
can be performed on a CPU, they are
more effectively performed in an
FPGA (see “CCD cameras embed
imaging algorithms,” Vision Sys-
tems Design, May 2007; http://bit.
ly/VSD-0507). In the past, developers of cam-
eras and frame grabbers needed to develop
these algorithms. Today, however, vendors
offer IP libraries to perform these tasks.
CoSynth (Oldenburg, Germany; www.
cosynth.com) offers several IP cores optimized
for use in FPGAs that support image process-
ing functions such as Bayer pattern demosaic-
ing, color-space conversion, image segmenta-
tion and morphological operations.
For use with camera based systems, IP cores
for the integration of cameras via LVDS and
Ethernet are available. Similarly, Logic Bricks
from Xylon (Zagreb, Croatia; www.logicbricks.
com) provide image processing functions
ranging from Bayer decoding (demosiacing),
perspective transformation and lens correction
and MJPEG decoding.
Just as many IP vendors implement basic image
processing functions such as these, other companies (including Xylon) have realized the need
to develop more application specific FPGA IP
to speed the time to market of their customers’ products. As well as providing such basic
functions, Xylon has developed IP that performs tasks such as face detection and tracking,
vehicle detection and vehicle driver drowsiness
detection, all targeted at niche markets.
Xylon is not the only company to have developed such IP. For motion detection of ground,
surface and aerial objects, Riftek (Minsk,
Republic of Belarus, https://riftek.com) offers
FPGA IP that allows both tracking and motion
detection of objects. The company’s T-COR-
30 tracking core, for example, can be used in
both Xilinx and Altera FPGAs.
For stereo vision applications, both Nerian
Vision Technologies (
Leinfelden-Echterdingen, Germany; https://nerian.com) and Fujisoft (Kanagawa, Japan; www.fsi.co.jp) both
offer FPGA IP to perform stereo object matching, albeit using different algorithms.
After images are first rectified to compensate for lens distortions and camera alignment errors, Nerian Vision Technologies
stereo vision core performs stereo matching on two grayscale input images by applying a variation of the Semi Global Matching
(SGM) algorithm, a method originally developed by Heiko Hirschmueller of the German
Aerospace Center (DLR; Wessling, Germany;
www.dlr.de). His original 2005 paper entitled
“Accurate and Efficient Stereo Processing by
Semi-Global Matching and Mutual Information,” can be found at http://bit.ly/VSD-SGM.
Other stereo methods can be used to perform this task, most notably the sum of absolute difference (SAD) algorithm that has been
used by Dr. Keiji Saneyoshi, Associate Professor of Tokyo Institute of Technology (Meguro,
Japan; www.titech.ac.jp). In conjunction with
Altera, Fujisoft deploys this algorithm in its
Figure 2: Supporting the switched mezzanine
card (XMC) format, the Condor 4100 XMC
Series from EIZO Rugged Solutions is based
on AMD’s Radeon E8860 GPU and support
camera input standards such as 3G-SDI, HDMI,
NTSC and CoaXPress (CXP).