ISP HDR Tuning Guide¶

REVISION HISTORY¶

Revision No.	Description	Date
1.0	Initial release	12/20/2023

HDR TUNING GUIDE¶

HDR

High Dynamic Range (HDR) combines images from two different exposure conditions to produce an image, of which the dynamic range is greater than that of the image in traditional single exposure condition (also known as the linear mode).

The primary concern of HDR adjustment is the distribution of screen brightness, i.e. how to squeeze as much information you want as possible into the 8-bit 0 ~ 255 range. In Chapter 1, we will first demonstrate AE settings, and then explain HDR/WDR adjustments, which are related to brightness distribution. Later in the document, we will also briefly talk about the pairing of NR/Edge. In Chapter 2, we will show how to deal some of the frequently asked questions.
Compressed HDR

Compressed HDR refers to the HDR image generated directly from the sensor side, and the adjustments to the dynamic range are made by ISP using only one image. In this case, the sensor side plays the most important role in determining the image quality. For ISP, the HDR effect essentially depends on sensor characteristics. For compressed HDR, you may:
1. Lower the AE Target value (to one third or one fourth of the original value);
2. Adjust WDR parameters – GlobalDarkToneEnhance or WDR Curve to determine how much brightess should be enhanced in dark area.

AE Setting¶

By combining one long-exposure image and one short-exposure image together, HDR mode allows the synthesized image to contain more information with the help of short-exposure image, especially when the area in concern tends to get overexposed by long exposure. In this sense, the exposure ratio (HDR Ratio) of the frames in long and short exposures is directly related to the dynamic range of HDR.

However, the HDR Ratio is not always the larger the better, because short-exposure image must be multiplied by HDR ratio to match its long-exposure counterpart in a linear manner. As a result, the larger the HDR Ratio, the greater gain will have to be multiplied, and the greater noise problem may occur to the image.

There are three major parameters in AE settings: 1. Max Short Shutter 2. HDR Ratio 3. AE Target

Max Short Shutter: This parameter is related to a sensor’s frames-per-second capability and its frequency during operation. The maximum limit for this can be derived from the sensor specification and our recommendation is to set this parameter to the maximum value available.

The log file after camera initialization provides the information about gain/shutter/fps of the current sensor driver, as the following figure shows:

isp_gain_max: The maximum value of ISP gain.

sgl_min/sgl_max: The minimum/maximum value of sensor gain for long exposure.

sgs_min/sgs_max: The minimum/maximum value of sensor gain for short exposure.

shl_min/shl_max: The minimum/maximum value of sensor shutter (μs) for long exposure.

shl_step: The time required by sensor -1 line (ns) for long exposure.

shs_min/shl_max: The minimum/maximum value of sensor shutter (μs) for short exposure.

shs_step: The time required by sensor -1 line (ns) for short exposure.
HDR Ratio: This parameter should be determined according to the required dynamic range of the scene involved. Note that HDR ratio determines the “maximum” dynamic range of the HDR effect, but the actual dynamic range of the picture still depends on the settings of Global Dark Tone Enhance, which is set out in the subsequent WDR Adjustment section. Aiming at a very high HDR ratio is not suggested. If HDR ratio is set too low, however, the exposure time of long exposure will be limited, causing long-exposure frame to increase the gain too early, hence producing noise. Normally, we suggest setting the value to 10x (=10240). 16x (= 16384) is also supported.
AE Target: This parameter controls the global brightness of the picture, and is normally set to the same value as the one used in single-frame linear mode. If noise in dark area cannot be brought under good control in the final HDR effect, and WDR has also been adjusted, you may then increase the target slightly to allow AE to help brighten the screen. The noise issue will be smaller than that with WDR.
Adjustment Flow

Step 1: Calculate the longest and shortest exposure time for short exposure based on sensor specification and fps, and fill in the results to the short exposure table. Then, select manual mode for AEHDRMLDynamicRatio, and fill the HDR ratio you want to use into the field. The horizontal axis represents the total gain, which can be set with the same HDR ratio.

Step 2: Multiply the Shutter in the short exposure table with the HDR Ratio and fill the value into the long exposure table (for example, short exposure 2100 μs, long exposure 21000 μs).

HDR Setting¶

The HDR API contains basic HDR parameter settings. Note: SensorExpRatio has been transfered to AE dynamic ratio. The rest of the parameters are described below:

Parameter Name	Description
YwtTh1_ML YwtTh2_ML	Upper and lower threshold of long and short exposure during HDR blending Long-exposure image data will be selected for brightness below th1, and short-exposure image data will be selected for brightness over th2 and above, while the blending of long and short exposure data will be applied to brightness between th1 and th2 (Limitation: th2 > th1). To adjust, set th2 to 0, and increase th1 to the highest value possible without any spot of color deviation caused by long exposure saturation in the image (See Note ). Once th1 is decided, set th2 to a value between th1 + (50 ~ 100). If th2 is set too high, the noise issue in the bright area will be improved, but the details will also be lost (as the blending involves too much long exposure data, which is over-exposed and lack of detail).
NoiseLevel	Noise level, to determine motion and stillness The horizontal axis represents brightness. The larger the value, the more likely an area will be judged as still. Default value is recommended. Fine-tune the default value if necessary.
YSrcStr	The strength of blending he brightness value of fusion by luma with max(R,G,B) The greater the value, the more max(R, G, B) will be blended. YSrcStr[0] represents the strength of long exposure, while YSrcStr[1] represents the strength of short exposure.
NrStr	The strength of noise reduction for short exposure. The greater the value, the stronger the noise reduction.
NrPreStr	The strength of front-end noise reduction for short exposure. The greater the value, the stronger the front-end noise reduction.
NrYTh1	Threshold value of the lowest brightness of noise reduction for short exposure. When brightness is lower than Th1, full noise reduction will be applied.
NrYTh2	Threshold value of the highest brightness of noise reduction for short exposure. When brightness is higher than Th2, noise reduction will not be applied. When brightness falls between Th1 and Th2, blending will be applied.
NrYGain	The vertical axis to control the strength of noise reduction for short exposure based on the brightness difference. The smaller the value, the stronger the noise reduction. 1x = 16.
NrYSft	The horizontal axis to control the strength of noise reduction for short exposure based on the brightness difference. Control the interval between each node. Parameter value: 0 ~ 7, in power of 2.
NrKernelStr	The strength of NR filter for short exposure The greater the value, the stronger the noise reduction. Parameter range: 0 ~ 7.
NrKernelStrWei	The NR filter blending weight table for short exposure The greater the value, the stronger the noise reduction. The horizontal axis represents the difference from the center point, while the vertical axis represents weighting. Under normal circumstances, the smaller the difference, the greater weighting should be set. Parameter range: 0 ~ 31.
MotAdj1Sft	The horizontal axis of Table 1 of long exposure ratio for fusion by motion. The horizontal axis represents brightness. The closer to the right end, the greater brightness is suggested. Parameter range: 0 ~ 7, in power of 2. Adjustment is not recommended.
MotAdj1	The vertical axis of Table 1 of long exposure ratio for fusion by motion. The horizontal axis represents brightness, while the vertical axis represents the blending ratio of long exposure. The greater the value, the more long exposure will be blended; the smaller the value, the more short exposure will be blended, and the dirtier the picute will look — but ghost image issue will be alleviated.
MotAdj2	The vertical axis of Table 2 of long exposure ratio for fusion by motion. The horizontal axis represents stillness to motion, while the vertical axis represents the blending ratio of long exposure. The greater the value, the more long exposure will be blended; the smaller the value, the more short exposure will be blended, and the dirtier the picute will look — but ghost image issue will be alleviated. We suggest that you use long exposure as much as possible for both motion and still areas.
MotAdj3	The vertical axis of Table 3 of long exposure ratio for fusion by motion. The horizontal axis represents stillness to motion, while the vertical axis represents the blending ratio of long exposure. The greater the value, the more long exposure will be blended; the smaller the value, the more short exposure will be blended, and the dirtier the picute will look — but ghost image issue will be alleviated. We suggest that you use long exposure as much as possible for both motion and still areas.
MoLuBlendX MoLuBlendY	Table of blending ratio of Fusion by Luma and Fusion by Motion Use motion information to query this table of blending ratio, and get the blending weight of fusion by motion. The horizontal axis represents motion; the closer to the right end, the more motion is suggested. The vertical axis represents the blending ratio; the larger the value, the higher blending ratio of the fusion by motion. MoLuBlendX controls the spacing of each node. Parameter range: 0 ~ 7, in power of 2. MoLuBlendY controls the weight of each node. Parameter range: 0 ~ 256.

Note: Because AWB gain will be applied before HDR fusion, various abnormal colors (not necessarily white) often occur in the saturated position of the long exposure image. The common feature is that details will be lost. To solve this issue, you need to increase YSrcStr or adjust th1 downward to avoid overexposed long exposure segments appearing in the final image.

HDR EX Setting¶

Parameter Name	Description
DbgMode	Debug mode Show motion information. Grey indicates still areas, while black and white indicates motion areas. 0: Disabled. 1: Enabled.
DynRatioEn	Dynamic brightness compensation of long and short exposures If a sensor’s actual difference in brightness between long and short exposures deviates from the value set to SensorExpRatio, enabling this function can alleviate the difference to a certain degree (for example, when the calculated shutter value of long and short exposures cannot be a specified multiple due to precision issue, you can enable this function to eliminate the difference).
NrEn	Enable additional noise reduction for short exposure. Enable this function allows short-exposure image to become cleaner, but blurring might occur.
DynRatioSrcSel	Select reference frame for dynamic brightness compensation of long and short exposures. 0: Long exposure; 1: Short exposure.
YSrcSel	The source of YwtTh1 and YwtTh2 brightness for fusion by luma. The blending ratio of long and short exposures as determined by fusion by Luma according to the current brightness (Yin), YwtTh1, and YwtTh2. 0: Long exposure; 1: Short exposure.
NoiseLevelSrcSel	The source of brightness for NoiseLevel NoiseLevel will query the table according to brightness. The value of Y can be obtained from the following four options: 0: Long exposure; 1: Short exposure; 2: Smaller value between long and short exposures; 3: Larger value between long and short exposures. The default setting is 3; adjustment is not recommended.
MotMaxVal	The maximum value of motion range Motion information will be checked by this maximum value. The smaller the value, the smaller the maximum value. That is, smaller difference will be regarded as greater motion; as a result, the place where noise occurs will more easily be judged as a motion area, causing more short exposure to be applied and noisier image. The default setting is 7; adjustment is not recommended.
MotionAdjSft2	The horizontal axis of MotAdj2 and MotAdj3 The horizontal axis represents motion intensity, from stillness to motion. Parameter: 0 ~ 7, in power of 2. Adjustment is not recommended.

Adjustment Flow

Step 1: Observe the gradual change from high brightness area to dark area in the scene, and then adjust fusion by luma accordingly. We recommend that you set YwtTh1 to 700, YwtTh2 to 900, YSrcStr to 32 or above, and YSrcSel to 0.

Step 2: Observe the issue of ghost image surrounding moving objects in the scene, and then adjust fusion by motion and fusion result. If the moving objects do not look clear enough, apply more short exposure; if the moving objects look too clear, apply more long exposure. We recommend that you use default value for MotAdj1, MotAdj2, MotAdj3, MoLuBlendY.

Step 3: Observe the noise issue of short exposure, and then adjust NR. Please refer to the subsequent chapter on NR adjustment for more details.

WDR Adjustment¶

After long and short exposure images are synthesized in a 16-bit space by HDR, Global Dark Tone Enhance of WDR will be applied to re-allocate the brightness and reduce the image to 12-bit. Generally, we need to apply Global Dark Tone Enhance to brighten the dark area of long exposure in the image, or to restore the brightness to that of a normal single frame. With the help of the curves provided by Global Dark Tone Enhance, we are able to integrate the data of short-exposure frame into the area of medium/high brightness or overexposed area.

Global Dark Tone Enhance features 16 curves to choose from by default, in which Curve 0 aims to maintain linearity without brightness enhancement. As curve number adds up, the level of brightness enhancement gradually increases. If, after adjusting Global Dark Tone Enhance, dark area is still not bright enough, you can try adjusting the main Gamma to brighten the dark area.

The list of WDR adjustment parameters is as follows:

Parameter Name	Description
GlobalDarkToneEnhance	Curves for Global Dark Tone Enhancement A total of 16 default curves are available for selection. For details, please refer to the descriptions in the first half of this chapter. GlobalDarkToneEnhance will be ineffective if WDRCurveFull Curve 2 is enabled.
WDRStrByY	Control of WDR blending strength by Y Control the strength of major WDR blending route according to the different positions of brightness in the picture. The table from left to right corresponds to blocks from dark to bright. It is recommended to fine tune this parameter based on the basis of default setting.
WDRStrByYSft	The nodes of WDRStrByY on the horizontal axis Parameter range: 1 ~ 15.
WDRStrByYOft	The offset of WDRStrByY Parameter range: 0 ~ 4095.
Strength	The global strength of WDR blending This function is ineffective in HDR mode, and the value is fixed at 255, which represents the result of using the major route of WDR. To reduce the strength, use WDRStrByY.
DarkLimit	The strength limit for the brightening of dark area If any adverse effect such as color abnormality or excessive noise emerges due to the brightnening of dark area after WDR parameters have been adjusted, you may adjust DarkLimit to alleviate the side effects by imposing the limit upon the the strength of dark area brightening.
BrightLimit	The strength limit for the darkening of bright area The usage of this parameter is similar to DarkLimit, but the area concerned is bright area.
DeSatSrcSel	Select the source of brightness for desaturation 0 represents the brightness processed by Curve1; 1 represents the brightness processed by global tone; and 2 represents the brightness not yet processed by WDR. We recommend using 1.
DeSatYMode	Select the mode of brightness for desaturation Parameter range: 0 ~ 1. 0 represents YCbCr mode, and 1 represents RGB mode.
DeSatCrEn	Enable independent setting of Cr color space Setting this parameter to 0 will not allow Cr color space to be set independently, which means that DeSatCrLut is not effective, and Cr will share the same setting with Cb. Setting it to 1 will enable independent setting of Cr color space.
DeSatCbLut	Adjust desaturation intensity in Cb color space based on brightness The horizontal axis represents brightness: the closer to the right end, the brighter the image. The vertical axis represents color gain (128 = 1x): the smaller the value, the stronger the desaturation, and the greyer the image. You may lower the value for dark areas if abnormal color or color noise occurs after the dark area has been brightened.
DeSatCrLut	Adjust desaturation intensity in Cb color space based on brightness This parameter shares the same usage as DeSatCbLut, but it works on Cr color space.
DeSatSft	The nodes of DeSatCbLut/DeSatCrLut on the horizontal axis Parameter range: 1 ~ 15.
DeSatOft	The offset of DeSatCbLut/DeSatCrLut Parameter range: 0 ~ 4095.
DeSatBlendV	The strength of blending Y (brightness) of desaturation with V (brightness) of HSV The greater the value, the more V will be blended. This parameter is only effective when DeSatSrcSel is set to 3. Parameter range: 0 ~ 255.
WDRCurveFull	Global Dark Tone Enhancement Curve Three curves for WDR adjustment are available for manual control: Global Tone: the curve that brightens the major route. Curve1: the curve that brightens the minor route. Curve2: the curve that brightens the WDR output, which shares the same function with GlobalDarkToneEnhance.

After applying Global Dark Tone Enhance to enhance the general brightness of the picture, the issues of poor contrast and insufficient transparency often occur. In this case, you may slightly add extra LTM (Local Tone Mapping) effects to enhance the sense of layering.

The list of WDR LTM Adjustment parameters is as follows:

Parameter Name	Description
DbgEn	Debug mode Debug mode, showing the effective area and strength of LTM. Parameter range: 0 ~ 1.
Coarse.BoxNum	The scale of local analysis in Coarse Four box sizes are available. Box size can be adjusted according to the size of item of focus in the scene. The box size corresponding to Coarse > the box size corresponding to Medium > the box size corresponding to Fine. The larger BoxNum is set, the smaller the scale of analysis will be, and more ideal for focusing on small objects in the picture. On the other hand, larger scale of analysis allows the overall picture to look more natural. We suggest that you set BoxNum to 4, the maximum value.
Medium.BoxNum	The scale of local analysis in Medium Four box sizes are available. Box size can be adjusted according to the size of item of focus in the scene. The box size corresponding to Coarse > the box size corresponding to Medium > the box size corresponding to Fine. The larger BoxNum is set, the smaller range of brightness optimization can be paid attention to. On the other hand, larger scale of analysis allows the overall picture to look more natural. We suggest that you set BoxNum to 4, the maximum value.
Fine.BoxNum	The scale of local analysis in Fine Four box sizes are available. Box size can be adjusted according to the size of item of focus in the scene. The box size corresponding to Coarse > the box size corresponding to Medium > the box size corresponding to Fine. The larger BoxNum is set, the smaller range of brightness optimization can be paid attention to. On the other hand, larger scale of analysis allows the overall picture to look more natural. We suggest that you set BoxNum to 3 or 2.
LocalStrength	The strength of local tone mapping kernel The overall strength of local tone mapping kernel. We suggest that you set the strength to the 255, the maximum value.
LevelStrength	The strength of level The greater the value, the stronger enhancement upon the contrast between bright and dark areas. We suggest that you set the strength to the 255, the maximum value.
LocalDetailEnhance	Local detail enhancement Parameter range: 0 ~ 4. The larger the value, the stronger the strength.
Coarse.FltCoef	The strength of coarse filter The greater the value, the more apparent the effect of local contrast enhancement, but halo is more likely to appear on the edge of object. We suggest that you set the value to 4, the maximum value.
Coarse.ToneMapStr	The strength of coarse local tone mapping The greater the value, the more apparent the effect of local contrast enhancement, but halo is more likely to appear on the edge of object. We suggest that you set the value to 255, the maximum value.
Fine.FltCoef	The strength of fine filter The greater the value, the more apparent the effect of local contrast enhancement, but halo is more likely to appear on the edge of object. We suggest that you set the value to 1.
Fine.ToneMapStr	The strength of fine local tone mapping The greater the value, the more apparent the effect of local contrast enhancement, but halo is more likely to appear on the edge of object. We suggest that you set the value to 255, the maximum value.

NR Adjustment¶

Short exposure NR is available for adjustment in HDR mode. A brief adjustment flow is set out below.

Adjust AE manually:
- Adjust the gain value to the ratio to be adjusted.
Switch HDR to pure short-exposure export and observe the status of short exposure.
- Switch to manual mode in HDR tab, set YwtTh1 = 0, YwtTh2 = 0, and fill 0 to the row of MoLuBlendY.
Adjust short-exposure OBC:
- Fill in the correct short-exposure OBC.
- Switch to and fro between normal HDR fusion and pure short-exposure export (switch between auto and manual modes in HDR tab), and observe the fusion area to see if color deviation occurs.
- If color deviation occurs, fine-tune short-exposure OBC and try to reduce the deviation as much as possible.
Adjust short-exposure NR:
- Switch to and fro between normal HDR fusion and pure short-exposure export (switch between auto and manual modes in HDR tab), and observe the fusion area to see if noise occurs.
- If noise occurs, increase short-exposure noise reduction. NrStr parameter adjustment should be prioritized before NrPreStr. Radical adjustment should be avoided, otherwise details in short-exposure area will be lost.
Adjust 3DNR:
- Perform normal HDR fusion.
- Enter debug mode and adjust Md.Thd, Md.ThdByY, Md.Gain, and Md.GainByY until general picture color approaches bright gray. Normally, when the brightness exceeds a certain level, there will be fusion area and short-exposure area, where noise is more intense and closer to motion (in black). Try to adjust 3DNR parameters until the general picture color approaches bright gray as best as you can.
Adjust NR Luma Adv:
- Perform normal HDR fusion.
- Enter debug mode and adjust EdgeThbyLuma so that detail areas in the picture show white, weaker detail areas show gray, and the area desired to be smoothed show black.
- Enter debug mode, and ask a guy to walk around or wave his/her hand in front of the camera. Adjust EdgeThbyMot and try not to make motion area totally black. Motion area should show light gray evenly.
- Exit debug mode, and ask a guy to walk around or wave his/her hand in front of the camera. Observe both motion and still areas for the balance between noises and details in the picture by adjusting SF3_StrByLuma and SF3_StrByMot.
Return to Step 1 and adjust the value of other gains.

FREQUENTLY ASKED QUESTIONS¶

Flicker¶

Under HDR mode, short-exposure shutters often fail to reach the shortest physical limit of de-flicker. As a result, flicker problems are usually inevitable in short-exposure areas under cathode ray tube light. In this case, we recommend adjusting fps to the multiple of power frequency (e.g. 60Hz/30fps, 50Hz/25fps) to fix the banding problem and avoid rolling, as the following figures show. Note that it is not yet possible to eradicate this problem.

→ Rolling bands occur when flicker and fps are not aligned.

→ When flicker and fps are aligned, banding is fixed, but the problem cannot be fundamentally alleviated.

We suggest making the following adjustments when banding occurs to short-exposure image in HDR mode:

Set MoLuBlendY to 256 and use the result of fusion by motion.
Set MotAdj2 and MotAdj3 to 256 so that fusion by motion adopts long exposure.
Set the brightest node of MotAdj1 to a smaller value so that short exposure will be adopted, and set all the other nodes to 256 to adopt long exposure.
Set NoiseLevelSrcSel to 3.
Reduce NoiseLevel.
Try to increase YwtTh1 and TwtTh2 as much as possible, such as setting them to 900 and 950, respectively.

Banding occurs in static area under HDR mode when short-exposure images are adopted.

No banding is found in static area under HDR mode when long-exposure images are adopted.

Flickering light in the blending area of long and short exposure¶

Flickering light often occurs due to the mismatched brightness between long and short exposure frames. To alleviate the issue, you may check the consistency between HDR Ratio setting in AE and SensorExpRatio setting in HDR API. You may also disable the dynamic compensation mechanism of Dynamic Ratio and set MotionTh to 256 in disregard of the change in motion. By doing so, the blending of long and short exposure frames will solely rely on the setting of YwtTh1 & YwtTh2. Take a closer look to see if there is a clear distinction on the boundary between long and short exposures, so as to clarify whether there is a brightness mismatch.

To verify whether Sensor – Gain/Shutter are linearly matched and to check whether Long & Short – Gain/Shutter corresponds to HDR Ratio.

To verify whether Sensor is linearly matched: Perform a test in single scene (not complex scene) with a fixed and stable light source, switch AE to manual mode. Adjust shutter from small to large and see if the brightness of long and short exposure images shows a linear change. Make sure that the picture is not overexposed at the maximum value. After that, apply the same steps to the sensor gain, and see if the brightness of long and short exposure images shows a linear change. Make sure again that the picture is not overexposed at the maximum value. When the brightness of long and short exposure images shows a linear change and the proportion corresponds to HDR Ratio, the brightness output of sensor can be judged as accurate.
Apply the same setting to gain and shutter in linear mode and HDR mode, and make sure that the resulting brightnesses closely matched each other.
If, after following the steps above, gain and shutter still appear to be non-linear, you may adjust DynamicRatio of HDR for compensation, as the following figures show.

Nonlinear brightness around the light source, DynamicRatio = Disabled

Brightness compensated by HDR parameter setting, DynamicRatio = Enabled

Regional color shift and loss of details in areas of medium to high brightness¶

To deal with this issue, try increase YSrcStr or lower the value of YwtTh1 in HDR API.

Human face too dark in indoor/outdoor/backlight scenario¶

If this happens, try the following adjustment in the order indicated:

Increase GlobalDarkToneEnhance. The larger the value, the brighter the dark area will become, but the blurrier the general picture will be.
Adjust Global Tone in WDRCurveFull to brighten the dark area, but the noise may increase as a result.

Note: If dark area is overly brightened, LTM effects may be weakened.
Increase AE target, but details in over-exposed outdoor area may be lost.
Do not set Fine.BoxNum too large. Although greater Fine.BoxNum helps achieve better contrast, human face in relatively dark area will become darker.
Do not set WDRStrByY too small. If dark area brightening needs to be limited, adjust DarkLimit instead.

Difference in the selection of long and short exposure for image compensation to solve ghost images in HDR mode.¶

When long exposure is chosen for image compensation, the area containing moving objects in the picture will show blurrier motion but less ghost image.
When short exposure is chosen for image compensation, the area containing moving objects in the picture will show clearer motion but relatively more severe ghost image.