SCL Debug SOP

REVISION HISTORY¶

Revision No.	Description	Date
1.0	Initial release	12/07/2023

Introduction¶

This document is written for FAE and software development personnel, aiming to introduce how to conduct preliminary troubleshooting when customers encounter SCL-related issues during the development process, and to provide relevant information to FAE for analysis after confirming it is a sgs SDK issue.

1. Frame Drops, Unstable Frame Rate¶

Debugging process reference diagram:

TIPS: First clarify whether it is an issue with this module.

Determine if the input frame rate is sufficient.

If the input fps meets the requirements, then it is an issue with this module.

If the input fps does not meet the requirements, but rewindcnt > 0, or dropcnt > 0, and is continuously increasing, then it is an SCL module issue; otherwise, analysis of the upstream is required.

1.1 Occasional Frame Drops¶

echo 4 > /proc/mi_modules/mi_scl/debug_level;
echo fpsth [chnid] [portid] [fpsth] [fpsfloatth] > /proc/mi_modules/mi_scl/mi_scl[x];

fpsth is the integer part of fps.

fpsfloatth is the decimal part of fps (multiplied by 1000).

Set the frame rate threshold; if it is below the threshold, it will be printed on the serial port. Confirm that no operations were performed on the module during the frame rate fluctuation.

Capture proc once when the frame rate is normal, then capture it again after the frame rate fluctuates, and provide the log to FAE for analysis.

1.2 Frame Rate is Stable but Not Meeting Expected Frame Rate¶

If the frame drops started with this module, observe the following fields in the proc information, distinguishing between hardware and software situations:

1.2.1 Troubleshooting Reasons¶

In "SCL CHN Info," if dropcnt is not 0 and is continuously increasing, it indicates that software is experiencing drops.

1.2.1.1 Unable to Obtain Output Buffer

If EnqOTNull is not 0, all outputs cannot obtain the output buffer; if FailCnt in "scl OUTPUT PORT info" is not 0, the corresponding port cannot obtain the output buffer.

The inability to obtain the output buffer can be categorized into the following four situations (as shown in the red box in the image above). Please observe which one is continuously increasing and find the corresponding debugging method below, then inform FAE of the experimental conclusions.

GetIntoMaxCnt: max enqueue count exceeds limit

The experimental case is as follows, and please send the experimental conclusions and logs to FAE for analysis:

Increase Max_EnqTasks Count:

echo set_Max_EnqTasks [modid] [devid] [pass] [chn] [cnt] > /proc/mi_modules/mi_sys/mi_sys0; //cnt defaults to 2, can be increased to around 4.

Clear the cached buffers in the module:
```
echo clearbindq [chnid] [bclear] > /proc/mi_modules/mi_scl/mi_scl[x]
exp: echo clearbindq [chnid] 1 > /proc/mi_modules/mi_scl/mi_scl[x]
```
After seeing atom=0, close the clearing action.

echo clearbindq [chnid] 0 > /proc/mi_modules/mi_scl/mi_scl[x]

If the frame rate is normal after closing the clearing action, and atom <=2, it is caused by accumulated buffers during initialization.

Modparam.json sets initialization parameters, Chndbglv and ChnMaxEnqTasks, where the first two parameters are DevId/ChnId.

If you see atom equal to the set max enqueue task count, you can capture the following logs:

echo 4 > /proc/mi_modules/mi_scl/debug_level; (before running the application)
echo debuglv [chnid] 9 > /proc/mi_modules/mi_scl/mi_scl[x];
echo debuglv 4 > /proc/mi_modules/mi_scl/mi_scl[x];
echo clearbindq [chnid] 1 > /proc/mi_modules/mi_scl/mi_scl[x];
cat /proc/kmsg;
echo clearbindq [chnid] 0 > /proc/mi_modules/mi_scl/mi_scl[x];

Provide proc & kmsg to FAE to observe whether the increase in enqueue task count is reasonable in terms of timing.

If the frame rate is normal after closing the clearing action, and 2 < atom < max enqueue task count, also capture log information according to the previous command for FAE analysis to determine if it is in a specification-critical situation.

GetIntoMmaLackCnt: insufficient MMA memory

Inform FAE of the issue and request SYS owner to assist in adjusting MMA memory.

GetIntoTotalCnt: total buffer depth exhausted

echo debug_frc on [Modid] [Devid] [Chnid] [Passid] [Portid] out > /proc/mi_modules/mi_sys/mi_sys0;

This will print the buffer distribution situation when the depth is exhausted on the serial port. If the printing is too frequent, it can be turned off by:

echo 0 > /proc/sys/kernel/printk;

Turn off printing and use cat /proc/kmsg to capture information. Observe where the buffer is being occupied more. Generally, the system will limit the number of buffers that the module can obtain to 2, so it is unlikely that this module is taking too many; it is usually due to insufficient output port depth or the downstream bindQ not being cleared in time. If it is due to insufficient depth, you can:

echo set_ouputport_depth [Modid] [Devid] [Chnid] [Passid] [Portid] u32UserFrameDepth u32BufQueueDepth > /proc/mi_modules/mi_sys/mi_sys0

Increase the depth and then observe where the buffer is being occupied more. If increasing the depth improves the frame rate, but the application scenario requires memory savings, capture proc information and the active time of hardware processing from the upstream and downstream modules to provide to FAE for analysis to determine if it is reasonable.

If the downstream bindQ is occupying more, please have the downstream module owner analyze it.

GetIntoFrcCnt: frame rate control situation causing frame drops

You can observe whether the downstream proc has set frame rate control. If it has been set correctly but the expected frame rate is not achieved, please have FAE contact the downstream owner for assistance in analysis.

1.2.1.2 Software is Slow to Push Buffers to Hardware

If there is no drop count in the proc information, but Atom0 count is steadily increasing, it indicates that hardware performance has idle time, but software is not pushing buffers in time.

Resolution steps:

Raise the module thread priority

Confirm whether the current CPU loading is high. Increase the SCL thread priority, which defaults to 98. If the current chip supports multi-core, you can also try setting the SCL thread to other idle cores;

Modparam.json adds parameters for SCL:

u32threadPriorityArray = [99], where the index is DevId, set thread priority to 99.

u16CpuMaskAffInityArray = [6], where the index is DevId, value 0: default use all CPU, bit0~3 represent cpu0~3, for example, 6 means Dev0 thread uses cpu1 & cpu2.
```
"E_MI_MODULE_ID_SCL" :
{
    "u32threadPriorityArray": [99],
    "u16CpuMaskAffInityArray": [6]
},
```

Capture timing logs to clarify where the delay occurs, and please have FAE analyze:

echo 4 > /proc/mi_modules/mi_scl/debug_level;
echo debuglv [chnid] 9 > /proc/mi_modules/mi_scl/mi_scl[x];
echo debuglv 4 > /proc/mi_modules/mi_scl/mi_scl[x];

2. No Output Image¶

2.1 Is There a Consumer?¶

The module will only operate if there is a consumer, which can be either a user or a downstream bind.

So, confirm that user depth > 0, or that there is a downstream bind, and both are enabled. SCL enable is determined by the MI_SCL_StartChannel API.

2.2 Is There a Stream from the Upstream?¶

Determine if the input frame rate is greater than 0.

If the input frame rate is greater than 0, but rewindcnt > 0, or dropcnt > 0, and is continuously increasing, logs need to be captured, and FAE should be involved in the analysis.

 echo 4 > /proc/mi_modules/mi_scl/debug_level;
 echo debuglv [chnid] 255 > /proc/mi_modules/mi_scl/mi_scl[x];
 echo debuglv 255 > /proc/mi_modules/mi_scl/mi_scl[x];
 cat /proc/kmsg

3. Image Distortion¶

3.1 Upstream Image is Distorted¶

Use the following command to perform an SCL pattern test.

echo patengen [chnid] [bEn] 0 > /proc/mi_modules/mi_scl/mi_scl[x];

If distortion does not occur, it is likely that the input source is already distorted. If the source is in frame mode, use the following command to dump the input and output to confirm that the content sent from the upstream is already problematic, and involve the upstream for analysis. If the source is in real-time mode, dumping is not necessary; directly involve the upstream for analysis.

echo dumptaskfile [chnid] 2 [path] > /proc/mi_modules/mi_scl/mi_scl[x];

3.2 Issue with This Module¶

If using the SCL pattern still results in distortion, check if the 'DVCnt' field in the SCL proc information is continuously increasing. If it is, perform the following experiments:

Increase the SCL clock to the maximum to speed up processing. Use cat /proc/mi_modules/mi_scl/debug_hal/clk to check supported clocks, and echo [clk] > /proc/mi_modules/mi_scl/debug_hal/clk to set the clock.
If increasing the SCL clock to the maximum still results in a continuously increasing 'DVCnt' field, increase the h blank of the front-end sensor for testing. The specific method needs to be provided by the FAE.

If the above two methods do not resolve the issue, please involve FAE for analysis.

4. Memory Issues¶

4.1 Increase Reproduction Probability¶

If it is a low-probability issue, enable the debug MMU feature (which can increase reproduction probability), and then try to reproduce the issue. Modify modparam.json to add "debugMmu":true under the E_MI_MODULE_ID_SYS node.

"E_MI_MODULE_ID_SYS" :
{
    "debugMmu": true
},

4.2 Use the Following Commands to Capture Logs, Please Involve FAE for Analysis¶

echo 4 > /proc/mi_modules/mi_scl/debug_level
echo debuglv [chnid] 1 > /proc/mi_modules/mi_scl/mi_scl[x];
echo debug_mmu debug_log 1 > /proc/mi_modules/mi_sys/mi_sys0
cat /proc/kmsg

5. CMDQ Timeout¶

5.1 Real-time Mode¶

If the SCL is bound to real-time sources, first check if the VIF side is normal.

cat /proc/mi_modules/mi_vif/debug_hal/vif0/vif_ints

Check if the Interval (frame interval) time is normal, and whether VREF_RISING is steadily increasing. If not normal, please involve FAE for assistance in analyzing the VIF module; otherwise, use the following commands to capture logs and provide them to FAE for analysis of the SCL module.

echo 4 > /proc/mi_modules/mi_scl/debug_level
cat /proc/kmsg

5.2 Frame Mode¶

Use the following commands to capture logs, and please involve FAE for analysis.

echo 4 > /proc/mi_modules/mi_scl/debug_level
cat /proc/kmsg

6. Blurry Image¶

6.1 Is the Upstream Stream Image Normal?¶

Use the following command to dump the input to first rule out whether the input image is normal.

echo dumptaskfile [chnid] 2 [path] > /proc/mi_modules/mi_scl/mi_scl[x];

6.2 Use the Following Script to Capture Logs and Provide Them to FAE for Analysis of SCL Issues¶

dump_scl.sh