MI SYS Debug SOP¶
REVISION HISTORY¶
| Revision No. | Description |
Date |
|---|---|---|
| 1.0 | 12/21/2023 |
Preface¶
This document is written for FAE and software development personnel. Its purpose is to introduce how customers can perform preliminary troubleshooting independently when encountering MI SYS-related issues during the development process, and provide relevant information to RD for analysis after confirming that the issue is an SDK problem.
1. MMU Memory Corruption Issue¶
-
The following log indicates that MMU memory corruption has occurred.
[MI WRN] MI_SYS_Mma_MmuCallback[227]: [MI_SYS_Mma_MmuCallback] Status=0x2, PhyAddr=0x40070000, ClientId=0x70,Name=CPU_CA35 IsWrite=0- Status
- The reason for triggering this callback. The current value is 0x2, which indicates an illegal read/write operation.
-
PhyAddr
- The starting address of the entry where the access triggered the exception. The current value is 0x40070000.
-
ClientId
- Module ID. The current value is 0x70.
-
Name
- Module name. The current value is CPU_CA35.
-
IsWrite
- Read/write operation. 1 indicates write, 0 indicates read. The current value is 0, indicating a read operation.
- Status
-
Analysis
-
Before running the application, enter the following command to collect detailed information about MMA memory allocation and deallocation:
echo debug_mmu debug_log 1 > /proc/mi_modules/mi_sys/mi_sys0If PhyAddr is located between the fr (free) and ed (end) addresses of a released buff, it indicates that the module accessed already freed memory.
al:64000 sz:cac0 ed:70ac0 na:app-mmaAlloc fr:64000 sz:cac0 ed:70ac0 na:app-mmaAlloc [MI WRN] MI_SYS_Mma_MmuCallback[227]: [MI_SYS_Mma_MmuCallback] Status=0x2, PhyAddr=0x40070000, ClientId=0x70,Name=CPU_CA35 IsWrite=0From the log, PhyAddr= 0x40070000. The starting address of the VPA can be obtained from /proc/mi_modules/mi_sys_mma/mma_heap_name0. The current VPA starting address is 0x40000000, so the offset address of PhyAddr relative to the VPA starting address is 0x70000, which is within the freed memory range (0x70000 is within the 0x64000-0x70ac0 range). It is highly likely that the CPU accessed the app-mmaAlloc buffer after it was freed.
-
If the PhyAddr printed by MmuCallback is not within the already freed memory address range, suspect that too many logs caused some logs to be overwritten. Since the module causing memory corruption has been identified, memory allocation information can be filtered for only the corresponding module.
echo debug_mmu debug_log [enable/disable] [enable/disable free] [enable/disable alloc] [module id list] > /proc/mi_modules/mi_sys/mi_sys0Examples:
- Enable VIF/AI alloc/free buffer logging
echo debug_mmu debug_log 1 1 1 4,6 > /proc/mi_modules/mi_sys/mi_sys0
- Enable VIF/AI alloc/free buffer logging
-
If the above does not reveal access to a freed buff, enable debugMmu for debugging. Modify the /config/modparam.json file, add {"debugMmu" : 1} to E_MI_MODULE_ID_SYS, and then restart the machine and run the application.
"E_MI_MODULE_ID_SYS" : { "cmdQBufSize" : 400, "debugmmu": 1 },At this point, when the hardware performs extra reads or writes, it can be captured 100% of the time, printing logs similar to the following:
al:1480000 sz:320000 ed:17a0000 na:vde_rec_0_0_0 [MI WRN] MI_SYS_Mma_MmuCallback[425]: [MI_SYS_Mma_MmuCallback] Status=0x2, PhyAddr=0x401780000, ClientId=0x38,Name=VENC0_CODEC0_R IsWrite=0First, calculate the entry size: after booting, cat /proc/mi_modules/mi_sys_mma/mma_heap_name0. The length of heap_info divided by freeEntryNum is the entry size. At this time, the MMU entry size is set to 128K. If PhyAddr+entry size=ed, it indicates that the module performed an out-of-bounds memory access. In the above log, 0x17a0000 (ed) = 0x1780000+0x20000 (entry size), indicating that the VENC0_CODEC0_R module performed an out-of-bounds memory access.
-
Find the corresponding module owner based on the Name, provide the above printed logs, and inform the owner of the above analysis conclusion.
-
2. MIU Protect Issue¶
When the following log appears, it indicates that MIU protect has occurred:
[HAL MIU ERR] [hal_miu_protect_show_hit_info@1301] [PROTECT] IP hit protect address.
Hit block 32, address: 0x400800000<->0x400802000, whitelist: [CPU_CA55, ]
Hit count: 1
Hit IP : 0x45 - BDMA
Hit MMU address: 0x400800000<->0x40080000f
The analysis and positioning can follow the following steps:
- The reason for MIU protect: "IP hit protect address" indicates that an IP outside the whitelist accessed a protected address; "IP write out of dram" indicates that an IP accessed an address outside the DRAM range or MMU memory corruption issue occurred. Kernel panic needs to be disabled to observe whether MMU memory corruption logs appear to further confirm the cause. (The method for disabling kernel is described in point 5)
- From the information in the log, it can be determined that the problematic IP is BDMA, which accessed the address 0x400800000<->0x40080000f in block 32 (the address range of this block is 0x400800000<->0x400802000, and the whitelist is set to only allow CPU_CA55 access)
- Seek help from the BDMA IP owner. Provide the logs as well as information about BDMA usage and parameter passing in the current scenario.
- By default, kernel panic occurs after MIU protect. If panic is not desired when a problem occurs, it can be achieved by setting the "g_bEnableMiuPanic" field in modparam.json to false. After setting, restart or reload mi_sys.ko.
- The current system's MIU protect status can be obtained by cat /proc/mi_modules/mi_sys_mma/miu_protect. The default policy is to protect all memory in kernel lx_mem except MMA, set BSP-related IPs as whitelist, and do not allow stream pipe-related IPs to access
3. Insufficient Frame Rate Issue¶
The insufficient frame rate issue specifically manifests as:
The player pulls the stream and sees that the corresponding fps does not reach the set value
If an insufficient frame rate issue occurs, analysis and positioning can follow the following steps:
- watch -n 1 'cat /proc/mi_modules/mi_venc/mi_venc*'
---------------------------- OutputPort of dev: 0 ----------------------------- ChnId CODEC Profile BufSize MinAllocSize RefNum bByFrame FrameCnt DropCnt ReEncCnt RingUnreadCnt RingTotalCnt UsrLockedCnt 0 H265 0 4147200 7273 0 1 871390 0 0 0 0 0 -----------------------------------Input port common info for mi_venc only dump enabled port----------------------------------- ChnId PassId PortId user_buf_quota UsrInjectQ_cnt BindInQ_cnt TotalPendingBuf_size usrLockedInjectCnt 0 0 0 4 0 4 37601280 0 ChnId PassId PortId curWorkingQ_cnt workingTask_cnt lazzyRewindTask_cnt 0 0 0 3 1 0
echo drop_out a d > /proc/mi_modules/mi_venc/mi_venc0
At this point, observe the Fps_1s field in the MI VENC procfs again to see if the frame rate has increased.
ChnId State EnPred base enhance MaxStreamCnt Fps_1s kbps1s Fps10s kbps10s lastQp
0 0 0 0 0 3 29.99 964 29.98 929 20
-
From the downstream module to the upstream module, sequentially cat /proc/mi_modules/mi_xxx/mi_xxx*
Check whether the DropCnt of the InputPort continues to increase. A continuous increase indicates that the corresponding module is dropping frames internally.
ChnId State EnPred base enhance MaxStreamCnt Fps_1s kbps1s Fps10s kbps10s lastQp 0 0 0 0 0 3 12.99 964 13.58 929 20Continue to check whether GetIntoMaxCnt/GetIntoMmaLackCnt/GetIntoTotalCnt/GetIntoFrcCnt continues to grow
ChnId PassId PortId GetFrame/Ms FPS FinishCnt RewindCnt GetTotalCnt GetOkCnt GetIntoMaxCnt GetIntoMmaLackCnt GetIntoTotalCnt GetIntoFrcCnt 0 0 0 13/1071 12.13 982123 10 1903946 982133 0 0 921813 0-- An increase in GetIntoMaxCnt indicates that MaxEnqTasks is full. You can try increasing it. If effective, you need to communicate with the corresponding MI module owner to have the MI module owner modify MaxEnqTasks in the driver.
echo set_Max_EnqTasks <eModuleId> <u8DevId> <u8PassId> <u16ChannelId> <EnqTasksNum> > /proc/mi_modules/mi_sys/mi_sys0 echo set_Max_EnqTasks 23 0 1 0 5 > /proc/mi_modules/mi_sys/mi_sys0-- An increase in GetIntoMmaLackCnt indicates that MMA is insufficient. You can try increasing it in bootargs. If effective, modify the value of the CONFIG_MMA_MEM_SIZE field in defconfig to the adjusted size.
-- An increase in GetIntoTotalCnt indicates that the output port buffer may be insufficient. You can try increasing it. If effective, modify the APP code to change the MI module output port depth. If ineffective, it may be that the downstream module is processing too slowly, causing buffers to pile up in the downstream module. At this point, the reason for the slow processing of the downstream module needs to be analyzed.
echo set_ouputport_depth [Modid] [Devid] [Chnid] [Passid] [Portid] u32UserFrameDepth u32BufQueueDepth > > /proc/mi_modules/mi_sys/mi_sys0 echo set_ouputport_depth23 0 0 1 0 0 6 > /proc/mi_modules/mi_sys/mi_sys0-- An increase in GetIntoFrcCnt indicates that frame dropping is caused by frame rate control. Generally, this means that frame rate control exists.
-
After distinguishing which module and cause are involved in step 2, you can find the module owner for specific analysis to see whether the module HW time consumption is normal, whether BW is normal, and whether the pipe exceeds specifications.
- Enter the sdk/verify/mi_demo directory, make source/miu_server
- Copy sdk/verify/mi_demo/out/arm/app/prog_miu_server to the board
- Run ./prog_miu_server on the board
- Use the BWLA Tool to connect to the board and view BW data
4. MMA Memory Insufficiency Issue¶
The MMA memory insufficiency issue specifically manifests as:
- The application or module returns failure when calling memory allocation interfaces such as MI_SYS_MMA_Alloc
- The module drops frames, and it is found through the corresponding module's proc info that the "GetIntoMmaLackCnt" field is increasing
- Others
If a memory insufficiency problem occurs, analysis and positioning can follow the following steps:
-
Increase MMA memory:
- If it is a pure Linux system, modify the bootargs field in the environment variable and increase the sz of mma_heap_name0.
- If it is a dual OS system, you can increase mma_size by modifying the bootargs environment variable, and correspondingly reduce the size of LX_MEM and mma_base. LX_MEM requires 1M alignment. The mma_base and mma_size of bootargs_rtos must be consistent with the mma_base and mma_size of bootargs.
-
Check the specific memory usage in /proc/mi_modules/mi_sys_mma/mma_heap_name0 and review whether there are issues such as memory leaks (whether the MMA buffer is continuously increasing, whether there are MMA buffers not recycled after destroying the pipeline).
-
/proc/mi_modules/mi_sys_mma/mma_heap_name0 cannot present dynamic information about memory allocation and deallocation. The following commands can be used for further confirmation. When applying for MMA buffer fails, the current MMA heap memory information will be printed. Further analysis can determine whether it is memory insufficiency, leakage, or fragmentation.
echo MI_SYS_Chunk_DumpMgrPrintk > /proc/mi_modules/mi_debug/debug_func
echo 7 > /proc/mi_modules/mi_debug/debug_level
-
Adjust the pipeline. Use binding methods such as realtime and hwring between modules to save memory as much as possible.
-
Provide the scenario diagram, /proc/mi_modules/mi_sys_mma/mma_heap_name0 information, and kmsg printing to the mi_sys owner for analysis and conclusion.
If a memory fragmentation problem occurs, analysis and positioning can follow the following steps:
- Enable MMU. This can be achieved by setting the "bEnableMmu" field in modparam.json to true. After setting, restart or reload mi_sys.ko.
- Optimize the application and use private pools to limit fragmentation issues within a certain range. (For the use of private pools, please refer to the usage instructions of the MI_S32 MI_SYS_ConfigPrivateMMAPool() interface in the MI SYS API)
- Increase MMA memory
5. CMDQ Related Issues¶
5.1. cmdq timeout¶
When CMDQ timeout occurs, it is usually because the hardware interrupt is not generated/a register setting does not meet expectations. At this time, the following information will be output to the serial port:
[CMDQ]cmdq(0) ERR: WAIT_TRIG_TIMEOUT (0x00000400)
[CMDQ]Cmd data = 0x2000 : 0x0000 : 0x0000 : 0xFFFE
[CMDQ]Cmd:WAIT, dbg:0, adr:000000, dat:0000, mask:fffe
[CMDQ]Wait command timeout. Trigger_Bus Bit [0] Event [ISP_CMDQ_DONE]
[CMDQ]dram(s, e, w, r)=(0x40000000, 0x40003ff0, 0x40000020, 0x40000020) sram(w,r)=(002,002)
[CMDQ]cmdq(7) ERR: POLLEQ_TIMEOUT (0x00000200)
[CMDQ]Cmd data = 0x3008 : 0x94F8 : 0xFF00 : 0x00FF
[CMDQ]Cmd:POLLEQ, dbg:0, adr:1129f0, dat:ff00, mask:00ff
[CMDQ]Poll eq command timeout. Bank:0x1129; Offset:0x78; Value:0xff00; Mask:0x00ff
[CMDQ]dram(s, e, w, r)=(0x4001c000, 0x4001fff0, 0x4001c030, 0x4001c030) sram(w,r)=(003,003)
First, determine which module's CMDQ has the problem. There are two methods:
5.1.1. wait trigger timeout¶
When the timeout type is WAIT_TRIG_TIMEOUT, there will be Event [ISP_CMDQ_DONE], which can be judged as an ISP problem.
5.1.2. other¶
In addition, the first line of each log contains the cmdq id, indicating the cmdq id where the timeout occurred. We can view the cmdq id used by the specified module through cat /proc/mi_modules/mi_xx/mi_xx, as follows:
----------------------CMDQ kickoff counter-----------------------
DevId current_buf_size Peak_buf_size
0 0 0
each dev buf info:
offset length used_flag task_name
Compare with the problematic cmdq id to determine which module, and finally find the corresponding module's SW owner and give the log and running scenario to him for analysis.