Sensor_Porting_Guide

REVISION HISTORY

Revision No.	Description	Date
1.0	Initial release	07/16/2024

1. Environmental Preparation

Before starting porting, please make sure that the sensor wiring is correct. For MIPI sensor wiring, please refer to this document SENSOR USER GUIDE. The basic principles are the same on different CHIPs.

First, you need to confirm the requirements, including the sensor's resolution, frame rate, pixel width, number of MIPI Lanes, and mclk frequency, give the requirements to the manufacturer, and ask them to provide a data sheet and initialization table.

2. FILE NAME SPECIFICATION FOR SENSOR DRIVER PORTING

In order to rule all the names of Sgs Sensor Driver names, we have formulated the Sgs Sensor Driver porting specification. Please follow the instructions below when porting the Sensor driver.

2.1. Sensor Driver Name Specification

drv_ss_<Sensor Type>_<Sensor interface>_<others>.c

Parameter	Description
Sensor Type	Sensor naming, ex imx307, imx415. Recommended to use lowercase.
Sensor interface	Sensor interface type, ex: mipi, lvds, but parallel sensor can be ignored. Recommended to use lowercase.
others	If there are no other supplements, you can ignore them. Recommended to use lowercase.

3. SENSOR PORTING RELEVANT KNOWLEDGE OF HARDWARE

This document describes how to porting Sgs sensor driver including basic hardware knowledge, software control sensor API (sensor handle), sensor interface API (Sensor-IF) of Sgs and software porting flow. In this document, we take the sensor driver of imx415 as examples. You can follow the instructions in this document to complete the sensor porting step by step.

First we introduce the relevant knowledge of hardware and corresponding source code to make it easier to understand for you.

3.1. Sensor power sequence

Check the power-on sequence spec in the sensor datasheet. Please follow below power sequence 1.1V > 1.8V > 2.9V as the imx415 sensor datasheet required.

Figure 1: The IMX415 sensor power sequence spec

Table 1: The IMX415 sensor power sequence timing table

3.1.1 Corresponding source code

You can control sensor power-on and reset pins through “handle->pCus_sensor_poweron” in the sensor driver handle initial function. If you need to modify sensor power sequence timing, please check the characteristics of sensor and hardware design.

static int pCus_poweron(ms_cus_sensor *handle, u32 idx)  {
    ISensorIfAPI *sensor_if = &handle->sensor_if_api;
    //Sensor power on sequence
    sensor_if->PowerOff(idx, handle->pwdn_POLARITY);   // Powerdn Pull Low
    sensor_if->Reset(idx, handle->reset_POLARITY);     // Rst Pull Low
    sensor_if->SetIOPad(idx, handle->sif_bus, handle->interface_attr.attr_mipi.mipi_lane_num);
    sensor_if->SetCSI_Clk(idx, CUS_CSI_CLK_216M);
    sensor_if->SetCSI_Lane(idx, handle->interface_attr.attr_mipi.mipi_lane_num, ENABLE);
    sensor_if->SetCSI_LongPacketType(idx, 0, 0x1C00, 0);
    if (handle->interface_attr.attr_mipi.mipi_hdr_mode == CUS_HDR_MODE_SONY_DOL)
        sensor_if->SetCSI_hdr_mode(idx, handle->interface_attr.attr_mipi.mipi_hdr_mode, 1);

    sensor_if->PowerOff(idx, !handle->pwdn_POLARITY);
    //Sensor board PWDN Enable, 1.8V & 2.9V need 30ms then Pull High
    SENSOR_MSLEEP(31);
    sensor_if->Reset(idx, !handle->reset_POLARITY);
    SENSOR_UDELAY(1);
    sensor_if->MCLK(idx, 1, handle->mclk);
    SENSOR_DMSG("Sensor Power On finished\n");
    return SUCCESS;
}

NOTE:

Below is Sgs Sensor board schematic of power circuit. 1.1V, 1.8V, 2.9V power will pull high after power-on enable. First 1.1V, then 1.8V and 2.9V, you can use an oscilloscope to measure it on your sensor board.

​

Figure 2: The IMX415 sensor power circuit​​

3.2. Sensor I2C slave address check

Sgs SoC needs to communicate with a sensor through I2C, please check the sensor I2C slave address first.

Table 2: The IMX415 sensor slave id table

NOTE:

Check the sensor board schematic of I2C slave mode Option.

​​

Figure 3: The IMX415 sensor board I2C slave mode​​

3.2.1 Corresponding source code

The sensor driver I2C Setting in Marco define

////////////////////////////////////
// I2C Info                         //
////////////////////////////////////
#define SENSOR_I2C_ADDR     0x34        //I2C slave address
#define SENSOR_I2C_SPEED    300000      //I2C speed, 60000~320000
#define SENSOR_I2C_LEGACY  I2C_NORMAL_MODE
#define SENSOR_I2C_FMT     I2C_FMT_A16D8 //CUS_I2C_FMT_A8D8,
                                            CUS_I2C_FMT_A8D16,
                                            CUS_I2C_FMT_A16D8,
                                            CUS_I2C_FMT_A16D16

NOTE: Sensor-IF of I2C control Interface in handle initial function

handle->i2c_cfg.mode     = SENSOR_I2C_LEGACY;    //(CUS_ISP_I2C_MODE) FALSE;
handle->i2c_cfg.fmt      = SENSOR_I2C_FMT;       //CUS_I2C_FMT_A16D8;
handle->i2c_cfg.address  = SENSOR_I2C_ADDR;      //0x34;
handle->i2c_cfg.speed    = SENSOR_I2C_SPEED;     //NOT Implementation, please config dts
or sysdesc

3.3. Sensor MCLK support

Each sensor can support in different MCLK. That the MCLK list table that we can provide is as follows. You can find the MCLK corresponding to the sensor from the initialization settings provided by the sensor vendor. The sensor MCLK Setting in Marco define

////////////////////////////////////
// MCLK Info                        //
////////////////////////////////////
/******* Support MCLK List *******
*    CUS_CMU_CLK_27MHZ   / CUS_CMU_CLK_21P6MHZ  / CUS_CMU_CLK_12MHZ,
*    CUS_CMU_CLK_5P4MHZ  / CUS_CMU_CLK_36MHZ    / CUS_CMU_CLK_54MHZ,
*    CUS_CMU_CLK_43P2MHZ / CUS_CMU_CLK_61P7MHZ  / CUS_CMU_CLK_72MHZ,
*    CUS_CMU_CLK_48MHZ    / CUS_CMU_CLK_24MHZ    / CUS_CMU_CLK_37P125MHZ,
******End of Support MCLK List*******/
#define Preview_MCLK_SPEED           CUS_CMU_CLK_27MHZ

3.3.1 Corresponding source code

The sensor MCLK control Interface in handle initial function:

////////////////////////////////////
//   set mclk                        //
//   please check pCus_poweron     //
////////////////////////////////////
sensor_if->MCLK(idx ,  ENABLE, UseParaMclk(SENSOR_DRV_PARAM_MCLK()));

4. SENSOR PORTING RELEVANT SOFTWARE CONTROL

After introducing the relevant knowledge of hardware and corresponding source code, we will introduce software control API (sensor handle) and sensor interface API (Sensor-IF) of Sgs in this part.

4.1. Sensor driver support mode define

Sgs sensor driver can provide a variety of registered sensor support behavior modes. Below we will provide instructions on the behavior modes supported by these registered sensors. You can view more details of the macro definition in the path:

driver/SensorDriver/drv/inc/drv_sensor_common.h

4.1.1 Macro definition for pure linear, parallel sensor and 2-frame HDR

#define SENSOR_DRV_ENTRY_IMPL_END_EX (Name, LinearEntry, HdrSefEntry,
                                        HdrLefEntry, PrivateDataType)

Table 3: The sensor driver “SENSOR_DRV_ENTRY_IMPL_END_EX” definition

Parameter	Description
Name	Sensor Name and support mode
LinearEntry	Sensor Linear mode handle initial function
HdrSefEntry	Sensor HDR Short Exposure handle initial function
HdrLefEntry	Sensor HDR Long Exposure handle initial function
PrivateDataType	Sensor Private Data Type

4.1.2 Macro definition for 3-frame HDR

#define SENSOR_DRV_ENTRY_IMPL_END_EX2 (Name, LinearEntry, HdrSefEntry,
                                        HdrLefEntry, HdrMefEntry,
　　　　　　　　　　　　　　　　　　　　 　PrivateDataType)

Table 4: The sensor driver “SENSOR_DRV_ENTRY_IMPL_END_EX2” definition

Parameter	Description
Name	Sensor Name and support mode
LinearEntry	Sensor Linear mode handle initial function
HdrSefEntry	Sensor HDR Short Exposure handle initial function
HdrLefEntry	Sensor HDR Long Exposure handle initial function
HdrMefEntry	Sensor HDR Middle Exposure handle initial function
PrivateDataType	Sensor Private Data Type

For example, the IMX415 sensor supports linear and 2-frame HDR mode, you can refer to the IMX415 sensor driver and fill it in define of “SENSOR_DRV_ENTRY_IMPL_END_EX”.

SENSOR_DRV_ENTRY_IMPL_END_EX(  IMX415_HDR,
                            cus_camsensor_init_handle,
                            cus_camsensor_init_handle_hdr_dol_sef,
                            cus_camsensor_init_handle_hdr_dol_lef,
                            imx415_params
                            );

4.2. Sensor driver handle function introduction

Next we will introduce and explain handle API in the linear and HDR handle initial functions. We can control, set or read any state of the sensor through the handle API. Here we take the sensor driver of imx415 as an example. You can step through the sensor porting of linear and HDR mode according to the instructions in this document.

4.2.1 Handle of Sensor model id API

Set sensor name and mode information

Table 5: Handle of model name API list

Parameter	Description	NOTE
handle ->strSensorStreamName	Register set sensor model for libcamera user by using	MIPI: IMX415_MIPI PARALLEL: IMX323_PARL HDR: IMX415_MIPI_HDR_SEF, IMX415_MIPI_HDR_LEF

4.2.2 Handle of I2C

Set I2C hardware setting for configuration

Table 6: Handle of I2C configuration lists

Parameter	Description	NOTE
handle->i2c_cfg.mode	Register I2C Mode	I2C_NORMAL_MODE /Sensor driver can only use I2C_NORMAL_MODE/ I2C_LEGACY_MODE /Do not use/
handle->i2c_cfg.fmt	Register set I2C format for bit of address & data length.	I2C_FMT_A8D8, I2C_FMT_A16D8, I2C_FMT_A8D16, I2C_FMT_A16D16, I2C_FMT_END /Reserved/ /Ex I2C_FMT_A16D8 means 16 bits Address, 8 bits Data/
handle->i2c_cfg.address	Register set sensor I2C slave address, Sgs SoC supports 8bits slave address	Check From Sensor Datasheet
handle->i2c_cfg.speed	Register set sensor max I2C speed (NOT USED)	I2C speed configuration via dts or sysdesc

4.2.3 Handle of sensor resolution capability

Set sensor support and implement output image information.

Table 7: Handle of sensor resolution capability lists

Parameter	Description	NOTE
handle->video_res_supported.num_res	Register set sensor support resolution number
handle->video_res_supported.ulcur_res	Register set sensor current resolution	Default : 0
handle->video_res_supported.res[n].u16width	Register set sensor support width
handle->video_res_supported.res[n].u16height	Register set sensor support height
handle->video_res_supported.res[n].u16max_fps	Register set sensor support preview maximum fps
handle->video_res_supported.res[n].u16min_fps	Register set sensor support preview minimum fps
handle->video_res_supported.res[n].u16crop_start_x	Register set VIF crop sensor output image start of X
handle->video_res_supported.res[n].u16crop_start_y	Register set VIF crop sensor output image start of Y
handle->video_res_supported.res[n].u16OutputWidth	Register set VIF output active pixel
handle->video_res_supported.res[n].u16OutputHeight	Register set VIF output active line
handle->video_res_supported.res[n].strResDesc	Register set sensor resolution & fps string	CamOsStrncpy(handle->video_res_supported.res[i].strResDesc, “1920x1080@30fps”, CamOsStrlen(1920x1080@30fps));
handle->video_res_supported.res[n].u16MinFrameLengthLine	Register get sensor each line of minimum Frame Length
handle->video_res_supported.res[n].u16RowTime	Register get sensor each line of time
handle->video_res_supported.res[n].u16PixelSize	Register get sensor pixel size of line

4.2.4 Handle of sensor info

Set sensor information for configuration

Table 8: Handle of sensor info interface register lists

Parameter	Description	NOTE
handle->sif_bus	Register set sensor interface bus type	SENSOR_IFBUS_TYPE CUS_SENIF_BUS_PARL = 0, /sensor data bus is parallel/ CUS_SENIF_BUS_MIPI = 1, /sensor data bus is mipi/ CUS_SENIF_BUS_BT601 = 2, CUS_SENIF_BUS_BT656 = 3, CUS_SENIF_BUS_BT1120 = 4,
handle->data_prec	Register set sensor output raw data precision	SENSOR_DATAPREC CUS_DATAPRECISION_8 = 0, /raw data precision 8bits/ CUS_DATAPRECISION_10 = 1, /raw data precision 10bits/ CUS_DATAPRECISION_16 = 2, /raw data precision 16bits/ CUS_DATAPRECISION_12 = 3, /raw data precision 12bits/ CUS_DATAPRECISION_14 = 4, /raw data precision 14bits/
handle->bayer_id	Register set sensor bayer raw start id	SENSOR_BAYERID CUS_BAYER_RG = 0, CUS_BAYER_GR, CUS_BAYER_BG, CUS_BAYER_GB,
handle->RGBIR_id	Register set sensor RGB-IR start id If NOT RGB-IR sensor, please set CUS_RGBIR_NONE	SENSOR_RGBIRID CUS_RGBIR_NONE = 0, CUS_RGBIR_R0 = 1, CUS_RGBIR_G0 = 2, CUS_RGBIR_B0 = 3, CUS_RGBIR_G1 = 4, CUS_RGBIR_G2 = 5, CUS_RGBIR_I0 = 6, CUS_RGBIR_G3 = 7, CUS_RGBIR_I1 = 8
handle->snr_pad_mode	Register sensor master and slave mode	CUS_SENSOR_MASTER_MODE = 0, CUS_SENSOR_SLAVE_MODE = 1,

4.2.5 Handle of mipi info

Set mipi setting for configuration

Table 9: Handle of mipi sensor interface register lists

Parameter	Description	NOTE
handle->interface_attr.attr_mipi.mipi_lane_num	Register set MIPI lane number	Default : 4
handle->interface_attr.attr_mipi.mipi_data_format	Register set MIPI data format	CUS_SEN_INPUT_FORMAT_YUV422, CUS_SEN_INPUT_FORMAT_RGB,
handle->interface_attr.attr_mipi.mipi_yuv_order	Register set MIPI yuv order	For yuv sensor only
handle->interface_attr.attr_mipi.mipi_hdr_mode	Register set MIPI HDR mode	CUS_HDR_MODE_NONE /No HDR/ CUS_HDR_MODE_SONY_DOL /Sony standard Line information output/ CUS_HDR_MODE_DCG /Dual Gain HDR/ CUS_HDR_MODE_EMBEEDED_RAW8 /Embedded 8bit HDR/ CUS_HDR_MODE_EMBEEDED_RAW10 /Embedded 10bit HDR/ CUS_HDR_MODE_EMBEEDED_RAW12 /Embedded 12bit HDR/ CUS_HDR_MODE_EMBEEDED_RAW14 /Embedded 14bit HDR/ CUS_HDR_MODE_EMBEEDED_RAW16 /Embedded 16bit HDR/ CUS_HDR_MODE_COMP /Compress HDR/ CUS_HDR_MODE_LI /Line Interleaved HDR/ CUS_HDR_MODE_COMP_VS /Compress HDR + very short/ CUS_HDR_MODE_VC /Virtual Channel mode/ CUS_HDR_MODE_MAX NOTE: In the old platform, CUS_HDR_MODE_DCG represented virtual channel HDR, and in the new platform it needs to rename CUS_HDR_MODE_VC.
handle->interface_attr.attr_mipi.mipi_hdr_virtual_channel_num	Register set HDR virtual channel number Based on each sensor VC mode header define	For example imx415: Long Exposure Frame : 0 Short Exposure Frame : 1
handle->interface_attr.attr_mipi.mipi_hdr_fusion_type	Register set HDR fusion type	CUS_HDR_FUSION_TYPE_NONE,(default) CUS_HDR_FUSION_TYPE_2T1, CUS_HDR_FUSION_TYPE_3T1,

4.2.6 Handle of parallel info

Set parallel setting for configuration

Table 10: Handle of parallel sensor interface register lists

Parameter	Description	NOTE
handle->interface_attr.attr_parallel.paral_data_format	Register set parallel sensor data format	CUS_SEN_INPUT_FORMAT_YUV422, CUS_SEN_INPUT_FORMAT_RGB,
handle->interface_attr.attr_parallel. paral_yuv_order	Register set parallel sensor yuv order	For yuv sensor only

4.2.7 Handle of lvds info

Set lvds setting for configuration

Table 11: Handle of lvds sensor interface register lists

Parameter	Description	NOTE
handle->interface_attr.attr_lvds.lvds_lane_num	Register set LVDS lane number	Default: 4
handle->interface_attr.attr_lvds.lvds_data_format	Register set LVDS data format	CUS_SEN_INPUT_FORMAT_YUV422, CUS_SEN_INPUT_FORMAT_RGB,
handle->interface_attr.attr_lvds.lvds_yuv_order	Register set LVDS yuv order	For yuv sensor only
handle->interface_attr.attr_lvds.lvds_hsync_mode	Register set HDR virtual channel0 hsync mode	SENSOR_MIPI_HSYNC_MODE PACKET_HEADER_EDGE1 = 0, /packet header edge/ PACKET_HEADER_EDGE2 = 1, /line end edge/ PACKET_HEADER_EDGE3 = 2, /line start edge/ PACKET_FOOTER_EDGE = 3, /packet footerge/
handle->interface_attr.attr_lvds.lvds_sampling_delay	Register set LVDS start sampling delay	BIT[7:0]: clk_skip_ns BIT[15:8]: data_skip_ns
handle->interface_attr.attr_ lvds.lvds_hdr_mode	Register set LVDS HDR mode	CUS_HDR_MODE_NONE /No HDR/ CUS_HDR_MODE_SONY_DOL /Sony standard Line information output/ CUS_HDR_MODE_DCG /Dual Gain HDR/ CUS_HDR_MODE_EMBEEDED_RAW8 /Embedded 8bit HDR/ CUS_HDR_MODE_EMBEEDED_RAW10 /Embedded 10bit HDR/ CUS_HDR_MODE_EMBEEDED_RAW12 /Embedded 12bit HDR/ CUS_HDR_MODE_EMBEEDED_RAW14 /Embedded 14bit HDR/ CUS_HDR_MODE_EMBEEDED_RAW16 /Embedded 16bit HDR/ CUS_HDR_MODE_COMP /Compress HDR/ CUS_HDR_MODE_LI /Line Interleaved HDR/ CUS_HDR_MODE_COMP_VS /Compress HDR + very short/ CUS_HDR_MODE_VC /Virtual Channel mode/ CUS_HDR_MODE_MAX NOTE: In the old platform, CUS_HDR_MODE_DCG represented virtual channel HDR, and in the new platform it needs to rename CUS_HDR_MODE_VC.
handle->interface_attr.attr_lvds.lvds_hdr_virtual_channel_num	Register set HDR virtual channel number Based on each sensor VC mode header define	Long Exposure Frame: 0 Short Exposure Frame: 1

4.2.8 Handle of sensor interface register function point

Table 12: Handle of sensor control register function point lists

Parameter	Description	NOTE
handle->pCus_sensor_poweron (struct __ss_cus_sensor* handle, u32 idx)	Parameter: u32 idx: Sensor Pad return value: SUCCESS or FAIL Fill Up Sensor IF Info!!! Register set sensor power-on sequence function point.	Function ref: pCus_poweron
handle->pCus_sensor_poweroff (struct __ss_cus_sensor* handle, u32 idx)	Parameter: u32 idx: Sensor Pad return value: SUCCESS or FAIL Register set sensor power-off sequence function point.	Function ref: pCus_poweroff
handle->pCus_sensor_init (struct __ss_cus_sensor* handle, u32 idx)	Parameter: u32 idx: Sensor Pad return value: SUCCESS or FAIL Register load sensor initial setting function point.
handle->pCus_sensor_post_init (struct __ss_cus_sensor* handle, u32 idx)	Parameter: u32 idx: Sensor Pad return value: SUCCESS or FAIL Set and enable SoC mipi csi status.	Special case is like SmartSens product SC4236/SC2315
handle->pCus_sensor_poweroff (struct __ss_cus_sensor* handle, u32 idx)	Parameter: u32 idx: Sensor Pad return value: SUCCESS or FAIL Register set sensor power-off sequence function point.	Function ref: pCus_poweroff
handle->pCus_sensor_suspend (struct __ss_cus_sensor* handle, u32 idx)	Parameter: u32 idx: Sensor Pad return value: SUCCESS or FAIL Register set sensor suspend function point.
handle->pCus_sensor_resume (struct __ss_cus_sensor* handle, u32 idx)	Parameter: u32 idx: Sensor Pad return value: SUCCESS or FAIL Register set sensor resume function point.
handle->pCus_sensor_release (struct __ss_cus_sensor* handle)	Parameter: return value: SUCCESS or FAIL Register sensor release function point.	Function ref: cus_camsensor_release_handle
handle->pCus_sensor_reopen (struct __ss_cus_sensor* handle, u32 idx)	Parameter: u32 idx: Sensor Pad return value: SUCCESS or FAIL Register set sensor reopen function point.	Need to remove, pCus_sensor_str resume replaced
handle->pCus_sensor_streamon (struct __ss_cus_sensor* handle, u32 idx)	Parameter: u32 idx: Sensor Pad return value: SUCCESS or FAIL Register set sensor stream on function point.	In sensor early init mode, this function point can be called when vif sets the streamon.
handle->pCus_sensor_GetVideoResNum(struct __ss_cus_sensor* handle, u32 *ulres_num)	Parameter: u32 ulres_num: Support resolution list number return value: SUCCESS or FAIL Register get sensor support resolution number list number	Function ref: pCus_GetVideoResNum
handle->pCus_sensor_GetVideoRes (struct __ss_cus_sensor* handle, u32 res_idx, cus_camsensor_res **res)	Parameter: u32 res_idx: resolution index cus_camsensor_res: please ref drv_ss_cus_sensor.h return value: SUCCESS or FAIL Register get sensor support resolution list info function point.
handle->pCus_sensor_GetCurVideoRes (struct __ss_cus_sensor* handle, u32 res_idx, cus_camsensor_res **res)	Parameter: u32 res_idx : resolution index cus_camsensor_res : please ref drv_ss_cus_sensor.h return value: SUCCESS or FAIL Register get sensor current resolution info function point.
handle->pCus_sensor_GetCurVideoRes (struct __ss_cus_sensor* handle, u32 res_idx)	Parameter: u32 res_idx : resolution index return value: SUCCESS or FAIL Register set sensor resolution index function point.
handle->pCus_sensor_GetOrien(struct __ss_cus_sensor* handle, CUS_CAMSENSOR_ORIT *ori)	Parameter: CUS_CAMSENSOR_ORIT : please ref drv_ss_cus_sensor.h return value: SUCCESS or FAIL Register get sensor mirror-flip status function.	Function ref: pCus_GetOrien
handle->pCus_sensor_SetOrien(struct __ss_cus_sensor* handle, CUS_CAMSENSOR_ORIT ori)	Parameter: CUS_CAMSENSOR_ORIT : Please ref drv_ss_cus_sensor.h return value: SUCCESS or FAIL Register set Sensor mirror-flip status function.	Function ref: pCus_SetOrien
handle->pCus_sensor_AEStatusNotify(struct __ss_cus_sensor* handle, u32 idx, CUS_CAMERA_AE_STATUS_NOTIFY status)	Parameter: u32 idx: Sensor Pad CUS_CAMERA_AE_STATUS_NOTIFY : Please ref drv_ss_cus_sensor.h return value: SUCCESS or FAIL Register update AE Gain, Shutter and fps status at frame start or frame end.	Function Ref : pCus_AEStatusNotify Status 0 is frame end update Status 1 is frame start update
handle->pCus_sensor_GetAEUSecs(struct __ss_cus_sensor* handle, u32 *us)	Parameter: u32 *us : AE shutter time return value: SUCCESS or FAIL Register get sensor output image exposure time.	Function Ref : pCus_GetAEUSecs
handle->pCus_sensor_SetAEUSecs(struct __ss_cus_sensor* handle, u32 us)	Parameter: u32 us : AE shutter time return value: SUCCESS or FAIL Register set sensor output image exposure time.	Function Ref : pCus_SetAEUSecs
handle->pCus_sensor_GetAEGain(struct __ss_cus_sensor* handle, u32 *gain)	Parameter: u32 *gain : AE gain return value: SUCCESS or FAIL Register get sensor AE Gain.	Function ref: pCus_GetAEGain
handle->pCus_sensor_SetAEGain(struct __ss_cus_sensor* handle, u32 gain)	Parameter: u32 gain : AE gain return value: SUCCESS or FAIL Register set sensor AE Gain.	Function Ref : pCus_SetAEGain
handle->pCus_sensor_TryAEGain(struct __ss_cus_sensor* handle, u32 gain)	Parameter: u32 gain : AE gain return value: SUCCESS or FAIL Register try to get sensor AE Gain
handle->pCus_sensor_TryAEShutter(struct __ss_cus_sensor* handle, u32 us)	Parameter: u32 gain : AE shutter return value: SUCCESS or FAIL Register try to get sensor AE Shutter
handle->pCus_sensor_GetAEInfo(struct __ss_cus_sensor* handle, CUS_SENSOR_AE_INFO_t *info)	Parameter: CUS_SENSOR_AE_INFO_t : Please ref drv_ss_cus_sensor.h return value: SUCCESS or FAIL Register try to get sensor AE(Shutter and Gain) Info
handle->pCus_sensor_GetFPS(struct __ss_cus_sensor* handle)	Parameter: return value: Sensor fps Register get sensor FPS status function.	Function ref: pCus_GetFPS
handle->pCus_sensor_SetFPS(struct __ss_cus_sensor* handle, u32 fps)	Parameter: u32 fps : Sensor fps return value: SUCCESS or FAIL Register set sensor FPS status function.	Function ref: pCus_SetFPS
handle->pCus_sensor_Get_ShutterGainShare(struct __ss_cus_sensor* handle, int eHDRType)	Parameter: int eHDRType : Sensor fps return value: SUCCESS or FAIL Register Get sensor gain and shutter status function.	CUS_HDR_MODE_NONE /No HDR/ CUS_HDR_MODE_SONY_DOL /Sony standard Line information output/ CUS_HDR_MODE_DCG /Dual Gain HDR/ /Embedded 16bit HDR/ CUS_HDR_MODE_COMP /Compress HDR/ CUS_HDR_MODE_LI /Line Interleaved HDR/ CUS_HDR_MODE_COMP_VS /Compress HDR + very short/ CUS_HDR_MODE_VC /Virtual Channel mode/ CUS_HDR_MODE_MAX
handle->pCus_sensor_SetISPConverStatus(struct __ss_cus_sensor* handle, u8 bStatus)	Parameter: u8 bStatus : Sensor AE convergence status return value: SUCCESS or FAIL Register set sensor AE convergence status function.	Used in fast AE convergence mode, it can notify ISP and Sensor AE convergence status

4.2.9 Handle of slave sensor

User can set sensor handle variables to control the signals timing of hsync/vsync/trigger in external slave mode through struct _ms_cus_sensor::slave_mode_attr

Table 13: Handle of sensor slave timing lists

Parameter	Description	NOTE
handle->slave_mode_attr.eSrcSck	Select Hsync clock source	CUS_CMU_CLK_27MHZ, CUS_CMU_CLK_21P6MHZ, CUS_CMU_CLK_12MHZ, CUS_CMU_CLK_5P4MHZ, CUS_CMU_CLK_36MHZ, CUS_CMU_CLK_54MHZ, CUS_CMU_CLK_43P2MHZ, CUS_CMU_CLK_61P7MHZ, CUS_CMU_CLK_72MHZ, CUS_CMU_CLK_48MHZ, CUS_CMU_CLK_24MHZ, CUS_CMU_CLK_37P125MHZ*,
handle->slave_mode_attr.ePolHsync	Hsync initial polarity	CUS_CLK_POL_POS, CUS_CLK_POL_NEG
handle->slave_mode_attr.ePolVsync	Vsync initial polarity	CUS_CLK_POL_POS, CUS_CLK_POL_NEG
handle->slave_mode_attr.ePolXtrig	XTrig initial polarity	CUS_CLK_POL_POS, CUS_CLK_POL_NEG
handle->slave_mode_attr.uHsyncStartT	Hsync start timing = uHsyncStartT * source clock period	range 0 to 65535
handle->slave_mode_attr.uHsyncEndT	Hsync end timing = uHsyncEndT*source clock period	range 0 to 65535
handle->slave_mode_attr.uHsyncPeriod	Hsync period = uHsyncPeriod*source clock period	range 0 to 65535
handle->slave_mode_attr.uVsyncEndT	Vsync low timing = uVsyncEndT*Hsync period	range 0 to 65535
handle->slave_mode_attr.uVsyncPeriod	Vsync period = uVsyncPeriod*Hsync period	range 0 to 65535
handle->slave_mode_attr.uXTrigEndT	XTrigger low timing = uXTrigEndT * source clock period	range 0 to (2^32)-1
handle->slave_mode_attr.uXTrigPeriod	XTrigger period = uXtrigPeriod * source clock period	range 0 to (2^32)-1

​​

Figure 4: Hsync timing in slave mode

​

Figure 5: Vsync timing in slave mode​​

4.2.10 Handle of user define function point

User can set sensor handle variables to control the signals timing of hsync/vsync/trigger in external slave mode through struct _ms_cus_sensor::slave_mode_attr

Table 14: Handle of sensor user define function point list

Parameter	Description	NOTE
handle->pCus_sensor_CustDefineFunction(struct __ss_cus_sensor* handle, u32 cmd_id, void *param)	Parameter: cmd_id : User-defined function items *param : User-defined function parameters return value: SUCCESS or FAIL Register Get sensor gain and shutter status function.	Ref：chapter 5.7

4.2.11 Handle of SensorIF callback

SENSOR_IF API for Connect with the sensor driver and the VIF driver, this callback control in below handle control function.

Set the sensor MCLK configuration

Table 15: Handle of sensorif callback function lists

Parameter	Description	NOTE
sensor_if->PowerOff(u32 idx, CUS_CLK_POL pol);	Parameter: idx : Sensor Pad CUS_CLK_POL : Pull high or pull low Set Sensor-IF PowerDown pull high or not.	Callback to VIF driver
sensor_if->Reset(u32 idx, CUS_CLK_POL pol);	Parameter: idx : Sensor Pad CUS_CLK_POL : Pull high or pull low Set Sensor-IF SW RESET pull high or not.	Callback to VIF driver
sensor_if->MCLK(u32 idx , u8 bon_off, CUS_MCLK_FREQ mclk);	Parameter: idx : Sensor Pad bon_off : Enable or disable CUS_MCLK_FREQ : Sensor clk Register set VIF module output MCLK to sensor For MI Query!!	Main Chip Support MCLK List as : CUS_CMU_CLK_27MHZ, CUS_CMU_CLK_21P6MHZ, CUS_CMU_CLK_12MHZ, CUS_CMU_CLK_5P4MHZ, CUS_CMU_CLK_36MHZ, CUS_CMU_CLK_54MHZ, CUS_CMU_CLK_43P2MHZ, CUS_CMU_CLK_61P7MHZ, CUS_CMU_CLK_72MHZ, CUS_CMU_CLK_48MHZ, CUS_CMU_CLK_24MHZ, CUS_CMU_CLK_37P125MHZ, CUS_CMU_CLK_LPLL_DIV1, CUS_CMU_CLK_LPLL_DIV2, CUS_CMU_CLK_LPLL_DIV4, CUS_CMU_CLK_LPLL_DIV8,
sensor_if->SetIOPad (u32 idx, CUS_SENIF_BUS ulSnrType, NULL)	Parameter: idx : Sensor Pad CUS_SENIF_BUS : Sensor interface For SENSOR-IF I/O Bus Mode	Callback to VIF driver
sensor_if->SetSkipFrame(u32 idx, u16 skip_num, u8 bon_off)	Parameter: idx : Sensor Pad skip_num : skip frame cnt bon_off : Enable or disable Skip vif output frame	Callback to VIF driver

4.2.12 Handle of MIPI CSI configuration callback

SENSOR_IF API for Connect with the sensor driver and the MIPI CSI driver, this callback control in below handle control function.

Table 16: mipi csi callback function lists

Parameter	Description	NOTE
sensor_if->SetCSI_Clk (u32 idx, CUS_CSI_CLK clk)	Parameter: idx : Sensor Pad CUS_CSI_CLK : Mipi clk Set MIPI Interface MAC CLK	Callback to CSI driver
sensor_if->SetCSI_Lane(u32 idx, u16 num_lane, u8 bon_off)	Parameter: idx : Sensor Pad num_lane : Mipi lane number bon_off : Enable or disable Set MIPI Interface Data Lane	Callback to CSI driver
sensor_if->SetCSI_hdr_mode(idx, CUS_HDR_MODE hdr_mode, u8 bon_off)	Parameter: idx : Sensor Pad hdr_mode : bon_off : Enable or disable Set HDR Data Format	Callback to CSI driver CUS_HDR_MODE_NONE /No HDR/ CUS_HDR_MODE_SONY_DOL /Sony standard Line information output/ CUS_HDR_MODE_DCG /Dual Gain HDR/ /Embedded 16bit HDR/ CUS_HDR_MODE_COMP /Compress HDR/ CUS_HDR_MODE_LI /Line Interleaved HDR/ CUS_HDR_MODE_COMP_VS /Compress HDR + very short/ CUS_HDR_MODE_VC /Virtual Channel mode/ CUS_HDR_MODE_MAX
sensor_if->SetCSI_LongPacketType(u32 idx, u16 dt0_15, u16 dt16_31, u16 u32_47)	Parameter: idx : Sensor Pad Type [15:0], [31:16], [47:32] Set MIPI Long Packet Type enable	Callback to CSI driver long packet type enable [0]: Null [1]: blinking [2]: embedded [14]: YUV422_8B [26]: RAW8 [27]: RAW10 [28]: RAW12 [29]: RAW14 [30]: RAW16 [32]: UD1 [33]: UD2 [34]: UD3 [35]: UD4 [36]: UD5 [37]: UD6 [38]: UD7 [39]: UD8
Sensor_if->SetCSI_yuv_order_swap(u32 idx, u8 swap)	Parameter: idx : Sensor Pad swap: Enable or disable Set MIPI Input YUV422 data order swap	Callback to CSI driver

4.2.13 Handle of parallel polarity callback

Set parallel sensor hardware signal polarity

Table 17: parallel sensor callback function list

Parameter	Description	NOTE
Sensor_if-> SetVIF_ParPinPol(u32 idx, CUS_CLK_POL HsynPol, CUS_CLK_POL VsynPol)	Parameter: u32 idx : Sensor Pad HsynPol: Parallel sensor hsync polarity VsynPol: Parallel sensor vsync polarity For sensor vysnc and hsync polarity configure	Callback to VIF driver

4.2.14 Private data

This chapter introduces the sensor driver private data in this part, this purpose is to record the parameters of the sensor without any modification. Here is imx415 as a reference example.

First, the Sgs sensor driver will configure the memory bank when the sensor is installed.

SENSOR_DRV_ENTRY_IMPL_END_EX(  IMX415_HDR,
                              cus_camsensor_init_handle,
                              cus_camsensor_init_handle_hdr_dol_sef,
                              cus_camsensor_init_handle_hdr_dol_lef,
                              imx415_params
                              );

This memory is declared according to the sensor driver, taking imx415 as an example:

​typedef struct {
    struct {
        bool bVideoMode;
        u16 res_idx;
        CUS_CAMSENSOR_ORIT  orit;
    } res;
    struct {
        float sclk;
        u32 hts;
        u32 vts;
        u32 us_per_line;
        u32 final_us;
        u32 final_gain;
        u32 fps;
        u32 preview_fps;
        u32 expo_lines;
        u32 expo_lef_us;
        u32 expo_sef_us;
        u32 expo_sef2_us;
    } expo;
    u32 min_shr1;
    u32 min_rhs1;
    u32 min_shr2;
    u32 min_rhs2;
    u32 min_shr0;
    u32 max_shr0;
    u32 fsc;
    u32 skip_cnt;
    I2C_ARRAY tVts_reg[3];
    I2C_ARRAY tVts_reg_hdr[3];
    I2C_ARRAY tExpo_reg[3];
    I2C_ARRAY tSHR0_reg[3];
    I2C_ARRAY tSHR1_reg[3];
    I2C_ARRAY tRHS1_reg[3];
    I2C_ARRAY tSHR2_reg[3];
    I2C_ARRAY tRHS2_reg[3];
    I2C_ARRAY tGain_reg[2];
    I2C_ARRAY tGain_hdr_dol_lef_reg[2];
    I2C_ARRAY tGain_hdr_dol_sef_reg[2];
    I2C_ARRAY tGain_hdr_dol_sef2_reg[2];
    bool dirty;
    bool orien_dirty;
    u8 snr_pad;
    const I2C_ARRAY *pTable_linear;
    const I2C_ARRAY *pTable_HDR;
    u32 Init_Array_Size;
    u32 Init_Vts;
    u32 Line_Period;
    u32 csi_clk;
    u32 PowerOnOff;
    CUS_CAMSENSOR_ORIT orient;
} imx415_params;

Next, when we mount the sensor register function, we configure the buffer as imx415_params:

​int cus_camsensor_init_handle_linear(ss_cus_sensor* drv_handle)
{
    ss_cus_sensor *handle = drv_handle;
    imx415_params *params;
    int res;

    if (!handle) {
        SENSOR_DMSG("[%s] not enough memory!\n", __FUNCTION__);
        return FAIL;
    }
    SENSOR_DMSG("[%s]", __FUNCTION__);
    ////////////////////////////////////
    // private data allocation & init //
    ////////////////////////////////////
    if (handle->private_data == NULL) {
        SENSOR_EMSG("[%s] Private data is empty!\n", __FUNCTION__);
        return FAIL;
    }

    params = (imx415_params *)handle->private_data;
    CamOsMemcpy(params->tVts_reg, vts_reg, sizeof(vts_reg));
    CamOsMemcpy(params->tGain_reg, gain_reg, sizeof(gain_reg));
    CamOsMemcpy(params->tExpo_reg, expo_reg, sizeof(expo_reg));
…
…

4.2.15 General Sensor Slave Mode Control API

Using general slave mode function struct _ms_cus_sensor::sensor_if_api::SlaveModeCtrl allows users to achieve a more detailed setting to signal timing control. You can find sample codes and more details in drv_ms_cus_imx226_lvds.c

Control ID	Description	NOTE
CUS_SLAVE_MODE_CTRL_ID_INIT	Initialize Hsync and Vsync	CUS_SLAVE_MODE_CTRL_ID_INIT
CUS_SLAVE_MODE_CTRL_ID_DEINIT	Free and disable Hsync and Vsync	CUS_SLAVE_MODE_CTRL_ID_DEINIT
CUS_SLAVE_MODE_CTRL_ID_ENABLE_XVS	Enable or disable Vsync immediately	CUS_SLAVE_MODE_CTRL_ID_ENABLE_XVS

5. SOFTWARE PORTING FLOW

This chapter introduces the process of software porting sensor driver. First, it will explain how to initialize the sensor. Then describe the information required by the sensor driver 3A and control the functions that 3A will call.

5.1. The sensor driver initial flow introduction

​

Figure 6: porting the sensor driver flow

This document introduces the initial process of transplanting the sensor according to the process in the figure above.

5.1.1 Set the sensor information setting

These settings are the hardware information of the sensor, we have listed the necessary hardware information, you can refer to the IMX415 / IMX335 sensor driver.

////////////////////////////////////
// Sensor-If Info                 //
////////////////////////////////////
//MIPI config begin.
#define SENSOR_MIPI_LANE_NUM        (lane_num)          //IMX335 Linear mode supports MIPI 2/4 Lane
#define SENSOR_MIPI_LANE_NUM_DOL    (4) //(hdr_lane_num)//IMX335 DOL mode supports MIPI 4 Lane
#define SENSOR_IFBUS_TYPE           CUS_SENIF_BUS_MIPI  //CFG //CUS_SENIF_BUS_PARL, CUS_SENIF_BUS_MIPI
#define SENSOR_DATAPREC             CUS_DATAPRECISION_10
#define SENSOR_DATAPREC_DOL         CUS_DATAPRECISION_10
#define SENSOR_BAYERID              CUS_BAYER_RG         //0h: CUS_BAYER_RG, 1h: CUS_BAYER_GR, 2h: CUS_BAYER_BG, 3h: CUS_BAYER_GB
#define SENSOR_BAYERID_HDR_DOL     CUS_BAYER_RG
#define SENSOR_RGBIRID              CUS_RGBIR_NONE
CUS_ORIT_M1F1,
//#define SENSOR_YCORDER              CUS_SEN_YCODR_YC     //CUS_SEN_YCODR_YC, CUS_SEN_YCODR_CY
//#define long_packet_type_enable     0x00 //UD1~UD8 (user define)

////////////////////////////////////
// MCLK Info                        //
////////////////////////////////////
#define Preview_MCLK_SPEED          CUS_CMU_CLK_27MHZ//CUS_CMU_CLK_37P125MHZ//CUS_CMU_CLK_27MHZ
#define Preview_MCLK_SPEED_HDR_DOL  CUS_CMU_CLK_27MHZ

5.1.2 Set the sensor I2C setting

SoC communicates with sensors mainly through I2C. Therefore, you need to set I2C before sending sensor setting value to the sensor.

Please refer to this document chapter 1.2.

5.1.3 Set support image resolution information

You need to fill in the sensor image information provided by the sensor provider FAE. This information includes preview size, image crop size and site etc. We define this information as a global variable.

////////////////////////////////////
// Image Info                     //
////////////////////////////////////
static struct {     // LINEAR
    // Modify it based on number of support resolution
    enum {LINEAR_RES_1 = 0, LINEAR_RES_2, LINEAR_RES_3, LINEAR_RES_4, LINEAR_RES_5, LINEAR_RES_6, LINEAR_RES_7, LINEAR_RES_8, LINEAR_RES_END}mode;
    // Sensor Output Image info
    struct _senout{
        s32 width, height, min_fps, max_fps;
    }senout;
    // VIF Get Image Info
    struct _sensif{
        s32 crop_start_X, crop_start_y, preview_w, preview_h;
    }senif;
    // Show resolution string
    struct _senstr{
        const char* strResDesc;
    }senstr;
}imx415_mipi_linear[] = {
    {LINEAR_RES_1, {3860, 2250, 3, 20}, {0, 0, 3840, 2160}, {"3840x2160@20fps"}},
    {LINEAR_RES_2, {3096, 2190, 3, 30}, {0, 0, 3072, 2048}, {"3072x2048@30fps"}}, // Modify it
    {LINEAR_RES_3, {3096, 1758, 3, 30}, {0, 0, 3072, 1728}, {"3072x1728@30fps"}}, // Modify it
    {LINEAR_RES_4, {2616, 1974, 3, 30}, {0, 0, 2592, 1944}, {"2592x1944@30fps"}}, // Modify it
    {LINEAR_RES_5, {2976, 1686, 3, 30}, {0, 0, 2944, 1656}, {"2944x1656@30fps"}}, // Modify it
    {LINEAR_RES_6, {2592, 1470, 3, 30}, {0, 0, 2560, 1440}, {"2560x1440@30fps"}}, // Modify it
    {LINEAR_RES_7, {1920, 1080, 3, 60}, {0, 0, 1920, 1080}, {"1920x1080@60fps"}}, // Modify it
    {LINEAR_RES_8, {3864, 2192, 3, 30}, {0, 0, 3840, 2160}, {"3840x2160@30fps"}}, // Modify it
    //{LINEAR_RES_9, {2616, 1974, 3, 25}, {0, 0, 2592, 1944}, {"2592x1944@25fps"}}, // Modify it
};

5.1.4 The sensor power on sequence function

Please refer to the function registered by “handle->pCus_sensor_poweron”.

5.1.5 The sensor initial function

Here is the initial setting of the sensor through the i2c device, first read the product ID or sensor ID of the sensor device to ensure that the correct sensor is used. Please refer to the function registered by “handle->pCus_sensor_init”.

5.1.6 Select the sensor output resolution function

MI will get the image resolution information supported by the sensor driver and select one of the sensor's initial settings.

Please refer to the function registered by “handle->pCus_sensor_SetVideoRes”.

5.1.7 The sensor power off sequence function

Please refer to the function registered by “handle->pCus_sensor_poweroff”.

5.2. Implement the sensor control function

The functions that the sensor driver needs to support include fps adjustment, AE shutter adjustment, AE gain adjustment, and mirror flip function settings, which we will not describe more detail, because this needs to be implemented according to the characteristics of each sensor.

The following is our control the sensor implement function flow. Relatively, we also need to get the sensor information, please refer to the figure below:

​​​

Figure 7: control the sensor implement function flow​​

​​​

Figure 8: get the sensor implement function information or status​​

5.2.1 AE shutter control and get AE shutter information

How to control AE shutter, please refer to the function registered by “handle->pCus_sensor_SetAEUSecs”.

Also, ISP algo or MI need to get AE shutter information, please refer to the function registered by “handle->pCus_sensor_GetAEInfo”.

5.2.2 AE gain control and get AE gain information

How to control AE gain, please refer to the function registered by “handle->pCus_sensor_SetAEGain”.

Also, ISP algo or MI need to get AE gain information, please refer to the function registered by “handle->pCus_sensor_GetAEInfo”

More detail please ref 5.2.6 and 5.2.7.

5.2.3 Sensor frame rate control and get sensor frame rate information

How to control the sensor output frame rate, please refer to the function registered by “handle->pCus_sensor_SetFPS”.

Also, MI needs to get the sensor output frame rate information, please refer to the function registered by “handle->pCus_sensor_GetFPS”.

5.2.4 Sensor mirror-flip control and get sensor mirror-flip information

How to control the sensor image mirror-flip, please refer to the function registered by “handle->pCus_sensor_SetOrien”.

Also, MI needs to get the sensor image mirror-flip information, please refer to the function registered by “handle->pCus_sensor_GetOrien”.

5.2.5 I2C execution time control

The above AE shutter, AE gain, sensor frame rate and mirror control, we hope everyone can send commands to the sensor through I2C at the same time

So we will implement this using the function "handle->pCus_sensor_AEStatusNotify"

Therefore, we can control the time of I2C send commands to the sensor through two parameters "CUS_FRAME_ACTIVE" and "CUS_FRAME_INACTIVE". Most sensors can be set at the frame start, so the parameter "CUS_FRAME_ACTIVE" can be selected.

​​​

Figure 9: I2C execution time​​

NOTE:

If the porting sensor cannot find out what caused the image abnormality, you can call “return success” directly in the function.

5.2.6 Sensor AE shutter/gain delay count

How to set “handle->pCus_sensor_GetAEInfo” of “ae shutter delay” and “ae gain delay”, please refer to this document chapter 4.3.

5.2.7 Sensor AE shutter/gain ctrl num

“handle->pCus_sensor_GetAEInfo” of “ae gain ctrl num” and “ae shutter ctrl num” represents ISP that can control the number of AE streams

Get AE Info sample code:

static int pCus_sensor_GetAEInfo(ss_cus_sensor *handle, CUS_SENSOR_AE_INFO_t *info)
{
    imx415_params *params = (imx415_params *)handle->private_data;

    info->u8AEGainDelay                      = SENSOR_GAIN_DELAY_FRAME_COUNT;
    info->u8AEShutterDelay                   = SENSOR_SHUTTER_DELAY_FRAME_COUNT;
    info->u8AEGainCtrlNum                    = SENSOR_GAIN_CTRL_NUM;
    info->u8AEShutterCtrlNum                 = SENSOR_SHUTTER_CTRL_NUM;
    info->u32AEGain_min                      = SENSOR_MIN_GAIN;
    info->u32AEGain_max                      = SENSOR_MAX_GAIN;
    info->u32AEShutter_min                   = params->Line_Period * 1;
    info->u32AEShutter_max                   = 1000000000/handle->video_res_supported.res[handle->video_res_supported.ulcur_res].u16min_fps;
    info->u32AEShutter_step                  = params->Line_Period;
    return SUCCESS;
}

6. SUPPLYMENT

6.1. Install the Sensor Driver and parameter introduce

Sgs sensor driver provides several installation parameters that can be used when installing the sensor driver. Here are the parameters and instructions as below:

Table 17: sensor driver install parameter

Parameter	Description
chmap	Specify SoC Channel pad for sensor, expressed in bitmap format, Bit[0] represents sensor 0, Bit[1] represents sensor 1, and so on.
mclk	Specify sensor mclk info
lane_num	Specify Sensor linear mode MIPI lane number
hdr_lane_num	Specify Sensor hdr mode MIPI lane number
i2c_slave_id	Specify Sensor i2c slave id

For example:

insmod imx335_MIPI.ko chmap=1 lane_num=4 hdr_lane_num=4 i2c_slave_id=0x20

If the above parameters are not used when installing the sensor driver, the default values in the sensor driver are used.

6.2. Verify shutter and gain delay

If you want to verify that the sensor AE gain/shutter delay, you can use apitool, also follow the steps below to control the sensor AE gain/shutter function step by step.

STEP 1. You need to execute process of prog_vif_isp_ut first

/customer # ./prog_vif_isp_ut param_snr0.ini

STEP 2. Open IQ tool in sdk path “project/tools/apitool”

​

STEP 3. Host Name is EVB’s IP address, check it then connect to EVB

​

STEP 4. Select Macaron AE option, and select “ExposureMode” menu table to M_Mode

​

Step 5. Adjust “DebugLev” menu and set AE Debug Level from 0 to 64, then you can check your console log

​

Check the AE message shown on the console

​

Step 6. Adjust AE sensor gain or shutter manually. At the same time, you can observe the AE message shown on the console.

​​

Observer the value of “CurYx10” change from 308 to 1436

​​

After we set the gain of AE through ISP tool, it will take effect in the next frame. The CurYx10 represents the brightness information of a previous frame. The following figure is an example. The AE gain delay needs to set 2.

​​​​

Figure 10: AE Y value response time​​

6.3. Verify shutter and gain linearity

If you want to verify that the sensor AE gain/shutter function is normal after porting the sensor AE gain/shutter function finish, you can use apitool. You can follow the steps below to control the sensor AE gain/shutter function step by step.

STEP 1. You need to execute process of mixer

/customer # ./prog_vif_isp_ut param_snr0.ini

STEP 2. Open IQ tool in sdk path “project/tools/apitool”

​​

STEP 3. Host Name is EVB’s IP address, check it then connect to EVB

​​​

STEP 4. Select Macaron AE option, and select “ExposureMode” menu table to M_Mode

​​​

Step 5. Adjust AE sensor gain and shutter manually. At the same time, you can observe image changes through VLC or your debug message.

​​​

The IQ Tool has defined upper and lower limit for the AE settings, you can adjust it according to your needs. You can reference Exposure limit table.

​​​​

6.4. Skip frame feature

The sensor driver that has been implemented has imx307. You can refer to the sensor driver of imx307. Here I will briefly explain how to use skip frame function.

Step 1. Add a variable "skip _cnt" inside Params structure of sensor driver

    u32 max_rhs1;
    u32 lef_shs2;
    u32 skip_cnt;
    bool dirty;
    bool change;
    I2C_ARRAY tVts_reg[3];
    I2C_ARRAY tGain_reg[3];
    I2C_ARRAY tExpo_reg[3];
    I2C_ARRAY tShs2_reg[3];
    I2C_ARRAY tRhs1_reg[3];
    I2C_ARRAY tGain_hdr_dol_lef_reg[1];
    I2C_ARRAY tGain_hdr_dol_sef_reg[1];
    bool orien_dirty;
} imx307_params;

Step 2. Set the number of skip frames in the function that needs to skip frames.

For an example of imx307, in HDR mode, it may flicker when changing fps (from 10 fps to 30 fps) in high brightness environment, so skip 1 frame to avoid flicker.

if(params->expo.expo_lines > 2 * params->expo.vts - params->max_rhs1 -3){
    vts = (params->expo.expo_lines + params->max_rhs1 + 3 + 1) / 2;
}else{
    vts = params->expo.vts;
}

params->expo.vts = vts;
pCus_SetAEUSecsHDR_DOL_SEF1(handle, params->expo.expo_sef_us);
params->skip_cnt = 1;
return SUCCESS;

Step 3. The actual time to skip frame in function of “AESStatusNotify”. Please add the following features. SensorIF will calculate the time to set the VIF mask disable based on the number of skip frames and the current fps time period.

        if(params->skip_cnt){
            sensor_if->SetSkipFrame(handle->snr_pad_group, params->expo.fps, params->skip_cnt);
            params->skip_cnt = 0;
        }
        break;
    default :
            break;
}
return SUCCESS;

6.5. Sensor post initial feature (Special case for SmartSens)

Since the Smartsens sensor first transmit i2c cmd after power-on, the probability of the sensor mipi csi and clock will be pulled down, it leads to SoC mipi csi abnormal.

​​​​​​

Figure 11: Smartsense sensor signal​​

Currently, the solution is to change the sequence of the sensor power-on and the sensor initialization. After the sensor initialize completed, then enable SoC mipi csi to resolve this problem. Please refer to the following process:

​​​​​​

Figure 12: power on sequence flow change​​

NOTE:

There are other ways to solve this problem. Please ask the Sensor manufacturer to provide mipi csi non-continuous mode, which can solve the problem of mipi csi getting stuck due to the abnormal receiving status of the first frame of mipi csi.

6.6. Parallel Sensor Configuration

Compared with MIPI Sensor, Parallel sensor needs to confirm signal polarity and hardware pin arrangement. The following will explain how to set Signal polarity and hardware Pin arrangement.

6.6.1 Parallel sensor signal polarity

Generally, parallel sensors usually need to configure 3 kinds of clock signals

Table 18: Parallel sensor signal type

Signal Name	Abbreviation	Description
Vertical Sync	VSYNC	Frame Start (FS)
Horizontal Sync	HSYNC	Line Start (LS)
Pixel Clock	PCLK	Carry Pixel data

The Frame Start (FS) signal is generated upon the rising or falling edge of the VSYNC. The Line Start (LS) signal is generated upon the rising or falling edge of the HSYNC, and PCLK is internally AND-ed to provide a valid pixel clock signal. VSYNC and HSYNC can be active high or active low. Data is latched into the FIFO on every valid pixel clock edge.

​​​​

Figure 13: Parallel sensor signal​​

We Provide an APIs at Sensor driver to configure parallel sensor vsync and hsync polarity. It need to configure polarity based on sensor init setting. Ref:

int (*SetVIF_ParPinPol) (u32 idx, CUS_CLK_POL HsyncPol, CUS_CLK_POL VsyncPol);

static int pCus_poweron(ss_cus_sensor *handle, u32 idx)
{
    ISensorIfAPI *sensor_if = handle->sensor_if_api;

    //Sensor power on sequence
    sensor_if->PowerOff(idx, CUS_CLK_POL_NEG);   // Powerdn Pull Low
    sensor_if->Reset(idx, CUS_CLK_POL_NEG);     // Rst Pull Low
    sensor_if->SetVIF_ParPinPol(idx, CUS_CLK_POL_NEG);     // Rst Pull Low
    sensor_if->SetIOPad(idx, handle->sif_bus, CUS_CLK_POL_NEG, CUS_CLK_POL_NEG);

6.6.2 Parallel sensor pad arrangement

The parallel sensor configuration needs to determine the sensor format based on the sensor pad arrangement, rather than using the sensor data precision

Example 1 (8-bit Sensors):

Parallel padmux mode provides 2 sets of settings, and the placement of 8-bit Sensors is as follows: red frame/blue frame/orange frame. Different placement positions on the hardware, the set sensor format will be different.

​​

Figure 14: 8bit parallel sensor placement

Example 2 (10-bit Sensors):

Parallel padmux mode provides 2 sets of settings, and the placement of 10-bit Sensors is as follows: blue frame/orange frame. Different placement positions on the hardware, the set sensor format will be different.

​​​​​

Figure 15: 10bit parallel sensor placement

Always, VIF Configure sensor format is 12-bit, it is recommended that the hardware parallel data pins are arranged in the MSB direction. If not, then we provide the MSB shift parameter set in DTS file “vif_sr0_par_msb_shift=shift_pin”

Hardware limitation: shift_pin can only be 2/4/6

6.7. Customer function

If users want to implement customized functions for the sensor driver or obtain Sensor information, Sgs sensor driver provides a register function point for customers to develop according to their needs, customers can use this Functions to define their own content :

​​

Figure 16: user define control flow

user define implementation in sensor driver:

#define CMDID_I2C_READ   (0x01)
#define CMDID_I2C_WRITE  (0x02)

static int pCus_sensor_CustDefineFunction(ss_cus_sensor* handle,u32 cmd_id, void *param) {

    if(param == NULL || handle == NULL)
    {
        SENSOR_EMSG("param/handle data NULL \n");
        return FAIL;
    }

    switch(cmd_id)
    {
        case CMDID_I2C_READ:
        {
            I2C_ARRAY *reg = (I2C_ARRAY *)param;
            SensorReg_Read(reg->reg, &reg->data);
            SENSOR_EMSG("reg %x, read data %x \n", reg->reg, reg->data);
            break;
        }
        case CMDID_I2C_WRITE:
        {
            I2C_ARRAY *reg = (I2C_ARRAY *)param;
            SENSOR_EMSG("reg %x, write data %x \n", reg->reg, reg->data);
            SensorReg_Write(reg->reg, reg->data);
            break;
        }
        default:
            SENSOR_EMSG("cmd id %d err \n", cmd_id);
            break;
    }

    return SUCCESS;
}

7. TROUBLE SHOOTING

7.1. SONY sensor output format

There are two mipi formats output by Sony sensors, namely DOL or VC. Sgs's mipi csi will receive data based on the settings comparison header file, so please confirm the current sensor init settings.

​​​​

7.2. VIF status

At the same time you can open telnet and enter the following cmd to confirm the VIF and ISP status.

Usually 30fps means that a frame period is 33.3ms.

/ # cat /proc/mi_modules/mi_vif/debug_hal/vif0/vif_ints
== VIF0 INTS @:58129759 ==
== VIF CH-0 ==
CH State:                       : WORKING
Interval(us)                    : 41672
FramePeriod(us)                 : 39057
VREF_FALLING                    : 691 @:58099735
LINE_CNT_0                      : 0 @:0
LINE_CNT_1                      : 0 @:0
TOTAL_PIX_CNT                   : 670 @:58097120

== VIF CH-1 ==
CH State:                       : WORKING
Interval(us)                    : 41670
FramePeriod(us)                 : 39041
VREF_FALLING                    : 691 @:58102994
LINE_CNT_0                      : 0 @:0
LINE_CNT_1                      : 0 @:0
TOTAL_PIX_CNT                   : 670 @:58100365

== VIF FIFO-0 ==
Aff State:                      : WORKING
AFF Count:                      : 0 @:0

== VIF FIFO-1 ==
Aff State:                      : WORKING
AFF Count:                      : 0 @:0

== VIF FIFO-2 ==
Aff State:                      : READY
AFF Count:                      : 0 @:0

== VIF FIFO-3 ==
Aff State:                      : READY
AFF Count:                      : 0 @:0

Confirm the information of the sensor signal received by VIF device

/ # cat /proc/mi_modules/mi_vif/debug_hal/vif0/vif_info
== VIF0 INFO @:49165780 ==
== VIF CH-0 ==
CH State:: WORKING
[vg:0 link:REAL_ISP]--[chn:0]--[afifo:1]--[pipe:1(1)]--[isp:1]--[end]
CH_EN: 1
SRC: MIPI0
INPUT_FMT: RGB
PIX_FMT: 10 bits
CROP_EN: 1
CROP_START_X: 0 - 3839
CROP_START_Y: 0 - 2159
PIXEL_CNT: 3840
LINE_CNT: 2160
TOTAL_PIXEL_CNT: 3864
TOTAL_LINE_CNT: 2192

== VIF CH-1 ==
CH State:: WORKING
[vg:0 link:REAL_ISP]--[chn:1]--[afifo:0]--[pipe:0(0)]--[isp:0]--[end]
CH_EN: 1
SRC: MIPI0
INPUT_FMT: RGB
PIX_FMT: 10 bits
CROP_EN: 1
CROP_START_X: 0 - 3839
CROP_START_Y: 0 - 2159
PIXEL_CNT: 3840
LINE_CNT: 2160
TOTAL_PIXEL_CNT: 3864
TOTAL_LINE_CNT: 2192

7.3. MIPI status

If the transmission speed of one line or pixel per frame is too fast, even if the VIF can receive the image data normally, there is no guarantee that the ISP can process it immediately, it causes isp fifo full.

/ # echo 0 0 > /proc/mi_modules/mi_sensor/debug_hal/csi_dbg_mask
/ # cat /proc/mi_modules/mi_sensor/debug_hal/csi_ints

NOTE:

echo “sensor pad” “mask_status” > /proc/mi_modules/mi_sensor/debug_hal/csi_dbg_mask

== MIPI pad0 err ints==
DT      : 22569
PA_LENS : 0
PH_LENS : 0
ECC_ONEBIT      : 0
FRAME_START     : 0
FRAME_END       : 0
ECC_TWOBIT      : 0
CRC     : 0
PA_WC_EQ0       : 0
RAW10_LENS      : 0
CON_FE  : 0
CON_FS  : 0
LE      : 0
LS      : 0
OVERRUN : 0
== MIPI pad0 rtp ints==
LINE_NUM        : 0
FRAME_NUM       : 0
SHORT_PKT       : 0
VC3_DONE        : 0
VC2_DONE        : 0
VC1_DONE        : 0
VC0_DONE        : 0


== MIPI pad1 err ints==
DT      : 75
PA_LENS : 0
PH_LENS : 0
ECC_ONEBIT      : 0
FRAME_START     : 0
FRAME_END       : 0
ECC_TWOBIT      : 0
CRC     : 0
PA_WC_EQ0       : 0
RAW10_LENS      : 0
CON_FE  : 0
CON_FS  : 0
LE      : 0
LS      : 0
OVERRUN : 0
== MIPI pad1 rtp ints==
LINE_NUM        : 0
FRAME_NUM       : 0
SHORT_PKT       : 0
VC3_DONE        : 0
VC2_DONE        : 0
VC1_DONE        : 0
VC0_DONE        : 0


== MIPI pad2 err ints==
DT      : 0
PA_LENS : 0
PH_LENS : 0
ECC_ONEBIT      : 0
FRAME_START     : 0
FRAME_END       : 0
ECC_TWOBIT      : 0
CRC     : 0
PA_WC_EQ0       : 0
RAW10_LENS      : 0
CON_FE  : 0
CON_FS  : 0
LE      : 0
LS      : 0
OVERRUN : 0
== MIPI pad2 rtp ints==
LINE_NUM        : 0
FRAME_NUM       : 0
SHORT_PKT       : 0
VC3_DONE        : 0
VC2_DONE        : 0
VC1_DONE        : 0
VC0_DONE        : 0

​

Mipi csi interrupt error message

• OVERRUN: lane clock domain to mpll clock domain asyn fifo overrun
• LS: received line end but lost line start error
• LE: received line start but lost line end error
• CON_FS: no frame end between 2 cosecutive frame start
• CON_FE: no frame start between 2 consecutive frame end
• RAW10_LENS: tie 0 (not used)
• PA_WC_EQ0: payload word count in packet header eqaul to 0
• CRC: packet payload CRC check result error
• ECC_TWOBIT: packet header ECC check result has 2 bit error
• FRAME_END: no frame end packet after a frame start packet packet
• FRAME_START (hardware auto-ignore): received a normal data packet but no frame start packet before it
• ECC_ONIVIT: packet header ECC check result has 1 bit error
• PH_LENS: packet header lengths error
• PA_LENS: payload lenghts error shorten than wc
• DT: reserved and not supported data type