[data grid] What is the principle behind mui's server-side lazy loading?

[idoall]

Steps to reproduce

Steps:

1. DataGrid Args:

<div style={{maxHeight:'calc(100vh - 250px)',minHeight:'200px',display: "flex",flexDirection: "column"}}>
<DataGrid
      apiRef={apiRef}
      rowCount={rowCount}
      dataSource={dataSource}
      lazyLoadingRequestThrottleMs={1000}
      lazyLoading
      paginationModel={{ page: 0, pageSize: 10 }}
      loading={pageState.loading}
      showToolbar
      rowHeight={rowHeight}
      columns={columns}
      listViewColumn={listViewColumn}
      listView={isMobile}
      getRowId={(row: modelItemSchemaType) => row.Id || 0}
      sx={{
        [`& .${gridClasses.columnHeader}:focus-within, & .${gridClasses.cell}:focus-within`]: {
          outline: "none"
        },
        "--DataGrid-overlayHeight": "200px"
      }}
      initialState={{
        pinnedColumns: {
          left: ["Id", "ExchangeName"],
          right: ["Time", "CreateTime"]
        },
        sorting: {
          sortModel: [{ field: "CreateTime", sort: "desc" }]
        }
      }}
/>
</div>

Current behavior

1. My DataGrid is set to display only 10 rows per page. When I output params in dataSource.getRows, the pageSize value in it inexplicably becomes 100.

The first time the page loads, the output params parameters are as follows:

{
    "groupKeys": [],
    "paginationModel": {
        "page": 0,
        "pageSize": 100
    },
    "sortModel": [
        {
            "field": "CreateTime",
            "sort": "desc"
        }
    ],
    "filterModel": {
        "items": [],
        "logicOperator": "and",
        "quickFilterValues": [],
        "quickFilterLogicOperator": "and"
    },
    "start": 0,
    "end": 99,
    "groupFields": [],
    "aggregationModel": {}
}

In getRows, the program retrieves data from the server based on start and end, returning rows for the first 100 rows, and rowCount for 3931. At this point, the footer will display Total Rows: 3931.

2、When scrolling down to the 90th row again, the server is triggered again with the following parameters:

{
    "groupKeys": [],
    "paginationModel": {
        "page": 0,
        "pageSize": 100
    },
    "sortModel": [
        {
            "field": "CreateTime",
            "sort": "desc"
        }
    ],
    "filterModel": {
        "items": [],
        "logicOperator": "and",
        "quickFilterValues": [],
        "quickFilterLogicOperator": "and"
    },
    "start": 100,
    "end": 199,
    "groupFields": [],
    "aggregationModel": {}
}

The server will return rows 100-200 from the rows list, with a rowCount of 3931. At this point, the footer will display Total Rows: 200/3931.

3. Once you scroll down to 200 items in the viewport area, you can no longer scroll down.

Expected behavior

The rowCount has already been set, and the returned value is 3931.

The viewport should continue scrolling if not all data has been loaded, so why does it stop loading data before reaching the 200th row?

Context

No response

Your environment

npx @mui/envinfo

  Don't forget to mention which browser you used.
  Output from `npx @mui/envinfo` goes here.

Search keywords: lazy-loading

[idoall]

If you set the rowCount property of the DataGrid to -1, you can load data beyond 200 rows.

[arminmeh]

@idoall
Can you share a reproduction demo?

I have checked our lazy loading demo and changing pageSize in the paginationModel works as expected. The initial request is made with start = 0 and end = 9 params. If I change the pageSize to 50, I see that the request's end parameter becomes 49.

Setting rowCount to -1 makes the DataGrid work in Infinite loading mode.

Also, you don't have to use the rowCount prop. You can pass the rowCount with your server response to the dataSource.

[arminmeh]

One additional question.
Since you say that the grid does not scroll after 200 rows are loaded, do you see initially that the scroller size indicates that the grid has filled the scrolling area with skeleton rows?

[idoall]

The issue of paginationModel not taking effect has been identified. It was because I had encapsulated the DataGrid and it had the paginationModel property, but it wasn't being passed to the DataGrid. After fixing this issue, the number of pages loaded is now normal.

[idoall]

One additional question.
Since you say that the grid does not scroll after 200 rows are loaded, do you see initially that the scroller size indicates that the grid has filled the scrolling area with skeleton rows?

As shown in the image above, the initial total number of rows was 4000. In this test, I set paginationModel={{ page: 0, pageSize: 50 }}. Judging from the size of the scrollbar in the viewport area, there is a lot of content below, which should be filled with a skeleton. At this time, the footer displays all 4000 rows.

When I scroll down to item 39, it requests data from the server for the second page, then the scrollbar zooms in, the initial filled skeleton is cleared, and it stops loading data when I scroll to row 100, No more requests will be sent to the server from the dataSource. as shown in the image above.

Here are my DataGrid parameters:

  const rowCount = React.useMemo((): number => {
    return pageState.totalRowCount;
  }, [pageState.totalRowCount]);

  if (isDev) {
    console.log("pageState", pageState);
    console.log("rowCount for DataGrid:", rowCount);
  }
<Responsive__DataGrid
        apiRef={apiRef}
        rowCount={rowCount}
        dataSource={dataSource}
        lazyLoadingRequestThrottleMs={1000}
        lazyLoading
        loading={pageState.loading}
        showToolbar
        rowHeight={rowHeight}
        columns={columns}
        listViewColumn={listViewColumn}
        listView={isMobile}
        getRowId={(row: modelItemSchemaType) => row.Id || 0}
        paginationModel={{ page: 0, pageSize: 50 }}
        gridBoxSx={{
          display: "flex",
          flexDirection: "column",
          maxHeight,
          minHeight,
          mr: "1rem",
          ml: "1rem"
        }}
        sx={{
          [`& .${gridClasses.columnHeader}:focus-within, & .${gridClasses.cell}:focus-within`]: {
            outline: "none"
          },
          "--DataGrid-overlayHeight": "200px"
        }}
        initialState={{
          pinnedColumns: {
            left: ["Id", "Name"],
            right: ["Time", "CreateTime"]
          },
          sorting: {
            sortModel: [{ field: "CreateTime", sort: "desc" }]
          }
        }}
      />

Below is the code for dataSource:

const dataSource: GridDataSource = React.useMemo(
      () => ({
        getRows: async (params: GridGetRowsParams) => {
          if (pageState.exchangeSchema.Text == "" || pageState.assetTypeSchema.Value == AssetType.Empty) {
            return {
              rows: [],
              rowCount: 0
            };
          }

          console.log("params", params);
          setPageState(prev => ({ ...prev, loading: true }));

          const sorting = params.sortModel.map(item => ({
            field: item.field,
            sort: item.sort
          })) as SortingListSchemaType;

          const limit = Number(params.end) - Number(params.start) + 1; 
          const offset = Number(params.start);

     
          const filterBaseModel = transformFilterModelToQuery(params.filterModel);
          if (!filterBaseModel) {
            console.error(`[${hookName}-dataSource->transformFilterModelToQuery Error`);

            setPageState(prev => ({ ...prev, loading: false }));
            return {
              rows: [],
              rowCount: 0
            };
          }

          const rawData = {
            ExchangeName: pageState.exchangeSchema.Text,
            AssetType: pageState.assetTypeSchema.Text as AssetType,
            Offset: offset,
            Limit: limit,
            Sorting: pageState.sorting ? pageState.sorting : sorting,
            FilterBaseModel: {
              filterModels: [
                ...(baseFilterModel ? [baseFilterModel] : []),
                ...pageState.defaultFilters,
                ...filterBaseModel.filterModels 
              ]
            }
          };

          const result = BaseRequestCandleStrategyBaseRow.safeParse(rawData);
          if (!result.success) {
            const prettyError = parseZodErrorMessage(result.error);
            console.error(`[${hookName}-dataSource->BaseRowRequestSchema2 prettyError`, prettyError, rawData, params);

            setPageState(prev => ({ ...prev, loading: false, totalRowCount: 0, sellTotal: 0, buyTotal: 0 }));
            return {
              rows: [],
              rowCount: 0
            };
          }
          try {
            const postData = result.data;
            const data = await service.QueryList({
              postData
            });
            if (data.Code == 200 && data.Data) {
              const rowCount = data.TotalCount || 0;

              const newList = data.Data.map(item => {
                const id = autoIdRef.current;
                autoIdRef.current += 1;
                return {
                  ...item,
                  Id: id.toString()
                };
              });

              setPageState(prev => ({
                ...prev,
                loading: false,
                totalRowCount: rowCount,
                sellTotal: data.BuyTotal ?? 0,
                buyTotal: data.SellTotal ?? 0
              }));

              const resultObject = {
                rows: newList,
                rowCount: rowCount
              };

              console.log("resultObject", resultObject);

              return resultObject;
            }

            notifyError(`[${hookName}-${data.Code}]${data.Message}`);

            setPageState(prev => ({ ...prev, loading: false, totalRowCount: 0, sellTotal: 0, buyTotal: 0 }));
            return {
              rows: [],
              rowCount: 0
            };
          } catch (error) {
            console.error("dataSource Load Data Error：", error);
            notifyError(`[${hookName}]${JSON.stringify(error)}`);

            setPageState(prev => ({ ...prev, loading: false, totalRowCount: 0, sellTotal: 0, buyTotal: 0 }));
            return {
              rows: [],
              rowCount: 0
            };
          }
        },
        updateRow: async (params: GridUpdateRowParams) => {
          // const syncedRow = await editRow(params.rowId, params.updatedRow);
          // return syncedRow;
          console.log("updateRow->params", params);
          return params.updatedRow;
        }
      }),
      [pageState.defaultFilters, baseFilterModel, pageState.assetTypeSchema, pageState.exchangeSchema]
    );

[arminmeh]

I am still missing your pageState implementation.
Can you add more code so that I can build the reproduction demo?

Is there a specific reason why you are passing rowCount and loading props to the grid?
With lazy loading, the loading state will be changed for you (while waiting for the response), and will use the rowCount prop from the getRows() response to set the new row count.

If you don't pass those for some specific reasons, you can remove them.