Fetching Hierarchical data for a tree component

[cmark]

Hi All,

Is it possible to use react-query for hierarchical data?
My data structure is something like this:

interface Items {
  items: Array<Item>
  after: string
  limit: integer
  total: integer
}
interface Item {
  id: string
  parents: Array<string>
}

The API has a single endpoint GET /items to retrieve a list of items along with a cursor (after), the number of items requested limit and the total number of items that match the query.
The endpoint allows filtering by id, parent and any other property Item has.

Basically what I'd like to achieve is to replace the current Fetch API solution to use react-query instead. The content will be rendered in an Ant.Design Tree, rc-tree, react-virtualized-tree or equivalent.

To even simplify the idea and remove the hierarchical nature, I think this can be achieved by using multiple useInfiniteQuery hooks (one hook per parent item list, queryKey is the parentId and the after value is the cursor), but unfortunately dynamic number of hooks is not allowed in React (unless I'm missing something).

Any help will be much appreciated,
Mark

[cmark]

Apologize for the long story, but I'd like to show my findings, so future adventurers might find it useful when they are trying to use react-query for hierarchical data.

Initially I have created something like this:

  ...
  // fetch config is basically the parentId and a cursor in an object { parentId, searchAfter }
  const [fetchConfig, setFetchConfig] = useState(initialFetchConfig)
  const previousFetchConfig = usePrevious(fetchConfig)
  
  // the query needs to alter the currently cached state and return that, since we are only partially updating the tree state
  const fetchTreeData = (key, variables) => {
    const state = queryCache.getQueryData(key) as Map<string, any>
    console.log('Fetching', variables)
    return fetchNodes(state, query, variables, limit)
  }

  const { status, data: rawTreeData, error, refetch } = useQuery({
    queryKey: treeId, // unique key that identifies the state of the entire tree
    queryFn: fetchTreeData,
    variables: [fetchConfig]
  })

  // this is a workaround to supply additional parameters to the query and reuse the currently cached state, refetch unfortunately does not support optional query variables to be passed
  useEffect(() => {
    if (previousFetchConfig !== fetchConfig) {
      refetch()
    }
  })

  // on expansion we set the next fetch config 
  const onExpandNode = (node) => {
    const { parentId, searchAfter } = node
    setFetchConfig([{ parentId, searchAfter }])
    // it would be nice to use refetch here to return the promse for the tree so it won't stop the loading animation, but we cannot supply query parameters
    return Promise.resolve() 
  }
  
  if (status === 'loading') {
    return <Spinner />
  } else {
    // this just converts the remote data to the Tree UI model
    const treeData = mapToTreeData(rawTreeData)
    return (
      <Tree
        showIcon
        defaultExpandedKeys={defaultExpandedKeys}
        loadData={(node) => onExpandNode(node)}
        treeData={treeData}
      />
    )
  }

This is working, but the loading animations supported by the tree do not work (because of the resolved Promise I need to send back from onExpandNode(), and there is an additional render due to the extra useState + useEffect workaround.

To overcome the animation issue, I dug a bit into the useBaseQuery() file to find out that the refetch function actually accepts a __queryFn argument that is the assigned to query.queryFn in normal circumstances. In my case, I used that argument to pass an update queryFn that uses the updated query variables without actually using a custom useState + useEffect hook pair.

const onExpandNode = (node) => {
    const { parentId, searchAfter } = node
    // this is basically a workaround to call my 'queryFn' function but with an updated set of variables that should not be part of the queryKey. 
    return refetch({ __queryFn: () => fetchTreeData(treeId, [{ parentId, searchAfter }]) }).then(() => {})
}

I know __queryFn argument is probably internal API and I should not use it at all, but I couldn't find any other way to pass my new optional parameters to a queryFn, where the variables should not be part of the query key.
I saw another discussion (#351), where you suggest to use the variables in the queryKey, but in that case the entire tree state needs to be refetched, which would be undesirable.

Would it be possible to pass new optional query parameters to refetch()?
It looks like this is supported in useInfiniteQuery(), but not in useQuery(), may I ask why?
If there is a way to use the useInfiniteQuery(), I can rewrite it to that, but at the moment I don't think it's possible or easier than this.

Let me know what you think. Feel free to share.
Thank you for reading.

[ChambreNoire]

I'm facing the same issue. Did you happen to make any further progress regarding this ?
Thanks!

[cmark]

Hi @ChambreNoire,

Yes, recent changes in the React Query API made it possible to use custom fetch functions, so we were able to overcome all limitations.
More precisely, we still use Map<string, any> and a single useQuery hook to store the tree state in the React Query cache and preserve it on re-render, etc. When loading nodes we use a custom queryFn via the queryClient.prefetchQuery() function to override the cached value and also return the loaded data for the tree. Most tree implementations use an async loader where the loader must return the loaded data immediately, which makes it harder to integrate with React and React Query (we currently use this tree module and explicit version to render data in tree structure, https://www.npmjs.com/package/react-vtree/v/3.0.0-beta.3).

I hope this helps. Good luck.

Cheers,
Mark

[ChambreNoire]

Hi @cmark ,

Sorry for the late reply!

I did just this, wrapped it in a custom hook, and it all works fine!

Many thanks!

[alexsoshnikov]

In fact, there is a more appropriate solution using option behavior and creating your own class, but at the moment it is impossible to import useBaseQuery from the library. So here is one more solution

This post is about how to handle tree data structure with load on demand using @tanstack/react-query. I will try to tell you in detail about the approach and my findings).

The general idea of the proposed approach is to use useInfiniteQuery and store copies of the entire tree and additional children from the executed query instead of pages. I guess that provided example below will make it clearer. Let's have a look. We have a request that returns departments and has a parameter idParent. If it is empty, the request response will be the root of our tree. Otherwise, if the parameter exists, our request will return the children of our parent node.

Root – [Department 1, Department 2, Department 3]
Request 1 for Department 1 – [Department 1.1, Department 1.2, Department 1.3]

Now imagine that we are making a request using useInfiniteQuery.

  const {data, fetchNextPage} = useInfiniteQuery({
    queryKey: ['departments'],
    queryFn: (context) => {
      return fetchDepartments({ idParent: context.pageParam.id })
    }
  })

The root will be page with index 0 and pageParam will be null or undefined which is fine because we don’t need the param for the first request. Now, we would like to have child nodes of some parent node on demand.

Here we remember that useInfiniteQuery has a wonderful fetchNextPage function, to which we can pass the context with any parameters. Then we make a request with id of node.

  const onClickTreeNode = async (id: number) => {
    await fetchNextPage({ pageParam: { id }})
  }

We make a request with node id and receive an array. In the described approach we have to make a copy of our current tree state (only with root), find the parent, add our received children and add it to the next page.

And that's it. Now we take our last page from data.

Now let's have a closer look at how we can handle the data from the request and add it to the next page.
I will code a wrapper over our fetch function.

  type QueryContext = QueryFunctionContext<unknown[], { id: number }>;
 
  const treeRequest = async (ctx: QueryContext): Promise<Department[]>> => {
    try {
      // id = null or undefined for empty request to receive root
      const {pageParam = {id: null}} = ctx;
      const result = await fetchNextPage({id: pageParam.id});
      const mappedChildren = result.map<Department[]>((r) => ({
        ...r,
        idParent: pageParam.id,
        children: null
      }));
 
      const lastPage = data?.pages.at(-1);
      // if there is no last page it will we the root of tree
 
      if (!lastPage) return Promise.resolve(mapped);
      const tree = produce(lastPage, (draft) => {
        const node = searchTree(draft, {id: pageParam.id});
 
        if (node) {
          node.children = mapped;
        }
      });
 
      return Promise.resolve(tree);
    } catch (e) {
      throw new Error(`Error: unexpected behavior, ${ e }`);
    }
  }

The code is pretty primitive here except one thing. To simplify the tree search algorithm, the searchTree function returns the required parent node with a reference to the entire tree object. Further, using the immer library, we can directly mutate the found node and, thus, return the modified tree without unnecessary iterations. If you suddenly don't understand how it works, I advise you to read this article about object references. Certainly, you can use any tree search algorithm you know. You can search my solution here.

Also, if you need to fetch several tree nodes at the same time, I can provide the following solution. Since within one cache, requests can only be sequential, you just need to implement sequential function call through an array with id nodes.

  // first solution using async generators
  async function* getNodesRequests(nodes: number[]) {
    for (let i = 0; i < nodes.length; i++) {
      yield await fetchNextPage({pageParam: {node: nodes[i]}})
    }
  }
  
  async function loadAsyncGenerators() {
    try {
      for await (const response of getNodesRequests([1, 2, 3, 4, 5])) {
        console.log(response)
      }
    } finally {
      console.log('final')
    }
  }
  
  // second solution using recursion
  const loadRecursively = (nodes: number[]) => {
    if (nodes.length === 0) return
    
    const [head, ...tail] = nodes
    fetchNextPage({pageParam: {node: head}})
      .then(() => loadRecursively(tail))
      .finally(() => {
        if (tail.length === 0) console.log('final')
      })
  }

It seems to me that this solution works well with prefetching and refetching functionality that are provided in react-query. Since we save all the necessary parameters for each state of the tree, it will be able to update the data without issues. I hope this solution will help you to use react-query to solve your problems with tree-like data structures.