Question of LLM inference speed?

[cindy-zhoutingting]

Hello, I noticed that the article mentions 5,000 queries, and the total time for rewriting them using a llm (taking TPC-H as an example) is only 3.40 seconds. This suggests that the inference time for each query is approximately 0.0006 seconds. Could you share which API you used to achieve such speed, or what kind of locally deployed model was utilized? Additionally, was the measurement conducted using techniques such as batching multiple queries for submission to the model or parallel processing with multithreading?

[LJHzju]

Perhaps the Latency mentioned here refers to the average execution time for each query, rather than the total time to complete all 5,000 queries.

In Table 2, the Mean for TPCH is 37.23s, and adding the 3.4s rewrite latency from Table 4 gives exactly 40.63s as shown here.