Adding readme information and marking text as copyrighted

Author: timsaucer

examples/README.md

@@ -52,3 +52,10 @@ Here is a direct link to the file used in the examples:
 - [Executing SQL on Polars](./sql-on-polars.py)
 - [Executing SQL on Pandas](./sql-on-pandas.py)
 - [Executing SQL on cuDF](./sql-on-cudf.py)
+
+## TPC-H Examples
+
+Within the subdirectory `tpch` there are 22 examples that reproduce queries in
+the TPC-H specification. These include realistic data that can be generated at
+arbitrary scale and allow the user to see use cases for a variety of data frame
+operations.

examples/tpch/README.md

@@ -0,0 +1,57 @@
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+# DataFusion Python Examples for TPC-H
+
+These examples reproduce the problems listed in the Transaction Process Council
+TPC-H benchmark. The purpose of these examples is to demonstrate how to use
+different aspects of Data Fusion and not necessarily geared towards creating the
+most performant queries possible. Within each example is a description of the
+problem. For users who are familiar with SQL style commands, you can compare the
+approaches in these examples with those listed in the specification.
+
+- https://www.tpc.org/tpch/
+
+The examples provided are based on version 2.18.0 of the TPC-H specification.
+
+## Data Setup
+
+To run these examples, you must first generate a dataset. The `dbgen` tool
+provided by TPC can create datasets of arbitrary scale. For testing it is
+typically sufficient to create a 1 gigabyte dataset. For convenience, this
+repository has a script which uses docker to create this dataset. From the
+`benchmarks/tpch` directory execute the following script.
+
+```bash
+./tpch-gen.sh 1
+```
+
+The examples provided use parquet files for the tables generated by `dbgen`.
+An python script is provided to convert the text files from `dbgen` into parquet
+files expected by the examples. From the `examples/tpch` directory you can
+execute the following command to create the necessary parquet files.
+
+```bash
+python convert_data_to_parquet.py
+```
+
+## Description of Examples
+
+For easier access, a description of the techniques demonstrated in each file
+is in the README.md file in the `examples` directory.

examples/tpch/q01_pricing_summary_report.py

@@ -16,12 +16,17 @@
 # under the License.
 
 """
+TPC-H Problem Statement Query 1:
+
 The Pricing Summary Report Query provides a summary pricing report for all lineitems shipped as of
 a given date. The date is within 60 - 120 days of the greatest ship date contained in the database.
 The query lists totals for extended price, discounted extended price, discounted extended price
 plus tax, average quantity, average extended price, and average discount. These aggregates are
 grouped by RETURNFLAG and LINESTATUS, and listed in ascending order of RETURNFLAG and LINESTATUS.
 A count of the number of lineitems in each group is included.
+
+The above problem statement text is copyrighted by the Transaction Processing Performance Council
+as part of their TPC Benchmark H Specification revision 2.18.0.
 """
 
 import pyarrow as pa

examples/tpch/q02_minimum_cost_supplier.py

@@ -16,12 +16,17 @@
 # under the License.
 
 """
+TPC-H Problem Statement Query 2:
+
 The Minimum Cost Supplier Query finds, in a given region, for each part of a certain type and size,
 the supplier who can supply it at minimum cost. If several suppliers in that region offer the
 desired part type and size at the same (minimum) cost, the query lists the parts from suppliers with
 the 100 highest account balances. For each supplier, the query lists the supplier's account balance,
 name and nation; the part's number and manufacturer; the supplier's address, phone number and
 comment information.
+
+The above problem statement text is copyrighted by the Transaction Processing Performance Council
+as part of their TPC Benchmark H Specification revision 2.18.0.
 """
 
 import datafusion

examples/tpch/q03_shipping_priority.py

@@ -16,10 +16,15 @@
 # under the License.
 
 """
+TPC-H Problem Statement Query 3:
+
 The Shipping Priority Query retrieves the shipping priority and potential revenue, defined as the
 sum of l_extendedprice * (1-l_discount), of the orders having the largest revenue among those that
 had not been shipped as of a given date. Orders are listed in decreasing order of revenue. If more
 than 10 unshipped orders exist, only the 10 orders with the largest revenue are listed.
+
+The above problem statement text is copyrighted by the Transaction Processing Performance Council
+as part of their TPC Benchmark H Specification revision 2.18.0.
 """
 
 from datafusion import SessionContext, col, lit, functions as F

examples/tpch/q04_order_priority_checking.py

@@ -16,9 +16,14 @@
 # under the License.
 
 """
+TPC-H Problem Statement Query 4:
+
 The Order Priority Checking Query counts the number of orders ordered in a given quarter of a given
 year in which at least one lineitem was received by the customer later than its committed date. The
 query lists the count of such orders for each order priority sorted in ascending priority order.
+
+The above problem statement text is copyrighted by the Transaction Processing Performance Council
+as part of their TPC Benchmark H Specification revision 2.18.0.
 """
 
 from datetime import datetime

examples/tpch/q05_local_supplier_volume.py

@@ -16,12 +16,17 @@
 # under the License.
 
 """
+TPC-H Problem Statement Query 5:
+
 The Local Supplier Volume Query lists for each nation in a region the revenue volume that resulted
 from lineitem transactions in which the customer ordering parts and the supplier filling them were
 both within that nation. The query is run in order to determine whether to institute local
 distribution centers in a given region. The query considers only parts ordered in a given year. The
 query displays the nations and revenue volume in descending order by revenue. Revenue volume for all
 qualifying lineitems in a particular nation is defined as sum(l_extendedprice * (1 - l_discount)).
+
+The above problem statement text is copyrighted by the Transaction Processing Performance Council
+as part of their TPC Benchmark H Specification revision 2.18.0.
 """
 
 from datetime import datetime

examples/tpch/q06_forecasting_revenue_change.py

@@ -16,12 +16,17 @@
 # under the License.
 
 """
+TPC-H Problem Statement Query 6:
+
 The Forecasting Revenue Change Query considers all the lineitems shipped in a given year with
 discounts between DISCOUNT-0.01 and DISCOUNT+0.01. The query lists the amount by which the total
 revenue would have increased if these discounts had been eliminated for lineitems with l_quantity
 less than quantity. Note that the potential revenue increase is equal to the sum of
 [l_extendedprice * l_discount] for all lineitems with discounts and quantities in the qualifying
 range.
+
+The above problem statement text is copyrighted by the Transaction Processing Performance Council
+as part of their TPC Benchmark H Specification revision 2.18.0.
 """
 
 from datetime import datetime

examples/tpch/q07_volume_shipping.py

@@ -16,11 +16,16 @@
 # under the License.
 
 """
+TPC-H Problem Statement Query 7:
+
 The Volume Shipping Query finds, for two given nations, the gross discounted revenues derived from
 lineitems in which parts were shipped from a supplier in either nation to a customer in the other
 nation during 1995 and 1996. The query lists the supplier nation, the customer nation, the year,
 and the revenue from shipments that took place in that year. The query orders the answer by
 Supplier nation, Customer nation, and year (all ascending).
+
+The above problem statement text is copyrighted by the Transaction Processing Performance Council
+as part of their TPC Benchmark H Specification revision 2.18.0.
 """
 
 from datetime import datetime

examples/tpch/q08_market_share.py

@@ -16,10 +16,15 @@
 # under the License.
 
 """
+TPC-H Problem Statement Query 8:
+
 The market share for a given nation within a given region is defined as the fraction of the
 revenue, the sum of [l_extendedprice * (1-l_discount)], from the products of a specified type in
 that region that was supplied by suppliers from the given nation. The query determines this for the
 years 1995 and 1996 presented in this order.
+
+The above problem statement text is copyrighted by the Transaction Processing Performance Council
+as part of their TPC Benchmark H Specification revision 2.18.0.
 """
 
 from datetime import datetime

examples/tpch/q09_product_type_profit_measure.py

@@ -16,12 +16,17 @@
 # under the License.
 
 """
+TPC-H Problem Statement Query 9:
+
 The Product Type Profit Measure Query finds, for each nation and each year, the profit for all parts
 ordered in that year that contain a specified substring in their names and that were filled by a
 supplier in that nation. The profit is defined as the sum of
 [(l_extendedprice*(1-l_discount)) - (ps_supplycost * l_quantity)] for all lineitems describing
 parts in the specified line. The query lists the nations in ascending alphabetical order and, for
 each nation, the year and profit in descending order by year (most recent first).
+
+The above problem statement text is copyrighted by the Transaction Processing Performance Council
+as part of their TPC Benchmark H Specification revision 2.18.0.
 """
 
 import pyarrow as pa

examples/tpch/q10_returned_item_reporting.py

@@ -16,12 +16,17 @@
 # under the License.
 
 """
+TPC-H Problem Statement Query 10:
+
 The Returned Item Reporting Query finds the top 20 customers, in terms of their effect on lost
 revenue for a given quarter, who have returned parts. The query considers only parts that were
 ordered in the specified quarter. The query lists the customer's name, address, nation, phone
 number, account balance, comment information and revenue lost. The customers are listed in
 descending order of lost revenue. Revenue lost is defined as
 sum(l_extendedprice*(1-l_discount)) for all qualifying lineitems.
+
+The above problem statement text is copyrighted by the Transaction Processing Performance Council
+as part of their TPC Benchmark H Specification revision 2.18.0.
 """
 
 from datetime import datetime

examples/tpch/q11_important_stock_identification.py

@@ -16,10 +16,15 @@
 # under the License.
 
 """
+TPC-H Problem Statement Query 11:
+
 The Important Stock Identification Query finds, from scanning the available stock of suppliers
 in a given nation, all the parts that represent a significant percentage of the total value of
 all available parts. The query displays the part number and the value of those parts in
 descending order of value.
+
+The above problem statement text is copyrighted by the Transaction Processing Performance Council
+as part of their TPC Benchmark H Specification revision 2.18.0.
 """
 
 from datafusion import SessionContext, WindowFrame, col, lit, functions as F

examples/tpch/q12_ship_mode_order_priority.py

@@ -16,12 +16,17 @@
 # under the License.
 
 """
+TPC-H Problem Statement Query 12:
+
 The Shipping Modes and Order Priority Query counts, by ship mode, for lineitems actually received
 by customers in a given year, the number of lineitems belonging to orders for which the
 l_receiptdate exceeds the l_commitdate for two different specified ship modes. Only lineitems that
 were actually shipped before the l_commitdate are considered. The late lineitems are partitioned
 into two groups, those with priority URGENT or HIGH, and those with a priority other than URGENT or
 HIGH.
+
+The above problem statement text is copyrighted by the Transaction Processing Performance Council
+as part of their TPC Benchmark H Specification revision 2.18.0.
 """
 
 from datetime import datetime

examples/tpch/q13_customer_distribution.py

@@ -16,11 +16,16 @@
 # under the License.
 
 """
+TPC-H Problem Statement Query 13:
+
 This query determines the distribution of customers by the number of orders they have made,
 including customers who have no record of orders, past or present. It counts and reports how many
 customers have no orders, how many have 1, 2, 3, etc. A check is made to ensure that the orders
 counted do not fall into one of several special categories of orders. Special categories are
 identified in the order comment column by looking for a particular pattern.
+
+The above problem statement text is copyrighted by the Transaction Processing Performance Council
+as part of their TPC Benchmark H Specification revision 2.18.0.
 """
 
 from datafusion import SessionContext, col, lit, functions as F

examples/tpch/q14_promotion_effect.py

@@ -16,9 +16,14 @@
 # under the License.
 
 """
+TPC-H Problem Statement Query 14:
+
 The Promotion Effect Query determines what percentage of the revenue in a given year and month was
 derived from promotional parts. The query considers only parts actually shipped in that month and
 gives the percentage. Revenue is defined as (l_extendedprice * (1-l_discount)).
+
+The above problem statement text is copyrighted by the Transaction Processing Performance Council
+as part of their TPC Benchmark H Specification revision 2.18.0.
 """
 
 from datetime import datetime

examples/tpch/q15_top_supplier.py

@@ -16,9 +16,14 @@
 # under the License.
 
 """
+TPC-H Problem Statement Query 15:
+
 The Top Supplier Query finds the supplier who contributed the most to the overall revenue for parts
 shipped during a given quarter of a given year. In case of a tie, the query lists all suppliers
 whose contribution was equal to the maximum, presented in supplier number order.
+
+The above problem statement text is copyrighted by the Transaction Processing Performance Council
+as part of their TPC Benchmark H Specification revision 2.18.0.
 """
 
 from datetime import datetime

examples/tpch/q16_part_supplier_relationship.py

@@ -16,11 +16,16 @@
 # under the License.
 
 """
+TPC-H Problem Statement Query 16:
+
 The Parts/Supplier Relationship Query counts the number of suppliers who can supply parts that
 satisfy a particular customer's requirements. The customer is interested in parts of eight
 different sizes as long as they are not of a given type, not of a given brand, and not from a
 supplier who has had complaints registered at the Better Business Bureau. Results must be presented
 in descending count and ascending brand, type, and size.
+
+The above problem statement text is copyrighted by the Transaction Processing Performance Council
+as part of their TPC Benchmark H Specification revision 2.18.0.
 """
 
 import pyarrow as pa

examples/tpch/q17_small_quantity_order.py

@@ -16,11 +16,16 @@
 # under the License.
 
 """
+TPC-H Problem Statement Query 17:
+
 The Small-Quantity-Order Revenue Query considers parts of a given brand and with a given container
 type and determines the average lineitem quantity of such parts ordered for all orders (past and
 pending) in the 7-year database. What would be the average yearly gross (undiscounted) loss in
 revenue if orders for these parts with a quantity of less than 20% of this average were no longer
 taken?
+
+The above problem statement text is copyrighted by the Transaction Processing Performance Council
+as part of their TPC Benchmark H Specification revision 2.18.0.
 """
 
 from datafusion import SessionContext, WindowFrame, col, lit, functions as F

examples/tpch/q18_large_volume_customer.py

@@ -16,9 +16,14 @@
 # under the License.
 
 """
+TPC-H Problem Statement Query 18:
+
 The Large Volume Customer Query finds a list of the top 100 customers who have ever placed large
 quantity orders. The query lists the customer name, customer key, the order key, date and total
 price and the quantity for the order.
+
+The above problem statement text is copyrighted by the Transaction Processing Performance Council
+as part of their TPC Benchmark H Specification revision 2.18.0.
 """
 
 from datafusion import SessionContext, col, lit, functions as F

examples/tpch/q19_discounted_revenue.py

@@ -16,9 +16,14 @@
 # under the License.
 
 """
+TPC-H Problem Statement Query 19:
+
 The Discounted Revenue query finds the gross discounted revenue for all orders for three different
 types of parts that were shipped by air and delivered in person. Parts are selected based on the
 combination of specific brands, a list of containers, and a range of sizes.
+
+The above problem statement text is copyrighted by the Transaction Processing Performance Council
+as part of their TPC Benchmark H Specification revision 2.18.0.
 """
 
 import pyarrow as pa