BloomFilter, BloomFilterImpl and Bloom Filter Join

Author: jaceklaskowski

docs/Dataset.md

@@ -1,3 +1,7 @@
+---
+title: Dataset
+---
+
 # Dataset
 
 `Dataset[T]` is a strongly-typed data structure that represents a structured query over rows of `T` type.

docs/bloom-filter-join/BloomFilter.md

@@ -3,45 +3,65 @@
 `BloomFilter` is an [abstraction](#contract) of [bloom filters](#implementations) for the following:
 
 * [DataFrameStatFunctions.bloomFilter](../DataFrameStatFunctions.md#bloomFilter) operator
-* As an [aggregation buffer](../expressions/BloomFilterAggregate.md#createAggregationBuffer) in [BloomFilterAggregate](../expressions/BloomFilterAggregate.md) expression
+* [BloomFilterAggregate](../expressions/BloomFilterAggregate.md) expression (as an [aggregation buffer](../expressions/BloomFilterAggregate.md#createAggregationBuffer))
 * [BloomFilterMightContain](../expressions/BloomFilterMightContain.md#bloomFilter) expression
 
 ## Contract (Subset)
 
-### <span id="mightContain"> mightContain
+### bitSize { #bitSize }
+
+```java
+long bitSize()
+```
+
+See:
+
+* [BloomFilterImpl](BloomFilterImpl.md#bitSize)
+
+Used when:
+
+* `BloomFilterAggregate` is requested to [serialize a BloomFilter](../expressions/BloomFilterAggregate.md#serialize)
+
+### mightContain { #mightContain }
 
 ```java
 boolean mightContain(
   Object item)
 ```
 
-See [BloomFilterImpl](BloomFilterImpl.md#mightContain)
+See:
+
+* [BloomFilterImpl](BloomFilterImpl.md#mightContain)
 
 !!! note "Not Used"
     `mightContain` does not seem to be used (as [mightContainLong](#mightContainLong) seems to be used directly instead).
 
-### <span id="mightContainLong"> mightContainLong
+### mightContainLong { #mightContainLong }
 
 ```java
 boolean mightContainLong(
   long item)
 ```
 
-See [BloomFilterImpl](BloomFilterImpl.md#mightContainLong)
+See:
+
+* [BloomFilterImpl](BloomFilterImpl.md#mightContainLong)
 
 Used when:
 
 * `BloomFilterImpl` is requested to [mightContain](BloomFilterImpl.md#mightContain)
-* `BloomFilterMightContain` is requested to [eval](../expressions/BloomFilterMightContain.md#eval) and [doGenCode](../expressions/BloomFilterMightContain.md#doGenCode)
+* `BloomFilterMightContain` is requested to [evaluate](../expressions/BloomFilterMightContain.md#eval) and [doGenCode](../expressions/BloomFilterMightContain.md#doGenCode)
 
-### <span id="mightContainString"> mightContainString
+### mightContainString { #mightContainString }
 
 ```java
 boolean mightContainString(
   String item)
 ```
 
-See [BloomFilterImpl](BloomFilterImpl.md#mightContainString)
+See:
+
+* [BloomFilterImpl](BloomFilterImpl.md#mightContainString)
 
 Used when:
 
@@ -51,7 +71,7 @@ Used when:
 
 * [BloomFilterImpl](BloomFilterImpl.md)
 
-## <span id="create"> Creating BloomFilter
+## Creating BloomFilter { #create }
 
 ```java
 BloomFilter create(
@@ -66,7 +86,7 @@ BloomFilter create(
 
 `create` creates a [BloomFilterImpl](BloomFilterImpl.md) for the given `expectedNumItems`.
 
-Unless the false positive probability is given, `create` uses [DEFAULT_FPP](#DEFAULT_FPP) value to [determine the number of bits](#optimalNumOfBits).
+Unless the **False Positive Probability** (`fpp`) is given, `create` uses [DEFAULT_FPP](#DEFAULT_FPP) value to [determine the optimal number of bits](#optimalNumOfBits).
 
 ---
 

docs/bloom-filter-join/BloomFilterImpl.md

@@ -13,15 +13,20 @@
 
 * `BloomFilter` is requested to [create a BloomFilter](BloomFilter.md#create)
 
-## <span id="mightContainLong"> mightContainLong
+## mightContainLong { #mightContainLong }
 
-```java
-boolean mightContainLong(
-  long item)
-```
+??? note "BloomFilter"
 
-`mightContainLong` is part of the [BloomFilter](BloomFilter.md#mightContainLong) abstraction.
+    ```java
+    boolean mightContainLong(
+      long item)
+    ```
 
----
+    `mightContainLong` is part of the [BloomFilter](BloomFilter.md#mightContainLong) abstraction.
 
-`mightContainLong`...FIXME
+`mightContainLong` uses `Murmur3_x86_32` to generate two hashes of the given `item` with two different seeds: `0` and the hash result of the first hashing.
+
+`mightContainLong` requests the [BitArray](#bits) for the number of bits (`bitSize`).
+
+In the end, `mightContainLong` checks out if the bit for the hashes (combined) is set (non-zero) in the [BitArray](#bits) up to [numHashFunctions](#numHashFunctions) times.
+With all the bits checked and set, `mightContainLong` is positive. Otherwise, `mightContainLong` is negative.

docs/bloom-filter-join/index.md

@@ -1,12 +1,14 @@
 # Bloom Filter Join
 
-**Bloom Filter Join** is an optimization of join queries by pre-filtering one side of a join using a Bloom filter and IN predicate based on the values from the other side of the join.
+**Bloom Filter Join** is an optimization of join queries by pre-filtering one side of a join using [BloomFilter](BloomFilter.md) or `InSubquery` predicate based on the values from the other side of the join.
+
+Bloom Filter Join uses [BloomFilter](BloomFilter.md)s as runtime filters when [spark.sql.optimizer.runtime.bloomFilter.enabled](../configuration-properties.md#spark.sql.optimizer.runtime.bloomFilter.enabled) configuration property is enabled.
+
+Bloom Filter Join uses [InjectRuntimeFilter](../logical-optimizations/InjectRuntimeFilter.md) logical optimization to inject up to [spark.sql.optimizer.runtimeFilter.number.threshold](../configuration-properties.md#spark.sql.optimizer.runtimeFilter.number.threshold) filters ([BloomFilter](BloomFilter.md)s or `InSubquery`s).
 
 ??? note "SPARK-32268"
     Bloom Filter Join was introduced in [SPARK-32268]({{ spark.jira }}/SPARK-32268).
 
-Bloom Filter Join uses [InjectRuntimeFilter](../logical-optimizations/InjectRuntimeFilter.md) logical optimization to...FIXME
-
 ## Configuration Properties
 
 * [spark.sql.optimizer.runtime.bloomFilter.enabled](../configuration-properties.md#spark.sql.optimizer.runtime.bloomFilter.enabled)

docs/logical-optimizations/InjectRuntimeFilter.md

@@ -25,18 +25,24 @@
 
 With [runtimeFilterSemiJoinReductionEnabled](../SQLConf.md#runtimeFilterSemiJoinReductionEnabled) enabled and the new and the initial logical plans not equal, `apply` executes [RewritePredicateSubquery](RewritePredicateSubquery.md) logical optimization with the new logical plan. Otherwise, `apply` returns the new logical plan.
 
-## <span id="tryInjectRuntimeFilter"> tryInjectRuntimeFilter
+## tryInjectRuntimeFilter { #tryInjectRuntimeFilter }
 
 ```scala
 tryInjectRuntimeFilter(
   plan: LogicalPlan): LogicalPlan
 ```
 
-`tryInjectRuntimeFilter` [finds equi-joins](../ExtractEquiJoinKeys.md#unapply) in the given [LogicalPlan](../logical-operators/LogicalPlan.md).
+`tryInjectRuntimeFilter` transforms the given [LogicalPlan](../logical-operators/LogicalPlan.md) with regards to [equi-joins](../ExtractEquiJoinKeys.md#unapply).
 
-When _some_ requirements are met, `tryInjectRuntimeFilter` [injectFilter](#injectFilter) on the left side first and on the right side if on the left was not successful.
+For every equi-join, `tryInjectRuntimeFilter` [injects a runtime filter](#injectFilter) (on the left side first and on the right side if on the left was not successful) when all the following requirements are met:
 
-`tryInjectRuntimeFilter` uses [spark.sql.optimizer.runtimeFilter.number.threshold](../configuration-properties.md#spark.sql.optimizer.runtimeFilter.number.threshold) configuration property.
+1. A join side has no [DynamicPruningSubquery](#hasDynamicPruningSubquery) filter already
+1. A join side has no [RuntimeFilter](#hasRuntimeFilter)
+1. The left and right keys (pair-wise) are [simple expression](#isSimpleExpression)s
+1. [canPruneLeft](../JoinSelectionHelper.md#canPruneLeft) or [canPruneRight](../JoinSelectionHelper.md#canPruneRight)
+1. [filteringHasBenefit](#filteringHasBenefit)
+
+`tryInjectRuntimeFilter` tries to inject up to [spark.sql.optimizer.runtimeFilter.number.threshold](../configuration-properties.md#spark.sql.optimizer.runtimeFilter.number.threshold) filters.
 
 ## Injecting Filter Operator { #injectFilter }
 
@@ -48,7 +54,9 @@ injectFilter(
   filterCreationSidePlan: LogicalPlan): LogicalPlan
 ```
 
-`injectFilter`...FIXME
+With [spark.sql.optimizer.runtime.bloomFilter.enabled](../configuration-properties.md#spark.sql.optimizer.runtime.bloomFilter.enabled), `injectFilter` [injects a filter using BloomFilter](#injectBloomFilter).
+
+Otherwise, `injectFilter` [injects a filter using InSubquery](#injectInSubqueryFilter).
 
 ### Injecting BloomFilter { #injectBloomFilter }
 

docs/spark-sql-DataFrameNaFunctions.md

@@ -1,3 +1,7 @@
+---
+title: DataFrameNaFunctions
+---
+
 # DataFrameNaFunctions — Working With Missing Data
 
 `DataFrameNaFunctions` is used to work with <<methods, missing data>> in a structured query (a [DataFrame](DataFrame.md)).

docs/spark-sql-Dataset-basic-actions.md

@@ -1,3 +1,7 @@
+---
+title: Basic Actions
+---
+
 # Dataset API — Basic Actions
 
 **Basic actions** are a set of operators (_methods_) of the <<spark-sql-dataset-operators.md#, Dataset API>> for transforming a `Dataset` into a session-scoped or global temporary view and _other basic actions_ (FIXME).

docs/spark-sql-dataset-operators.md

@@ -1,3 +1,7 @@
+---
+title: Operators
+---
+
 # Dataset API — Dataset Operators
 
 Dataset API is a [set of operators](#methods) with typed and untyped transformations, and actions to work with a structured query (as a [Dataset](Dataset.md)) as a whole.