Skip to content

Commit 737ce74

Browse files
jaceklaskowskijaceklaskowski
committed
LessThanOrEqual Binary Comparison Expression
1 parent a55d79b commit 737ce74

File tree

8 files changed

+92
-157
lines changed

8 files changed

+92
-157
lines changed

‎docs/Filter.md‎

Lines changed: 0 additions & 65 deletions
Original file line numberDiff line numberDiff line change
@@ -48,59 +48,6 @@ Used when:
4848
* <<And, And>>, <<Or, Or>> and <<Not, Not>> filters are requested for the <<references, column references>>
4949
|===
5050

51-
[[implementations]]
52-
.Filters
53-
[cols="1,2",options="header",width="100%"]
54-
|===
55-
| Filter
56-
| Description
57-
58-
| `And`
59-
| [[And]]
60-
61-
| `EqualNullSafe`
62-
| [[EqualNullSafe]]
63-
64-
| `EqualTo`
65-
| [[EqualTo]]
66-
67-
| `GreaterThan`
68-
| [[GreaterThan]]
69-
70-
| `GreaterThanOrEqual`
71-
| [[GreaterThanOrEqual]]
72-
73-
| `In`
74-
| [[In]]
75-
76-
| `IsNotNull`
77-
| [[IsNotNull]]
78-
79-
| `IsNull`
80-
| [[IsNull]]
81-
82-
| `LessThan`
83-
| [[LessThan]]
84-
85-
| `LessThanOrEqual`
86-
| [[LessThanOrEqual]]
87-
88-
| `Not`
89-
| [[Not]]
90-
91-
| `Or`
92-
| [[Or]]
93-
94-
| `StringContains`
95-
| [[StringContains]]
96-
97-
| `StringEndsWith`
98-
| [[StringEndsWith]]
99-
100-
| `StringStartsWith`
101-
| [[StringStartsWith]]
102-
|===
103-
10451
=== [[findReferences]] Finding Column References in Any Value -- `findReferences` Method
10552

10653
[source, scala]
@@ -109,15 +56,3 @@ findReferences(value: Any): Array[String]
10956
----
11057

11158
`findReferences` takes the <<references, references>> from the `value` filter is it is one or returns an empty array.
112-
113-
NOTE: `findReferences` is used when <<EqualTo, EqualTo>>, <<EqualNullSafe, EqualNullSafe>>, <<GreaterThan, GreaterThan>>, <<GreaterThanOrEqual, GreaterThanOrEqual>>, <<LessThan, LessThan>>, <<LessThanOrEqual, LessThanOrEqual>> and <<In, In>> filters are requested for their <<references, column references>>.
114-
115-
## v2references
116-
117-
```
118-
v2references: Array[Array[String]]
119-
```
120-
121-
v2references...FIXME
122-
123-
v2references is used when...FIXME

‎docs/execution-planning-strategies/DataSourceStrategy.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -118,8 +118,8 @@ Catalyst Expression | Filter Predicate
118118
[EqualNullSafe](../expressions/EqualNullSafe.md) (with a "pushable" column and a `Literal`) | `EqualNullSafe`
119119
`GreaterThan` (with a "pushable" column and a `Literal`) | `GreaterThan` or `LessThan`
120120
`LessThan` (with a "pushable" column and a `Literal`) | `LessThan` or `GreaterThan`
121-
`GreaterThanOrEqual` (with a "pushable" column and a `Literal`) | `GreaterThanOrEqual` or `LessThanOrEqual`
122-
`LessThanOrEqual` (with a "pushable" column and a `Literal`) | `LessThanOrEqual` or `GreaterThanOrEqual`
121+
`GreaterThanOrEqual` (with a "pushable" column and a `Literal`) | `GreaterThanOrEqual` or [LessThanOrEqual](../expressions/LessThanOrEqual.md)
122+
[LessThanOrEqual](../expressions/LessThanOrEqual.md) (with a "pushable" column and a `Literal`) | [LessThanOrEqual](../expressions/LessThanOrEqual.md) or `GreaterThanOrEqual`
123123
[InSet](../expressions/InSet.md) (with a "pushable" column and values) | `In`
124124
[InSet](../expressions/In.md) (with a "pushable" column and expressions) | `In`
125125
`IsNull` (with a "pushable" column) | `IsNull`

‎docs/expressions/BinaryComparison.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,4 +9,4 @@
99
* `GreaterThan`
1010
* `GreaterThanOrEqual`
1111
* `LessThan`
12-
* `LessThanOrEqual`
12+
* [LessThanOrEqual](LessThanOrEqual.md)

‎docs/expressions/BinaryOperator.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# BinaryOperator
2+
3+
`BinaryOperator` is...FIXME

‎docs/expressions/LessThanOrEqual.md‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
# LessThanOrEqual
2+
3+
`LessThanOrEqual` is a [binary comparison expression](BinaryComparison.md) and a `NullIntolerant`.
4+
5+
`LessThanOrEqual`'s [symbol](#symbol) representation is `<=`.
6+
7+
## Creating Instance
8+
9+
`LessThanOrEqual` takes the following to be created:
10+
11+
* <span id="left"> Left [Expression](Expression.md)
12+
* <span id="right"> Right [Expression](Expression.md)
13+
14+
## <span id="symbol"> symbol
15+
16+
```scala
17+
symbol: String
18+
```
19+
20+
`symbol` is part of the [BinaryOperator](BinaryOperator.md#symbol) abstraction.
21+
22+
---
23+
24+
`symbol` is `<=`.
25+
26+
## <span id="nullSafeEval"> nullSafeEval
27+
28+
```scala
29+
nullSafeEval(
30+
input1: Any,
31+
input2: Any): Any
32+
```
33+
34+
`nullSafeEval` is part of the [BinaryOperator](BinaryOperator.md#nullSafeEval) abstraction.
35+
36+
---
37+
38+
`nullSafeEval` requests the [Ordering](BinaryComparison.md#ordering) to `lteq` the inputs.
39+
40+
## Catalyst DSL
41+
42+
`expressions` defines [<=](../catalyst-dsl/index.md#ExpressionConversions) operator to create a `LessThanOrEqual` expression.
43+
44+
```scala
45+
import org.apache.spark.sql.catalyst.dsl.expressions._
46+
47+
// LessThanOrEqual
48+
val e = col("a").expr <= 5
49+
50+
import org.apache.spark.sql.catalyst.expressions.LessThanOrEqual
51+
val lessThanOrEqual = e.asInstanceOf[LessThanOrEqual]
52+
```
53+
54+
!!! note "FIXME"
55+
56+
```scala
57+
import org.apache.spark.sql.catalyst.InternalRow
58+
val input = InternalRow(1, 2)
59+
```
60+
61+
```text
62+
scala> lessThanOrEqual.eval(input)
63+
org.apache.spark.SparkUnsupportedOperationException: Cannot evaluate expression: 'a
64+
at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotEvaluateExpressionError(QueryExecutionErrors.scala:73)
65+
at org.apache.spark.sql.catalyst.expressions.Unevaluable.eval(Expression.scala:344)
66+
at org.apache.spark.sql.catalyst.expressions.Unevaluable.eval$(Expression.scala:343)
67+
at org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute.eval(unresolved.scala:131)
68+
at org.apache.spark.sql.catalyst.expressions.BinaryExpression.eval(Expression.scala:634)
69+
... 54 elided
70+
```

‎docs/logical-operators/Join.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@
2222

2323
## Catalyst DSL
2424

25-
`DslLogicalPlan` comes with [join](../catalyst-dsl/DslLogicalPlan.md#join) operator to create a `Join`.
25+
`DslLogicalPlan` defines [join](../catalyst-dsl/DslLogicalPlan.md#join) operator to create a `Join`.
2626

2727
```scala
2828
import org.apache.spark.sql.catalyst.dsl.plans._

‎docs/physical-operators/InMemoryTableScanExec.md‎

Lines changed: 13 additions & 88 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,17 @@
11
# InMemoryTableScanExec Leaf Physical Operator
22

3-
`InMemoryTableScanExec` is a leaf physical operator that represents an [InMemoryRelation](#relation) logical operator at execution time.
3+
`InMemoryTableScanExec` is a [leaf physical operator](SparkPlan.md#LeafExecNode) that represents an [InMemoryRelation](#relation) logical operator at execution time.
4+
5+
## <span id="metrics"> Performance Metrics
6+
7+
Key | Name (in web UI) | Description
8+
----------------|-------------------------|---------
9+
numOutputRows | number of output rows | Number of output rows
10+
11+
![InMemoryTableScanExec in web UI (Details for Query)](../images/spark-sql-InMemoryTableScanExec-webui-query-details.png)
12+
13+
<!---
14+
## Review Me
415
516
`InMemoryTableScanExec` is <<creating-instance, created>> exclusively when [InMemoryScans](../execution-planning-strategies/InMemoryScans.md) execution planning strategy is executed and finds an InMemoryRelation.md[InMemoryRelation] logical operator in a logical query plan.
617
@@ -145,17 +156,6 @@ filteredCachedBatches(): RDD[CachedBatch]
145156
146157
`filteredCachedBatches` is used when `InMemoryTableScanExec` is requested for the [inputRDD](#inputRDD) internal property.
147158
148-
=== [[statsFor]] `statsFor` Internal Method
149-
150-
[source, scala]
151-
----
152-
statsFor(a: Attribute)
153-
----
154-
155-
`statsFor`...FIXME
156-
157-
NOTE: `statsFor` is used when...FIXME
158-
159159
=== [[createAndDecompressColumn]] `createAndDecompressColumn` Internal Method
160160
161161
```scala
@@ -270,79 +270,4 @@ With <<supportsBatch, supportsBatch>> flag on, `doExecute` creates a WholeStageC
270270
Otherwise, when <<supportsBatch, supportsBatch>> flag is off, `doExecute` simply gives the <<inputRDD, input RDD of internal rows>>.
271271
272272
`doExecute` is part of the [SparkPlan](SparkPlan.md#doExecute) abstraction.
273-
274-
=== [[buildFilter]] `buildFilter` Property
275-
276-
[source, scala]
277-
----
278-
buildFilter: PartialFunction[Expression, Expression]
279-
----
280-
281-
NOTE: `buildFilter` is a Scala lazy value which is computed once when accessed and cached afterwards.
282-
283-
`buildFilter` is a Scala https://www.scala-lang.org/api/2.11.11/#scala.PartialFunction[PartialFunction] that accepts an expressions/Expression.md[Expression] and produces an expressions/Expression.md[Expression], i.e. `PartialFunction[Expression, Expression]`.
284-
285-
[[buildFilter-expressions]]
286-
.buildFilter's Expressions
287-
[cols="1,2",options="header",width="100%"]
288-
|===
289-
| Input Expression
290-
| Description
291-
292-
| `And`
293-
|
294-
295-
| `Or`
296-
|
297-
298-
| [EqualTo](../expressions/EqualTo.md)
299-
|
300-
301-
| [EqualNullSafe](../expressions/EqualNullSafe.md)
302-
|
303-
304-
| `LessThan`
305-
|
306-
307-
| `LessThanOrEqual`
308-
|
309-
310-
| `GreaterThan`
311-
|
312-
313-
| `GreaterThanOrEqual`
314-
|
315-
316-
| `IsNull`
317-
|
318-
319-
| `IsNotNull`
320-
|
321-
322-
| `In` with a non-empty spark-sql-Expression-In.md#list[list] of spark-sql-Expression-Literal.md[Literal] expressions
323-
|
324-
For every `Literal` expression in the expression list, `buildFilter` creates an `And` expression with the lower and upper bounds of the <<statsFor, partition statistics for the attribute>> and the `Literal`.
325-
326-
In the end, `buildFilter` joins the `And` expressions with `Or` expressions.
327-
|===
328-
329-
NOTE: `buildFilter` is used exclusively when `InMemoryTableScanExec` is requested for <<partitionFilters, partitionFilters>>.
330-
331-
=== [[innerChildren]] `innerChildren` Method
332-
333-
[source, scala]
334-
----
335-
innerChildren: Seq[QueryPlan[_]]
336-
----
337-
338-
NOTE: `innerChildren` is part of catalyst/QueryPlan.md#innerChildren[QueryPlan Contract] to...FIXME.
339-
340-
`innerChildren`...FIXME
341-
342-
## <span id="metrics"> Performance Metrics
343-
344-
Key | Name (in web UI) | Description
345-
----------------|-------------------------|---------
346-
numOutputRows | number of output rows | Number of output rows
347-
348-
![InMemoryTableScanExec in web UI (Details for Query)](../images/spark-sql-InMemoryTableScanExec-webui-query-details.png)
273+
-->

‎mkdocs.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -366,6 +366,7 @@ nav:
366366
- AttributeSeq: expressions/AttributeSeq.md
367367
- Attribute: expressions/Attribute.md
368368
- BinaryComparison: expressions/BinaryComparison.md
369+
- BinaryOperator: expressions/BinaryOperator.md
369370
- BoundReference: expressions/BoundReference.md
370371
- CallMethodViaReflection: expressions/CallMethodViaReflection.md
371372
- CodeGeneratorWithInterpretedFallback: expressions/CodeGeneratorWithInterpretedFallback.md
@@ -401,6 +402,7 @@ nav:
401402
- InSubqueryExec: expressions/InSubqueryExec.md
402403
- InterpretedProjection: expressions/InterpretedProjection.md
403404
- JsonToStructs: expressions/JsonToStructs.md
405+
- LessThanOrEqual: expressions/LessThanOrEqual.md
404406
- ListQuery: expressions/ListQuery.md
405407
- Literal: expressions/Literal.md
406408
- MonotonicallyIncreasingID: expressions/MonotonicallyIncreasingID.md

0 commit comments

Comments
 (0)