Framework: spark-testing-base¶
Framework “spark-testing-base” located in github
Drawbacks: current version provided SparkContext and JavaSparkContext only, some additional code required for get SparkSession.
Inclusion in project with Maven¶
<dependency>
<groupId>com.holdenkarau</groupId>
<artifactId>spark-testing-base_2.11</artifactId>
<version>2.3.1_0.10.0</version>
<scope>test</scope>
</dependency>
Example of pom.xml
“SparkContext” and “JavaSparkContext”¶
Can be received with methods: “sc()” “jsc()”
Integer[] weights = {120, 150};
long expected = Arrays.stream(weights).reduce(0, (a, v) -> a + v);
Dataset<Row> df = new SQLContext(jsc()).createDataset(Arrays.asList(weights), Encoders.INT()).toDF("weight");
long actual = repository.mass(df);
assertEquals(expected, actual);