Framework: spark-test-jar

Inclusion in project with Maven

<dependency>
    <groupId>${spark.groupId}</groupId>
    <artifactId>spark-sql_${scala.suffix}</artifactId>
    <version>${spark.version}</version>
    <scope>provided</scope>
    <type>test-jar</type>
</dependency>
<dependency>
    <groupId>${spark.groupId}</groupId>
    <artifactId>spark-core_${scala.suffix}</artifactId>
    <version>${spark.version}</version>
    <scope>provided</scope>
    <type>test-jar</type>
</dependency>
<dependency>
    <groupId>${spark.groupId}</groupId>
    <artifactId>spark-catalyst_${scala.suffix}</artifactId>
    <version>${spark.version}</version>
    <scope>provided</scope>
    <type>test-jar</type>
</dependency>

Example of pom.xml

Enabling in testcases

Two traits have to be included for enable resources “QueryTest”, “SharedSQLContext”.

class SparkTestJarSpec extends QueryTest with SharedSQLContext with Matchers

“SparkSession” and “SparkContext”

Can be received with methods: “spark” “sparkContext”.

val weights = List(120, 150)
val expected = weights.reduce((a, v) => a + v)
val df = spark.createDataset(weights).toDF("weight")

Code example: SparkTestJarSpec.scala