Dynamically modified on flyΒΆ
If input data structure is complex, and only several fields take part in transformation, presaved data can be read, and fields modified on fly.
@Test
public void testMass_WhenWeightSpecified_ThenWeightsSum() {
int expected = 85;
Dataset<Row> input = getBulkData().withColumn("weight", lit(85));
assertEquals(expected, repository.mass(input));
}
@Test(expected = Exception.class)
public void testMass_WhenWeightIsNull_ThenNPE() {
Dataset<Row> input = getBulkData().withColumn("weight", lit(null).cast(DataTypes.IntegerType));
repository.mass(input);
}
private Dataset<Row> getBulkData() {
return spark().read().option("header", "true").csv(getTestDataFolder());
}
Example: DynamicallyModifiedOnFlyTest.java