Baleen

Library for Validating Legacy Data

License

License

GroupId

GroupId

com.shoprunner
ArtifactId

ArtifactId

baleen-xsd-generator
Last Version

Last Version

1.13.0
Release Date

Release Date

Type

Type

pom.sha512
Description

Description

Baleen
Library for Validating Legacy Data
Project URL

Project URL

https://github.com/ShopRunner/baleen
Project Organization

Project Organization

com.shoprunner
Source Code Management

Source Code Management

https://github.com/ShopRunner/baleen

Download baleen-xsd-generator

Dependencies

compile (2)

Group / Artifact Type Version
com.sun.xml.bind : jaxb-impl jar 3.0.0
com.shoprunner : baleen jar 1.13.0

runtime (3)

Group / Artifact Type Version
org.jetbrains.kotlin : kotlin-stdlib jar 1.4.21
com.shoprunner : baleen-base-schema-generator jar 1.13.0
jakarta.xml.bind : jakarta.xml.bind-api jar 3.0.0

Project Modules

There are no modules declared in this project.

Maven Central

Baleen

Baleen is fluent Kotlin DSL for validating data (JSON, XML, CSV, Avro)

Features

Example Baleen Data Description

import com.shoprunner.baleen.Baleen.describeAs
import com.shoprunner.baleen.ValidationError
import com.shoprunner.baleen.dataTrace
import com.shoprunner.baleen.types.StringType

val departments = listOf("Mens", "Womens", "Boys", "Girls", "Kids", "Baby & Toddler")

val productDescription = "Product".describeAs {

    "sku".type(StringType(min = 1, max = 500),
          required = true)

    "brand_manufacturer".type(StringType(min = 1, max = 500),
          required = true)

    "department".type(StringType(min = 0, max = 100))
         .describe { attr ->

        attr.test { datatrace, value ->
            val department = value["department"]
            if (department != null && !departments.contains(department)) {
                sequenceOf(ValidationError(dataTrace, "Department ($department) is not a valid value.", value))
            } else {
                sequenceOf()
            }
        }
    }
}

// Get your data
val data: Data = // get from file or database or whatever 

// Get Validation Results
val validation: Validation = dataDesc.validate(data)

// Each call on `isValid` and `results` will iterate over dataset again. 
// Warning: that for large datasets this will eat memory
val cachedValidation: CachedValidation = validation.cache()

// Check if any errors. True if no errors, false otherwise. 
// val isValid: Boolean = validation.isValid()
val isValid: Boolean = cachedValidation.isValid() 

// Iterate over results. Each iteration over results will execute entire flow again.
// validation.results.forEach { }
cachedValidation.results.forEach { }

// Summarize into Validation object with list of ValidationSummary with examples of errors included    
// val validationSummary: Validation= validation.createSummary()
val validationSummary: CachedValidation = cachedValidation.createSummary()
validationSummary.results.forEach { }

Getting Help

Join the slack channel

Core Concepts

  • Tests are great

    There are a lot of great libraries for testing code. We should use those same concepts for testing data.

  • Performance and streaming are important

    A data validation library should be able to handle large amounts of data quickly.

  • Invalid data is also important

    Warnings and Errors need to be treated as first class objects.

  • Data Traces

    Similar to a stack trace being used to debug a code path, a data trace can be used to debug a path through data.

  • Don't map data to Types too early.

    Type safe code is great but if the data hasn't been santized then it isn't really typed.

Warnings

Sometimes you will want an attribute or type to warn instead of error. The asWarnings() method will transform the output from ValidationError to ValidationWarning for all nested tests run underneath that attribute/type.

import com.shoprunner.baleen.Baleen.describeAs
import com.shoprunner.baleen.ValidationError
import com.shoprunner.baleen.dataTrace
import com.shoprunner.baleen.types.StringType
import com.shoprunner.baleen.types.asWarnings


val productDescription = "Product".describeAs {

    // The asWarnings() method is on StringType. Min/max are warnings, but required is still an error.
    "sku".type(StringType(min = 1, max = 500).asWarnings(), required = true) 

    // The asWarnings() method is on the attribute. Min/max and required are all warnings.
    "brand_manufacturer".type(StringType(min = 1, max = 500), required = true).asWarnings()

    // The asWarnings() method is on the attribute. The attribute's custom test will also be turned into a warning.
    "department".type(StringType(min = 0, max = 100)).describe { attr ->

        attr.test { datatrace, value ->
            val department = value["department"]
            if (department != null && !departments.contains(department)) {
                sequenceOf(ValidationError(dataTrace, "Department ($department) is not a valid value.", value))
            } else {
                sequenceOf()
            }
        }
    }.asWarnings()
}

Tagging

A feature of Baleen is to add tags to tests, so that you can more easily identify, annotate, and filter your results. There are a couple use-cases tagging becomes useful. For example, you have an identifier, like a sku, that you want each test to have so that you can group together failed tests by that identifier. Another use-case is that you have different priority levels for your tests that you can set so you can highlight the most important errors.

val productDescription = "Product".describeAs {

    // The tag() method is on StringType and dynamic tag pulls the value.
    "sku".type(StringType().tag("priority", "critical").tag("sku", withValue()))

    // The tag() method is on the attribute and the dynamic tag pulls an attribute value from sku.
    "brand_manufacturer".type(StringType(), required = true)
        .tag("priority", "low")
        .tag("sku", withAttributeValue("sku"))
 
    // The tag() method is on the attribute, and a custom tag function is used that returns a String
    "department".type(StringType(min = 0, max = 100))
        .tag("priority", "high")
        .tag("sku", withAttributeValue("sku"))
        .tag("gender") { d ->
            when {
                d is Data && d.containsKey("gender") -> 
                    when(d["gender"]) {
                        "male" -> "male"
                        "mens" -> "male"
                        "female" -> "female"
                        "womens" -> "femle"
                        else -> "other"
                    }
                else -> "none"
            }
        }
}
// Tag is on data description and the dynamic tag pulls attribute value from sku field  from the data
.tag("sku", withAttributeValue("sku"))

Tagging is also done at the data evaluation level. When writing tests, DataTrace can be updated with tags

    "department".type(StringType(min = 0, max = 100)).describe { attr ->
        attr.test { datatrace, value ->
            val department = value["department"]
            if (department != null && !departments.contains(department)) {

                // datatrace has the sku tag added
                sequenceOf(ValidationError(
                    dataTrace.tag("sku", value["sku"] ?: "null"), 
                    "Department ($department) is not a valid value.",
                     value
                ))

            } else {
                sequenceOf()
            }
        }
    }

Some Baleen Validation libraries, such as the XML or JSON validators, use tags to add line and column numbers as it parses the original raw data. This will help identify errors in the raw data much more quickly.

Gotchas

  • Baleen does not assume that an attribute is not set and an attribute that is set with the value of null are the same thing.

Similar Projects

com.shoprunner

ShopRunner

Versions

Version
1.13.0
1.12.0
1.11.2
1.11.1
1.11.0
1.10.4
1.10.3
1.10.2
1.10.0
1.9.0
1.8.0
1.7.0
1.6.0
1.5.0
1.4.1
1.4.0
1.3.0
1.2
1.1