spark-riak-connector

Spark Connector for Riak KV and Riak TS

License	License The Apache Software License, Version 2.0
GroupId	GroupId com.basho.riak
ArtifactId	ArtifactId spark-riak-connector
Last Version	Last Version 1.5.1
Release Date	Release Date Jun 8, 2016
Type	Type jar
Description	Description spark-riak-connector Spark Connector for Riak KV and Riak TS

Download spark-riak-connector

Filename	Size
spark-riak-connector-1.5.1.pom
spark-riak-connector-1.5.1.jar	469 KB
spark-riak-connector-1.5.1-uber.jar	5 MB
spark-riak-connector-1.5.1-sources.jar	61 KB
spark-riak-connector-1.5.1-javadoc.jar	685 KB
Browse

How to add to project

Apache Maven

<!-- https://jarcasting.com/artifacts/com.basho.riak/spark-riak-connector/ -->
<dependency>
    <groupId>com.basho.riak</groupId>
    <artifactId>spark-riak-connector</artifactId>
    <version>1.5.1</version>
</dependency>

Gradle Groovy

// https://jarcasting.com/artifacts/com.basho.riak/spark-riak-connector/
implementation 'com.basho.riak:spark-riak-connector:1.5.1'

Gradle Kotlin

// https://jarcasting.com/artifacts/com.basho.riak/spark-riak-connector/
implementation ("com.basho.riak:spark-riak-connector:1.5.1")

Apache Buildr

'com.basho.riak:spark-riak-connector:jar:1.5.1'

Apache Ivy

<dependency org="com.basho.riak" name="spark-riak-connector" rev="1.5.1">
  <artifact name="spark-riak-connector" type="jar" />
</dependency>

Groovy Grape

@Grapes(
@Grab(group='com.basho.riak', module='spark-riak-connector', version='1.5.1')
)

Scala SBT

libraryDependencies += "com.basho.riak" % "spark-riak-connector" % "1.5.1"

Leiningen

[com.basho.riak/spark-riak-connector "1.5.1"]

Dependencies

compile (6)

Group / Artifact	Type	Version
org.scala-lang : scala-reflect	jar	2.10.5
com.google.guava : guava	jar	14.0.1
com.fasterxml.jackson.module : jackson-module-scala_2.10	jar	2.4.4
org.apache.spark : spark-core_2.10	jar	1.6.1
org.apache.spark : spark-sql_2.10	jar	1.6.1
com.basho.riak » dataplatform-riak-client	jar	1.5.0

test (5)

Group / Artifact	Type	Version
com.basho.riak : spark-riak-connector-test-utils	jar	1.5.1
org.mockito : mockito-core	jar	1.10.19
org.powermock : powermock-module-junit4	jar	1.6.4
org.powermock : powermock-api-mockito	jar	1.6.4
org.hamcrest : hamcrest-all	jar	1.3

Project Modules

There are no modules declared in this project.

Spark-Riak Connector

The Spark-Riak connector enables you to connect Spark applications to Riak KV and Riak TS with the Spark RDD and Spark DataFrames APIs. You can write your app in Scala, Python, and Java. The connector makes it easy to partition the data you get from Riak so multiple Spark workers can process the data in parallel and it has support for failover if a Riak node goes down while your Spark job is running.

Features

Construct a Spark RDD from a Riak KV bucket with a set of keys
Construct a Spark RDD from a Riak KV bucket by using a 2i string index or a set of indexes
Construct a Spark RDD from a Riak KV bucket by using a 2i range query or a set of ranges
Map JSON formatted data from Riak KV to user defined types
Save a Spark RDD into a Riak KV bucket and apply 2i indexes to the contents
Construct a Spark Dataframe from a Riak TS table using range queries and schema discovery
Save a Spark Dataframe into a Riak TS table
Construct a Spark RDD using Riak KV bucket's enhanced 2i query (a.k.a. full bucket read)
Perform parallel full bucket reads from a Riak KV bucket into multiple partitions

Compatibility

Riak TS 1.3.1+
Apache Spark 1.6+
Scala 2.10 and 2.11
Java 8

Coming Soon

Support for Riak KV 2.3 and later

Prerequisites

In order to use the Spark-Riak connector, you must have the following installed: