com.microsoft.azure:spark-streaming-eventhubs_examples_2.11

Azure Eventhubs Connector for Spark Streaming Applications

License	License The Apache License, Version 2.0
GroupId	GroupId com.microsoft.azure
ArtifactId	ArtifactId spark-streaming-eventhubs_examples_2.11
Last Version	Last Version 2.1.4
Release Date	Release Date Sep 21, 2017
Type	Type jar
Description	Description Azure Eventhubs Connector for Spark Streaming Applications

Download spark-streaming-eventhubs_examples_2.11

Filename	Size
spark-streaming-eventhubs_examples_2.11-2.1.4.pom
spark-streaming-eventhubs_examples_2.11-2.1.4.jar	1 MB
spark-streaming-eventhubs_examples_2.11-2.1.4-sources.jar	23 KB
spark-streaming-eventhubs_examples_2.11-2.1.4-javadoc.jar	2 KB
Browse

How to add to project

Apache Maven

<!-- https://jarcasting.com/artifacts/com.microsoft.azure/spark-streaming-eventhubs_examples_2.11/ -->
<dependency>
    <groupId>com.microsoft.azure</groupId>
    <artifactId>spark-streaming-eventhubs_examples_2.11</artifactId>
    <version>2.1.4</version>
</dependency>

Gradle Groovy

// https://jarcasting.com/artifacts/com.microsoft.azure/spark-streaming-eventhubs_examples_2.11/
implementation 'com.microsoft.azure:spark-streaming-eventhubs_examples_2.11:2.1.4'

Gradle Kotlin

// https://jarcasting.com/artifacts/com.microsoft.azure/spark-streaming-eventhubs_examples_2.11/
implementation ("com.microsoft.azure:spark-streaming-eventhubs_examples_2.11:2.1.4")

Apache Buildr

'com.microsoft.azure:spark-streaming-eventhubs_examples_2.11:jar:2.1.4'

Apache Ivy

<dependency org="com.microsoft.azure" name="spark-streaming-eventhubs_examples_2.11" rev="2.1.4">
  <artifact name="spark-streaming-eventhubs_examples_2.11" type="jar" />
</dependency>

Groovy Grape

@Grapes(
@Grab(group='com.microsoft.azure', module='spark-streaming-eventhubs_examples_2.11', version='2.1.4')
)

Scala SBT

libraryDependencies += "com.microsoft.azure" % "spark-streaming-eventhubs_examples_2.11" % "2.1.4"

Leiningen

[com.microsoft.azure/spark-streaming-eventhubs_examples_2.11 "2.1.4"]

Dependencies

compile (4)

Group / Artifact	Type	Version
org.apache.spark : spark-sql_2.11	jar	2.1.0
org.scala-lang.modules : scala-xml_2.11	jar	1.0.6
com.microsoft.azure : azure-eventhubs	jar	0.14.3
org.scalaj : scalaj-http_2.11	jar	2.3.0

provided (1)

Group / Artifact	Type	Version
org.apache.spark : spark-streaming_2.11	jar	2.1.0

test (10)

Group / Artifact	Type	Version
org.apache.spark : spark-core_2.11	test-jar	2.1.0
org.apache.spark : spark-sql_2.11	test-jar	2.1.0
org.apache.spark : spark-streaming_2.11	test-jar	2.1.0
org.scalacheck : scalacheck_2.11	jar	1.11.3
junit : junit	jar	4.12
org.mockito : mockito-core	jar	1.10.8
org.powermock : powermock-module-junit4 Optional	jar	1.6.1
org.powermock : powermock-api-mockito Optional	jar	1.6.1
org.scalatest : scalatest_2.11	jar	2.2.6
com.novocode : junit-interface	jar	0.10

Project Modules

There are no modules declared in this project.

Azure Event Hubs Connector for Apache Spark

This is the source code of the Azure Event Hubs Connector for Apache Spark.

Azure Event Hubs is a highly scalable publish-subscribe service that can ingest millions of events per second and stream them into multiple applications. Spark Streaming and Structured Streaming are scalable and fault-tolerant stream processing engines that allow users to process huge amounts of data using complex algorithms expressed with high-level functions like map, reduce, join, and window. This data can then be pushed to filesystems, databases, or even back to Event Hubs.

By making Event Hubs and Spark easier to use together, we hope this connector makes building scalable, fault-tolerant applications easier for our users.

Latest Releases

Spark

Spark Version	Package Name	Package Version
Spark 3.0	azure-eventhubs-spark_2.12
Spark 2.4	azure-eventhubs-spark_2.11
Spark 2.4	azure-eventhubs-spark_2.12
Spark 2.3	azure-eventhubs-spark_2.11
Spark 2.2	azure-eventhubs-spark_2.11
Spark 2.1	azure-eventhubs-spark_2.11

Databricks

Databricks Runtime Version	Artifact Id	Package Version
Databricks Runtime 7.X	azure-eventhubs-spark_2.12
Databricks Runtime 6.X	azure-eventhubs-spark_2.11
Databricks Runtime 6.X	azure-eventhubs-spark_2.12
Databricks Runtime 5.X	azure-eventhubs-spark_2.11
Databricks Runtime 5.X	azure-eventhubs-spark_2.12
Databricks Runtime 4.X	azure-eventhubs-spark_2.11
Databricks Runtime 3.5	azure-eventhubs-spark_2.11

Roadmap

There is an open issue for each planned feature/enhancement.

Usage

Linking

For Scala/Java applications using SBT/Maven project definitions, link your application with the artifact below. Note: See Latest Releases to find the correct artifact for your version of Apache Spark (or Databricks)!

groupId = com.microsoft.azure
artifactId = azure-eventhubs-spark_2.11
version = 2.3.18

groupId = com.microsoft.azure
artifactId = azure-eventhubs-spark_2.12
version = 2.3.18

Documentation

Documentation for our connector can be found here. The integration guides there contain all the information you need to use this library.

If you're new to Apache Spark and/or Event Hubs, then we highly recommend reading their documentation first. You can read Event Hubs documentation here, documentation for Spark Streaming here, and, the last but not least, Structured Streaming here.

FAQ

We maintain an FAQ - reach out to us via gitter if you think anything needs to be added or clarified!

Further Assistance

If you need additional assistance, please don't hesitate to ask! General questions and discussion should happen on our gitter chat. Please open an issue for bug reports and feature requests! Feedback, feature requests, bug reports, etc are all welcomed!

Contributing

If you'd like to help contribute (we'd love to have your help!), then go to our Contributor's Guide for more information.

Build Prerequisites

In order to use the connector, you need to have:

Java 1.8 SDK installed
Maven 3.x installed (or SBT version 1.x)

More details on building from source and running tests can be found in our Contributor's Guide.

Build Command

// Builds jar and runs all tests
mvn clean package

// Builds jar, runs all tests, and installs jar to your local maven repository
mvn clean install

HDInsight

Versions

Version
2.1.4 Sep 21, 2017
2.1.3 Sep 18, 2017
2.1.2 Jul 31, 2017
2.1.1 Jun 26, 2017
2.1.0 May 25, 2017
2.0.9 Sep 18, 2017
2.0.8 Jul 31, 2017
2.0.7 Jun 26, 2017
2.0.6 May 25, 2017
2.0.5 Apr 19, 2017
2.0.4 Mar 28, 2017
2.0.3 Jan 28, 2017