JarCasting
JarCasting JarCasting

storm-crawler-tika

Tika-based parser bolt for StormCrawler

License

License
Categories

Categories

ORM Data
GroupId

GroupId

com.digitalpebble
ArtifactId

ArtifactId

storm-crawler-tika
Last Version

Last Version

0.7
Release Date

Release Date
Type

Type

jar
Description

Description

storm-crawler-tika
Tika-based parser bolt for StormCrawler
Project Organization

Project Organization

DigitalPebble Ltd

Download storm-crawler-tika

Filename Size
storm-crawler-tika-0.7.pom
storm-crawler-tika-0.7.jar 14 KB
storm-crawler-tika-0.7-sources.jar 12 KB
storm-crawler-tika-0.7-javadoc.jar 49 KB
Browse

How to add to project

<!-- https://jarcasting.com/artifacts/com.digitalpebble/storm-crawler-tika/ -->
<dependency>
    <groupId>com.digitalpebble</groupId>
    <artifactId>storm-crawler-tika</artifactId>
    <version>0.7</version>
</dependency>
// https://jarcasting.com/artifacts/com.digitalpebble/storm-crawler-tika/
implementation 'com.digitalpebble:storm-crawler-tika:0.7'
// https://jarcasting.com/artifacts/com.digitalpebble/storm-crawler-tika/
implementation ("com.digitalpebble:storm-crawler-tika:0.7")
'com.digitalpebble:storm-crawler-tika:jar:0.7'
<dependency org="com.digitalpebble" name="storm-crawler-tika" rev="0.7">
  <artifact name="storm-crawler-tika" type="jar" />
</dependency>
@Grapes(
@Grab(group='com.digitalpebble', module='storm-crawler-tika', version='0.7')
)
libraryDependencies += "com.digitalpebble" % "storm-crawler-tika" % "0.7"
[com.digitalpebble/storm-crawler-tika "0.7"]

Dependencies

compile (2)

Group / Artifact Type Version
com.digitalpebble : storm-crawler-core jar 0.7
org.apache.tika : tika-parsers jar 1.11

provided (1)

Group / Artifact Type Version
org.apache.storm : storm-core jar 0.9.5

test (1)

Group / Artifact Type Version
junit : junit jar 4.11

Project Modules

There are no modules declared in this project.

storm-crawler

license

Build Status

StormCrawler is an open source collection of resources for building low-latency, scalable web crawlers on Apache Storm. It is provided under Apache License and is written mostly in Java.

Quickstart

NOTE: These instructions assume that you have Apache Maven installed.

The easiest way to get started is to generate a brand new StormCrawler-based project using :

mvn archetype:generate -DarchetypeGroupId=com.digitalpebble.stormcrawler -DarchetypeArtifactId=storm-crawler-archetype -DarchetypeVersion=1.17

You'll be asked to enter a groupId (e.g. com.mycompany.crawler), an artefactId (e.g. stormcrawler), a version and package name.

This will not only create a fully formed project containing a POM with the dependency above but also the default resource files, a default CrawlTopology class and a configuration file. Enter the directory you just created (should be the same as the artefactId you specified earlier) and follow the instructions on the README file.

Alternatively if you can't or don't want to use the Maven archetype above, you can simply copy the files from archetype-resources.

Have a look at the code of the CrawlTopology class, the crawler-conf.yaml file as well as the files in src/main/resources/, they are all that is needed to run a crawl topology : all the other components come from the core module.

Getting help

The WIKI is a good place to start your investigations but if you are stuck please use the DigitalPebble mailing list or use the tag stormcrawler on StackOverflow.

DigitalPebble Ltd provide commercial support and consulting for Storm-Crawler.

Thanks

alt tag

YourKit supports open source projects with its full-featured Java Profiler. YourKit, LLC is the creator of YourKit Java Profiler and YourKit .NET Profiler, innovative and intelligent tools for profiling Java and .NET applications.

We are very grateful to our sponsors for their continued support.

com.digitalpebble

DigitalPebble Ltd

Versions

Version
0.7
0.6