Spring Data Elasticsearch Rest

Spring Data implementation for ElasticSearch based on Elasticsearch Rest client

Introduction

The spring-data-elasticsearch-rest library is a project based on the spring-data library and offers document indexing management and a reading API for elasticsearch using the REST client provided by elasticsearch. The current version of this library does not allow to perform a mapping of documents by annotations. The mapping of a document must be done by files describing the mapping of a document.

The library supports the following types of indexes:

• Simple index (index name used for write and read operations) with a json file (optional) describing the index (mapping, settings, etc ...)
• Simple index with an alias (the alias is used for read operations, the index for write operations) with a json file (optional) describing the index (mapping, settings, alias, etc ... )
• Time based index (Index based on the timestamp). Read operations are done via an alias (mandatory) and write operations on a specified index at the time of indexing the document. The definition of a template is mandatory.
• Rollover index. Read operations are done via an alias and write operations are done on a write alias.

Versions

Maven dependency

<dependency>
    <groupId>com.github.ydespreaux.spring.data</groupId>
    <artifactId>spring-data-elasticsearch-rest-starter</artifactId>
    <version>1.0.1</version>
</dependency>

Mapping domain class

Field annotations

@Id / @Version

The org.springframework.data.annotation.Id annotation is used to declare an attribute representing the identifier of the document. The type of the attribute must be of type String. An annotated @Id attribute is not stored in the document (stored in the _id metadata).

public class MyBean {
    @Id
    private String id;
}

The annotation org.springframework.data.annotation.Version is used to declare an attribute representing the version of the document. The type of the attribute must be of type Long. An annotated @Version attribute is not stored in the document (stored in the metadata version).

public class MyBean {
    @Version
    private Long version;
}

@IndexName

The IndexName annotation is used to declare an attribute representing the name of the index in which the document is indexed. The type of the attribute must be of type String. An annotated attribute @IndexName is not stored in the document (stored in the metada _index), it's initialized when reading a document. An annotated attribute @IndexName is read-only mode.

public class MyBean {
    @IndexName
    private String indexName;
}

@Score

The Score annotation is used to inject the _score value of a document during a search (without sorting). The type of the attribute must be Float type. An annotated attribute @Score is read-only mode.

public class MyBean {
    @Score
    private Float score;
}

@CompletionField

The CompletionField annotation allows the use of elasticsearch completion. The completion offers a feature of autocomplete / search on demand. It is a navigation feature that guides users to relevant results during typing, improving the accuracy of the search. The type of the attribute must be of type Completion or Collection.

public class MyBean {
    @CompletionField
    private Completion suggest;
}

The mapping of the entity must be declared with the completion type.

{
  "mappings": {
    "myType": {
      "properties": {
        "suggest": {
          "type": "completion"
        }
      }
    }
  }
}

@ParentId (version 1.0.2)

The ParentId annotation defines an attribute containing the identifier of the parent document. This annotaton can not be used in an @Child annotated document. The type of the attribute must be String type.

@Child(type = "child_type")
public class MyBean extends ParentBean{
    @ParentId
    private String parentId;
}

@ScriptedField (version 1.0.2)

The ScriptedField annotation defines an attribute whose value matches a custom expression. This attribute is not added to the document when it is indexed.

@IndexedDocument(
        index = @Index(
                name = "my_bean",
                type = "_doc"
        )
)
public class MyBean {

    private String name;
    
    @ScriptedField
    private String scriptedName;
}

Sample query:

SearchQuery searchQuery = new NativeSearchQuery.NativeSearchQueryBuilder()
        .withQuery(matchAllQuery())
        .withScriptField(new ScriptField("scriptedName",
                new Script(ScriptType.INLINE, "painless", "doc['name'].value.toUpperCase()", Collections.emptyMap())))
        .build();
List<MyBean> entities = this.operations.search(searchQuery, MyBean.class);

Geo Shape data type

The use of GeoShape types requires the addition of the following dependencies:

<dependency>
    <groupId>org.locationtech.spatial4j</groupId>
    <artifactId>spatial4j</artifactId>
    <version>0.7</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.locationtech.jts</groupId>
    <artifactId>jts-core</artifactId>
    <version>1.16.1</version>
</dependency>

Description of documents

@IndexedDocument

The @IndexedDocument annotation allows you to define elasticsearch documents for the following index types:

• Simple index with or without alias
• Time-based index

Attributes description

@IndexedDocument

The alias attribute is required for a time-based index.

@Index

@Alias

The example below defines a document with an simple index, an alias, and a json mapping file

@IndexedDocument(
        alias = @Alias(name = "read_products"),
        index = @Index(
                name = "write_products",
                type = "product",
                settingsAndMappingPath = "classpath:indices/product.json"
        )
)
public class Product {

    @Id
    private String id;
    private List<String> title;
    private String name;
    private String description;
    private String text;
    private List<String> categories;
    private Float price;
    private Integer popularity;
    private boolean available;
}

Simple index

Below is the corresponding json mapping file product.json:

{
    "settings" : {
        "number_of_shards" : 1,
        "refresh_interval" : "1s",
        "number_of_replicas" : 1,
        "store.type": "fs"
    },
    "aliases" : {
        "write_products" : {}
    },
    "mappings": {
      "product": {
        "properties": {
          "description": {
            "type": "text"
          },
          "text": {
            "type": "text"
          },
          "categories": {
            "type": "text"
          },
          "price": {
            "type": "double"
          },
          "popularity": {
            "type": "integer"
          },
          "available": {
            "type": "boolean"
          }
        }
      }
    }
}

Index Time-based

Declaring a document using time-based indexes requires the definition of a template and an alias for read operations. The index used during write operations is determined by a specified pattern. The template must contain the attributes "index_patterns", "aliases" . The attributes "settings", "order" and "mapping" are optional.

Templates can be managed by the application or manually using kibana or curl queries for instance.

Sample template managed by the application:

Adding the article.template file in json format to the src/resources/templates directory:

{
    "index_patterns": "article-*",
    "settings" : {
        "number_of_shards" : 1,
        "refresh_interval" : "1s",
        "number_of_replicas" : 1,
        "store.type": "fs"
    },
    "aliases" : {
        "articles" : {}
    },
    "mappings": {
      "article": {
        "properties": {
          "name": {
            "type": "text",
            "copy_to": "search_fields",
            "fielddata": true
          },
          "description": {
            "type": "text",
            "copy_to": "search_fields"
          },
          "entrepot": {
            "type": "text",
            "copy_to": "search_fields",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "search_fields": {
            "type": "text"
          }
        }
      }
    }
}

The extension of the template file can be any extension. The name of the template is the name of the file.

Adding configuration settings to the application.properties or application.yml file:

spring:
  elasticsearch:
    template:
      action: CREATE
      scripts: classpath:templates/article.template

Attribute 'spring.elasticsearch.template.action' : defines the action to be performed on the templates. The possible actions are:

• NONE: No template will be added to the elasticsearch server
• CREATE: If the template does not exist on the server then the template is created
• CREATE_OR_UPDATE: The template is either created or updated.

Attribute 'spring.elasticsearch.template.scripts' : set the list of templates (separated by commas)

The example below defines a Java bean Article with an index time-based:

@IndexedDocument(
        alias = @Alias(name = "${spring.elasticsearch.aliases.article"),
        index = @Index(
                type = "article",
                indexPattern = "${spring.elasticsearch.index-pattern.article}",
                indexTimeBasedSupport = ArticleTimeBasedSupport.class
        )
)
public class Article {

    @Id
    private String documentId;
    @Version
    private Long documentVersion;
    private String name;
    private String description;
    private EnumEntrepot entrepot;

    public enum EnumEntrepot {
        E1, E2, E3, UNKNOWN
    }

}

The value of the attributes of the annotation set can be externalized in the springboot configuration file.

spring:
  elasticsearch:
    template:
      action: CREATE
      scripts: classpath:templates/article.template
    index-pattern:
      article: "'article-%s-'yyyy"
    aliases:
      article: articles

Items are indexed based on the current year and the name of the warehouse. For example, the name of an article index will look like 'article-e1-2019'.

By default, the library takes into account the date expressions. If the pattern takes into account attributes of the document, it is necessary to define a class that extends from the class IndexTimeBasedSupport.

public class ArticleTimeBasedSupport extends IndexTimeBasedSupport<Article> {
    /**
     * Génère le nom d'un index de type time-based en fonction d'une date et d'un document à indexer.
     *
     * @param indexParameter
     * @return
     */
    @Override
    public String buildIndex(IndexTimeBasedParameter<Article> indexParameter) {
        return String.format(indexParameter.generateIndexWithTimeEvent(),
                indexParameter.getDocument() == null ? Article.EnumEntrepot.UNKNOWN : indexParameter.getDocument().getEntrepot())
                .toLowerCase();
    }
}

@RolloverDocument

Declaring a document using indexes rollover requires the definition of an alias for read operations and an another alias for write operations.

@Getter
@Setter
@RolloverDocument(
        alias = @Alias(name = "read-tracks"),
        index = @Index(name = "tracks"),
        rollover = @Rollover(
                alias = @Alias(name = "write-tracks"),
                maxSize = "10gb",
                trigger = @Trigger(true)
        )
)
public class Track {

    private String name;
    private Integer number;
    private Integer length;

}

Manage index rollover

If you have the x-pack plugin then using policies to manage index rollover.

If not, you can enable a trigger as in the above example using the trigger attribute. The @Trigger annotation has the following attributes:

@Parent / @Child (version 1.0.2)

Parent/Child annotation defines a parent-child relationship of a document.

The parent's document and the child's document must be contained in the same index and must have the same type.

A parent document must be annotated with the @Parent annotation. A child document must be annotated with the @Child annotation. The Java bean representing the child document must extend from the Java bean representing the parent.

In order to define a parent-child relationship, the document must have an attribute of type Join.

The mapping defines the relations between parent and child as below:

{
  "mappings": {
    "myType": {
      "properties": {
        "description" : {
          "type": "text"
        },
        "join_field": {
          "type": "join",
          "relations": {
            "question": "answer",
            "answer": "vote"
          }
        }
      }
     }
  }
}

Here is an example describing a parent document:

@IndexedDocument(index=@Index(name="my_index", type="my_type", settingsAndMappingPath = "classpath:scripts/my_index.index"))
@Parent(name="join_field", type="question")
public class Question {
    private String description;
}

When indexing a child document, the join attribute is automatically injected into the source of the document as follow:

PUT my_index/my_type/1
{
  "description": "This is an question",
  "join_field": {
    "name": "question"
  }
}

Here is an example describing a child document:

@Child(type="answer", routing = "1")
public class Answer extends Question{

    @ParentId
    private String parentId;
}

The routing value is mandatory because parent and child documents must be indexed on the same shard. For the same reason, the class of the child is not annotated by @IndexedDocument because the child inherits the mapping from the parent.

The attribute parentId is not mapping. The type of the attribute must be of type String. When indexing a child document, the value of the attribute is automatically injected into the source of the document as follow:

PUT my_index/my_type/2?routing=1 
{
  "description": "This is an answer",
  "join_field": {
    "name": "answer", 
    "parent": "1" 
  }
}

If multiple parent join levels are defined, the child document must be annotated with the child annotation and the isParent attribute must be true.

Here is an example describing a child document:

@Child(type="vote", routing = "1", isParent = true)
public class Vote extends Answer{

    private Integer stars;
}

Elasticsearch Repositories

Introduction

Repository support can be enabled by annotating through JavaConfig.

@Configuration
@EnableElasticsearchRepositories(basePackages = {
        "com.github.ydespreaux.sample.elasticsearch.repositories"
})
public class ElasticsearchConfiguration {
}

Repositories are enabled using the @EnableElasticsearchRepositories annotation, which has essentially the same attributes as the XML namespace. If no base package is configured, it will use the one in which the configuration class resides.

Query methods

Query lookup strategies

The Elasticsearch module supports all basic query building feature as String,Abstract,Criteria or have it being derived from the method name.

Declared queries Deriving the query from the method name is not always sufficient and/or may result in unreadable method names. In this case one might make either use of @Query annotation (see Using @Query Annotation ).

Query creation

Generally the query creation mechanism for Elasticsearch works as described in Query methods . Here’s a short example of what a Elasticsearch query method translates into:

public interface BookRepository extends ElasticsearchRepository<Book, String> {

    Book findByName(String value);
}

The method name above will be translated into the following Elasticsearch json query:

{"bool" : {"must" : {"field" : {"name" : "?"}}}}

A list of supported keywords for Elasticsearch is shown below.

Using @Query Annotation

Declare query at the method using the @Query annotation.

public interface ArticleRepository extends ElasticsearchRepository<Article, String> {
    @Query("{\"match\" : {\"entrepot\" : \"?0\"}}")
    List<Article> findByEntrepot(EnumEntrepot entrepot);
}

Miscellaneous Elasticsearch Operation Support

This chapter covers additional support for Elasticsearch operations that cannot be directly accessed via the repository interface. It is recommended to add those operations as custom implementation as described in Custom Implementations for Spring Data Repositories.

Filter Builder

Filter Builder improves query speed.

private ElasticsearchTemplate elasticsearchTemplate;

SearchQuery searchQuery = new NativeSearchQueryBuilder()
    .withQuery(matchAllQuery())
    .withFilter(boolFilter().must(termFilter("id", documentId)))
    .build();

List<SampleEntity> sampleEntities =
    elasticsearchTemplate.search(searchQuery,SampleEntity.class);

Using Scroll For Big Result Set

Elasticsearch has a scroll API for getting big result set in chunks. ElasticsearchTemplate has startScroll and continueScroll methods that can be used as below.

Using startScroll and continueScroll

SearchQuery searchQuery = new NativeSearchQueryBuilder()
    .withQuery(matchAllQuery())
    .withIndices(INDEX_NAME)
    .withTypes(TYPE_NAME)
    .withFields("message")
    .withPageable(PageRequest.of(0, 10))
    .build();

Page<SampleEntity> scroll = elasticsearchTemplate.startScroll(Duration.ofSeconds(60), searchQuery, SampleEntity.class);

String scrollId = ((ScrolledPage) scroll).getScrollId();
List<SampleEntity> sampleEntities = new ArrayList<>();
while (scroll.hasContent()) {
    sampleEntities.addAll(scroll.getContent());
    scrollId = ((ScrolledPage) scroll).getScrollId();
    scroll = elasticsearchTemplate.continueScroll(scrollId, Duration.ofSeconds(60), SampleEntity.class);
}
elasticsearchTemplate.clearScroll(scrollId);

Samples

https://github.com/ydespreaux/sample-project/tree/master/sample-spring-data-elasticsearch-rest