Aliyun LOG Java Producer (deprecated)

中文版README

This project is no longer maintained. Please use the new version of Aliyun LOG Java Producer.

The Aliyun LOG Java Producer is a high concurrency writing library written for Java applications. The Producer Library and Consumer Library is a reading and writing encapsulation for Loghub. It aims to reduce the threshold for data collection and consumption.

Feature

• Provide an asynchronous sending interface, thread safety.
• You can configure more than one project.
• You can control the thread number for sending data.
• You can control the number and size of logs being merged into one package.
• You can control the memory usage. When the memory usage reach the threshold configured by user, the producer's send interface will be blocked until the free memory is available.

Advantage

• Disk Free: the generation data will be send to AliCloud Log Service in real time through network.
• High Throughput: support more than 100 write operations per second.
• The Computing and IO is separated: recoding logs do not affect computing time.

In the above scenarios, the Aliyun LOG Java Producer will alleviate the burden of your development. You don't need to care about log collection implementation details. And there is no need to worry about that log collection will impact on the normal operation of your business. Reducing the threshold of data acquisition greatly.

The comparison of all kinds of access methods

Configuration Steps

1. Adding the Dependencies in pom.xml

<dependency>
	<groupId>com.google.protobuf</groupId>
	<artifactId>protobuf-java</artifactId>
	<version>2.5.0</version>
</dependency>
<dependency>
	<groupId>com.aliyun.openservices</groupId>
 	<artifactId>aliyun-log</artifactId>
	<version>0.6.27</version>
</dependency>
<dependency>
	<groupId>com.aliyun.openservices</groupId>
	<artifactId>log-loghub-producer</artifactId>
	<version>0.1.16</version>
</dependency>

2. Configure ProducerConfig

The configuration format is as follows. Please refer to the parameter description part for parameter value.

public class ProducerConfig {
    // Specify the timeout for sending package, in milliseconds, default is 3000
    public int packageTimeoutInMS = 3000;
    // Specify the maximum log count per package, the upper limit is 4096
    public int logsCountPerPackage = 4096;
    // Specify the maximum cache size per package, the upper limit is 3MB, in bytes
    public int logsBytesPerPackage = 3 * 1024 * 1024;
    // The upper limit of the memory that can be used by each producer instance, in bytes, default is 100MB
    public int memPoolSizeInByte = 100 * 1024 * 1024;
    // If shardHash is specified when you send data, you should care about this parameter, otherwise there is no need to care about it.
    // The backend thread will merge the data being sent to the same shard together, and shard is associated with a hash interval.
    // The producer will pull the hash interval information for each shard from AliCloud Log Service regularly and update the local value, this parameter stands for the time interval.
    public int shardHashUpdateIntervalInMS = 10 * 60 * 1000;
    // Specify the retry times, default is 10. If exceeds this value, the exception will pass to the callback.
    public int retryTimes = 10;
    // protobuf
    public String logsFormat = "protobuf";
    // Specify the I/O thread pool's maximum pool size, the main function of the I/O thread pool is to send data, default is 8
    public int maxIOThreadSizeInPool = 8;
    // userAgent
    public String userAgent = "loghub-producer-java";
}

3. Extends ILogCallback

The main function of callback is to handle the results of sending data. The results include successful status or exception. If you don't care about the result, there is no need to extends ILogCallback.

Parameter Description

Sample Code

ProducerSample.java

CallbackSample.java

RAM

If you want to write data to logstore through Aliyun LOG Java Producer, you should configure the following permissions for the account you use. RAM doc

Aliyun Log Java SDK

If the interface provided by producer can't meet your log collection requirements, you can develop your own log collection API based on Aliyun Log Java SDK.

Contact Us

• Alicloud Log Service homepage
• Alicloud Log Service doc
• Alicloud Log Servic official forum
• Alicloud Log Servic official technical support: submit tickets

Contributors

@zzboy made a great contribution to this project.

Thanks for the excellent work by @zzboy