I frequently find myself wanting a light append-only data store for pet projects, so I’ve recently released and published a small library called java-dirty to accomplish that. Each store is backed onto a file and uses Java’s NIO MappedByteBuffers for fast reads and writes.
Implementation
As an implementation choice, java-dirty only supports primitive fields on objects. MappedByteBuffer provides e.g. .putLong() to directly write out a long into the file.
Creating a store is easy, and takes a Class<?> as a parameter.
Store<Event> eventStore = Store.of(Event.class).from("/path/to/file");
The store is aware of the offset of an object in bytes, which it uses to both write and read objects. In addition to this, it keeps track of an additional integer - the number of entries in the store. This lets a store be re-opened and know the number of objects currently written to it.
To give an example, here is an Event class declaration
class Event {
public int one, two;
}
A store of event objects is laid out like the illustration below. 4 bytes are set aside at the start for the size of the store, and 8 bytes per object (4 bytes for each integer field).
On an SSD, java-dirty can store objects at around 4 million insertions/s, and read at a similar speed. I will be publishing the benchmarking code for these results shortly alongside the project.
Source is available on Github and the artifact is published on Maven Central.
You can include it with the following Maven coordinates
<dependency>
<groupId>uk.co.probablyfine</groupId>
<artifactId>java-dirty</artifactId>
<version>1.4</version>
</dependency>