Easily Return Values From a Transaction with Speedment 3.0.17 or Later

Transactions

In my previous post, I wrote about how to use Transactions in an easy way using Speedment where we updated two bank accounts atomically. As you all might know, transactions are a way of combining a number of database operation into a single operation that is atomically executed.

But transactions are not only about updating the database but also about performing atomic reads. With Speedment, we can compute values atomically using Java streams and then return the result to something outside the scope of the transaction in an easy way.

Returning Transaction Values

Suppose we have a database with films and artists (e.g. using the open-sourced Sakila database content for MySQL) and we would like to count the number of films plus the number of artist in the database in a single transaction. This might be the case if we want to show some kind of database size in a GUI or the likes. Here is how it might look like:

    long sumCount = txHandler.createAndApply(tx -> 
        films.stream().count() + actors.stream().count()
    );

When the transaction starts, the view of the database is "frozen" so that new films or artist that are inserted into the database by other threads are not visible within the transaction. Thus, we can safely assume that our view is unaffected by other threads.

As can be seen, with really minimal boiler plate code, we can express functions that are to be performed within transactions using Speedment.

Preparations

Before transactions can be used, we need to obtain a TransactionHandler like this:

    SakilaApplication app = ....
    TransactionComponent transactionComponent = app.getOrThrow(TransactionComponent.class);
    TransactionHandler txHandler = transactionComponent.createTransactionHandler();

The FilmManager and the ActorManager can be retrieved from the application as shown hereunder:

    FilmManager films = app.getOrThrow(FilmManager.class);
    ActorManager actors = app.getOrThrow(ActorManager.class);

Read more on Speedment transactions here.

What's Next?

Visit Speedment open-source on GitHub and try it out!

Transactions Made Simple Using Speedment 3.0.17 and Above

Transactions

Sometimes we want to make sure that our database operations are executed atomically and separated from other operations. This is where transactions come into play. A transaction is a set of operation
proposals that may or may not be accepted by the database as an atomic operation. So, either all operations in the transaction are accepted or no operations of the transaction are accepted. Another advantage with transaction is that the state of the database will be locally "frozen" when the transaction starts, so we will not see updates by other threads while in the transaction.

Speedment is an open-source Stream ORM Java Toolkit and Runtime Java tool that wraps an existing database and its tables into Java 8 streams. Later versions of Speedment support database transactions in an easy-to-use way.

Updates

Imagine we are writing a bank application with accounts and we are going to move $100 from one account (1) to another (2). In this case it is important that money does not disappear (i.e. is deducted from 1 but never deposited in 2) or perhaps even worse, will be duplicated (i.e. deposited in 2 but not deducted from 1). This can be assured using a Speedment database transaction like this:

txHandler.createAndAccept(tx ->

    Account sender = accounts.stream()
        .filter(Account.ID.equal(1))
        .findAny()
        .get();

    Account receiver = accounts.stream()
        .filter(Account.ID.equal(2))
        .findAny()
        .get();

    accounts.update(sender.setBalance(sender.getBalance() - 100));
    accounts.update(receiver.setBalance(receiver.getBalance() + 100));

    tx.commit();
}

When the method tx.commit() is called, the two updates are committed to the database atomically and will be visible to all other threads. If we do not call tx.commit() explicitly then the transaction will be automatically rolled back (i.e. the updates will not have any effect and will be discarded).

Preparations

Before transactions can be used, we need to obtain a TransactionHandler like this:

    BankApplication app = ....
    TransactionComponent transactionComponent = app.getOrThrow(TransactionComponent.class);
    TransactionHandler txHandler = transactionComponent.createTransactionHandler();

The AccountManager can be retrieved from the application as shown hereunder:

    AccountManager accounts = app.getOrThrow(AccountManager.class);

What's Next?

Read more on Speedment transactions here.

Visit GitHub and read all about Speedment open-source here.

The Need for Speed, Access Existing Data 1,000x Faster

Learn how you can speed up your analytics database applications by a factor of 1,000 by using standard Java 8 streams and Speedment’s In-JVM-Memory accelerator.

Web and mobile applications are sometimes slow because the backing database is slow and/or the connection to the database imposes latencies. Modern UIs and interactive applications require fast back-ends with ideally no observable latency or else users will move on to other services or will just get tired and stop using the service altogether.

In this article we will learn how analytic database applications can be speed up by orders of magnitude using standard Java 8 streams and Speedment's in-JVM-memory acceleration technology. At the end, we will run a JMH test suit with representative benchmarks that indicate a speedup factor exceeding 1,000 times.

Viewing the Database as Streams

Speedment is a modern stream based ORM meaning that the tables are viewed as standard Java 8 streams. In this article we will use the “Sakila” database that is an open-source example database available directly from Oracle here. The Sakila example database contains films, actors, etcetera. This is how a Java 8 stream from the database might look like:

    List<Film> secondPage = films.stream()
        .filter(Film.RATING.equal("PG-13"))
        .sorted(Film.TITLE.comparator())
        .skip(50)
        .limit(50)
        .collect(Collectors.toList());

This stream will filter out only those films that has a rating equal to “PG-13” and will then sort the remaining films by film title. After that, the first 50 films are skipped and then the next 50 films are collected to a list. Thus, we get the second page of all PG-13 films sorted in title order. Usually, we also need to know how many films there are in total that has a rating of “PG-13” in order to show a correctly scaled scrollbar in our application. This can be done like this:

     long count = films.stream()
        .filter(Film.RATING.equal("PG-13"))
        .count();

Using a Database

Speedment will automatically render the Streams to SQL. This way, we can remain in a pure typesafe Java environment without having to write SQL code. By enabling logging, we can see that the first paging stream will be rendered to the following SQL query (assuming we are using MySQL):

 
SELECT
    `film_id`,`title`,`description``release_year`,
    `language_id`,`original_language_id`,`rental_duration`,`rental_rate`,
    `length`,`replacement_cost`,`rating`,`special_features`,`last_update` 
FROM 
    `sakila`.`film` 
WHERE 
    (`sakila`.`film`.`rating`  = ? COLLATE utf8_bin) 
ORDER BY 
    `sakila`.`film`.`title` ASC 
LIMIT ? OFFSET ?

values:[PG-13, 50, 50]

The second count stream will be rendered to:

 
SELECT COUNT(*) FROM (
    SELECT
        `film_id`,`title`,`description``release_year`,
        `language_id`,`original_language_id`,`rental_duration`,`rental_rate`,
        `length`,`replacement_cost`,`rating`,`special_features`,`last_update`
     FROM
             `sakila`.`film` 
        WHERE 
            (`sakila`.`film`.`rating`  = ? COLLATE utf8_bin)
) AS A

values:[PG-13]

Thus, the stream operations are rendered to efficient SQL. When running one thousand of these queries in parallel on a laptop class of computer with the MySQL standard server configuration, they complete in 700 ms and 175 ms aggregated latency respectively. If you are thinking about how the second SQL statement could be efficient, the fact is that the database will be able to basically eliminate the inner select.

Using In-JVM-Memory Acceleration

Now to the fun part. Let us activate the in-JVM-memory acceleration component in Speedment, called DataStore, in our application. This is done in this way:

     SakilaApplication app = new SakilaApplicationBuilder()
        .withPassword("sakila-password")
        // Activate DataStore
        .withBundle(DataStoreBundle.class)
        .build();

        // Load a snapshot of the database into off heap memory
        app.get(DataStoreComponent.class)
            .ifPresent(DataStoreComponent::load);

When the application is started, a snapshot of the database is pulled into the JVM and is stored off-heap. Because data is stored off-heap, data will not influence garbage collection and the amount of data is only limited by the available RAM. Nothing prevents us from loading terabytes of data if we have that amount of RAM available.

If we now run the same application again, we get 22 ms and 1 ms aggregated latency. This means that the latency is reduced by a factor of 30 and 170 respectively. A significant improvement it has to be said. But, it is getting better still.

Using In-JVM-Memory Acceleration and Json

REST and JSON are commonly used to serve clients that request data these days. Speedment has a special collector that can collect JSON data using something called in-place de-serialization whereby only the fields that are needed by the collector are deserialized from off-heap memory. We can depend on the Json plugin by first adding a dependency in our pom file:

     <dependency>
        <groupId>com.speedment.enterprise.plugins</groupId>
        <artifactId>json-stream</artifactId>
        <version>${speedment.enterprise.version}</version>
    </dependency>

Then, we install the plugin in the ApplicationBuilder as shown hereunder:

     SakilaApplication app = new SakilaApplicationBuilder()
        .withPassword("sakila-password")
        .withBundle(DataStoreBundle.class)
        // Install the Json Plugin
        .withBundle(JsonBundle.class)
        .build();

If we only want the Film fields “title”, “rating” and “length” in the json output, we can create a Json encoder like this:

    final JsonComponent json = app.getOrThrow(JsonComponent.class);

    final JsonEncoder<Film> filmEncoder = json.<Film>emptyEncoder()
        .put(Film.TITLE)
        .put(Film.RATING)
        .put(Film.LENGTH)
        .build();

This decoder is immutable and can be reused over and over again in our application:

     String json = films.stream()
        .filter(Film.RATING.equal("PG-13"))
        .sorted(Film.TITLE.comparator())
        .skip(50 * pageNo)
        .limit(50)
        .collect(JsonCollectors.toList(filmEncoder));

This gives us an additional speed up factor of 2 compared to dealing with entire entities. The JsonComponent can do much more than simply collecting things to a list. For example, it can create aggregations, also using in-place de-serialization.

Run Your Own Projects with In-JVM-Memory Acceleration

It is easy to try in-JVM-Memory acceleration by yourself. There is a free Initializer that can be found here. Just tick in your desired database type and you will get a POM and an application template automatically generated for you. You also need a license key to run. Just click “Request a Free Trial License Key” on the same page to get one. If you need more help setting up your project, check out the Speedment GitHub page or explore the manual.

How Fast is it for Real?

Speedment supports a number of database types including Oracle, MySQL, MariaDB, PostgreSQL, Microsoft SQL Server, DB2 and AS400. Speedment can also work with Avro files that is used by Hadoop. In this example, we will run MySQL.

Testing performance in Java application is notoriously difficult. Using the JMH framework, I have written a number of typical applications and have run each test hundreds of thousand times and compared the result for pure MySQL and MySQL with Speedment’s in-JVM accelerator. The performance figures below are given as operations/second (higher is better).

Benchmark	Pure MySQL	MySQL with Speedment in-JVM	Speed up factor
Count all	5,324	43,615,967	8,000
Count with filter	5,107	2,465,928	400
Filtering	449	597,702	1,300
Sorting	109	171,304	1,500
Paging	1,547	1,443,015	900
Iterate all	108	5,556	50
Aggregation	117	167,728	1,400
Aggregation filter	453	608,763	1,300

As can be seen, MySQL with Speedment In-JVM accelerator outperforms Pure MySQL by a factor of 1,000 or more in most cases. The smallest speed up factor observed was 50 times which is still very good.

Test Environment

MySQL, 5.7.16 standard installation, MySQL JDBC Driver 5.1.42, Oracle Java 1.8.0_131, Speedment Enterprise 1.1.10, macOS Sierra 10.12.6, Macbook Pro 2.2 GHz i7 (mid 2015), 16 GB RAM.

Benchmark Code

Below are some examples of how the benchmark code looks like. The complete benchmark application can be found on GitHub here. I encourage you to clone it an run it to see the speedup factor on your own target machine.

    @Benchmark
    public String paging() {
        return films.stream()
            .filter(Film.RATING.equal("PG-13"))
            .skip(50)
            .limit(50)
            .collect(filmCollector);
    }


    @Benchmark
    public String aggregationWithFilter() {
        return films.stream()
            .filter(Film.RATING.equal("PG-13"))
            .collect(sumLengthCollector);
    }

How Much RAM is Needed?

Speedment can often store data in RAM more efficiently than the database itself. The Sakila database in the benchmark takes 6.6 MB on disk but Speedment is only using 3 MB of memory. Considering that Speedment indexes all columns by default whereas the database only indexes a few columns, the Speedment is remarkably memory efficient.

How Long Time Does it Take to Load Data?

The Sakila database was loaded and indexed by Speedment in less than 1 second. Speedment can refresh data from the database in the background and will keep track of which streams are running against which database snapshot version (MVCC).

How Much Faster Will My Own Applications Run?

How much latency reduction one would get in a particular project is anyone's guess. Is is x10, x50, x100 or even more? Take the opportunity and find out how much speed you can gain in your own projects!

Take it for a Spin

Learn more about Speedment on GitHub and kick-start your own project using the Speedment Initializer and remember to tick “enable in-memory acceleration” and use the initializer to get a free evaluation license key too. Explore the manual section for the Speedment in-JVM accelerator here or use my twitter handle @PMinborg

Use Precise Java Method Parameters

Learn how to pick the right method parameter types and get more robust and shorter code in your Java applications.

We Java developers generally have a bad habit of using method parameters without thinking of what is actually needed and just picking whatever we are used to, what we have available or whatever first comes into mind. Consider the following representative example:

 
    private static String poem(Map<Integer, String> numberToWord) {
        return new StringBuilder()
            .append("There can be only ")
            .append(numberToWord.get(1))
            .append(" of you.\n")
            .append("Harts are better off when there are ")
            .append(numberToWord.get(2))
            .append(" of them together.\n")
            .append("These ")
            .append(numberToWord.get(3))
            .append(" red roses are a symbol of my love to you.\n")
            .toString();
    }

When we use the method above, we provide a Map that translates from a number to a String. We might, for example, provide the following map:

    Map<Integer, String> englishMap = new HashMap<>();
        englishMap.put(1, "one");
        englishMap.put(2, "two");
        englishMap.put(3, "three");

When we call our poem method with the englishMap then the method will produce the following output:

 
There can be only one of you.
Harts are better off when there are two of them together.
These three red roses are a symbol of my love to you.

That sounds good. Now suppose that your significant other is a computer nerd and you want to spice up your poem and make an impression, then this is the way to go:

    Map<Integer, String> nerdMap = new HashMap<>();
        nerdMap.put(1, "1");
        nerdMap.put(2, "10");
        nerdMap.put(3, "11");

If we now submit our nerdMap to the poem method, it will produce the following poem:

 
There can be only 1 of you.
Harts are better off when there are 10 of them together.
These 11 red roses are a symbol of my love to you.

As with all poems, it is difficult to judge which poem is more romantic that the other but I certainly have my own personal view.

The Problems

There are several problems with the solution above:

First of all, as an outside caller, we cannot be sure that the poem method does not change the Map we provide. After all, we provide a Map and there is nothing preventing a receiver to do whatever possible with the map, even clearing the entire map altogether. This can of course be avoided by wrapping the Map using the Collections.unmodifiableMap() method or provide a copy of an existing map whereby the copy is later discarded.

Secondly, we are tied to use a Map when we only need something that translates from an integer to String. This might create unnecessary code in some cases. Think back of our nerdMap, where the values in the map could easily be computed using the Integer::toBinaryString instead of mapping them manually.

The Solution

We should strive to provide precisely what is needed in any given situation and not more. In our example we should modify the poem method to take a function that goes from an integer to a String. How this function is implemented on the caller side is of less importance, it can be a map or a function, or code or something else. Here is how it should be done in the first place:

 
    private static String poem(IntFunction<String> numberToWord) {
        return new StringBuilder()
            .append("There can be only ")
            .append(numberToWord.apply(1))
            .append(" of you.\n")
            .append("Harts are better off when there are ")
            .append(numberToWord.apply(2))
            .append(" of them together.\n")
            .append("These ")
            .append(numberToWord.apply(3))
            .append(" red roses are a symbol of my love to you.\n")
            .toString();
    }
 

If we want to use the poem method with a Map, we simply call it like this:

 
    // Expose only the Map::get method
    System.out.println(poem(englishMap::get));

If we want to compute the values like we did for the the nerd poem then we can do it even simpler:

    System.out.println(poem(Integer::toBinaryString));

Heck, we can even produce a poem to a significant other suffering from a dual personality disorder like this:

 
    System.out.println(
        poem(
            no -> englishMap.getOrDefault(no + 1, Integer.toString(no + 1))
        )
    );

This will produce the following poem:

  
There can be only two of you.
Harts are better off when there are three of them together.
These 4 red roses are a symbol of my love to you.

Be careful with your method parameters!