Minborg

Minborg
Minborg
Showing posts with label Stream. Show all posts
Showing posts with label Stream. Show all posts

Thursday, October 1, 2020

How to get Type-Safe and Intuitive Hibernate/JPA Queries by Leveraging Java Streams

A large proportion of Java database applications are using Hibernate/JPA to bridge the gap between Java and SQL. Until recently, we were forced to mix Java and JPQL or to use complex imperative criteria builders to create database queries. Both of these methods are inherently neither type-safe nor very intuitive.

The newly launched open-source library JPAstreamer addresses these issues by allowing you to express Hibernate/JPA queries using Java Streams. This means we can avoid any impedance mismatches between JPQL/HQL and Java and get full type-safety. In this article, I will show you how to put Java Stream queries to work in your application using JPAstreamer.

JPAstreamer in a nutshell

As mentioned, JPAstreamer allows JPA queries to be expressed as standard Java Streams using short and concise, type-safe declarative constructs. This makes our code shorter, less complex, and easier to read and maintain. Best of all, we can stick to using only Java code without needing to mix it with SQL/JPQL or other language constructs/DSL. 

In short, we can query a database like this:

jpaStreamer.stream(Film.class)
.sorted(Film$.length.reversed())
.limit(15)
.map(Film$.title)
.forEach(System.out::println);

This prints the title of the 15 longest films in the database. 

OSS License 

JPAstreamer is using the same license as Hibernate (LGPL). This makes it easy to use in existing Hibernate projects. JPAstreamer also works with other JPA providers such as EclipseLink, OpenJPA, TopLink etc.

Installation

Installing JPAstreamer entails just adding a single dependency in your Maven/Gradle configuration file as described here. For example, Maven users add the following dependency:


    <dependency>
        <groupId>com.speedment.jpastreamer</groupId>
        <artifactId>jpastreamer-core</artifactId>
        <version>0.1.8</version>
</dependency>

Let’s have a look at how JPAstreamer fits in an existing application.

Example Database and JPA Entities

In the examples below, we are using JPAstreamer to query the “Sakila” example database that is available for download directly from Oracle or as a Docker instance.

This is how you install and run the example database using Docker:

 
$ docker pull restsql/mysql-sakila
$ docker run -d --publish 3306:3306 --name mysqld restsql/mysql-sakila

We will also be relying on JPA entities like the Film-class partly shown here:

@Entity
@Table(name = "film", schema = "sakila")
public class Film implements Serializable {

@Id

@GeneratedValue(strategy = GenerationType.IDENTITY)
@Column(name = "film_id", nullable = false, updatable = false, columnDefinition = "smallint(5)")
private Integer filmId;

@Column(name = "title", nullable = false, columnDefinition = "varchar(255)")
private String title;

@Column(name = "description", nullable = false, columnDefinition = "text")
private String description;

@ManyToMany(cascade = CascadeType.ALL)
@JoinTable(
name = "film_actor",
joinColumns = @JoinColumn(name = "film_id") ,
inverseJoinColumns = @JoinColumn(name = "actor_id")
    )
private List<Artist> actors;
...
}

The complete code in this article is open-sourced and available here

JPAstreamer - Printing the Longest Films

Here is a complete example of how we can use JPAstreamer to create a query that prints out the length and title of the 15 longest films in the database:

public class LongestFilms  {
public static void main(String[] args) {
final JPAStreamer jpaStreamer = JPAStreamer.of("sakila");
jpaStreamer.stream(Film.class)
.sorted(Film$.length.reversed())
.limit(15)
.map(f -> String.format("%3d %s", f.getLength(), f.getTitle()))
.forEach(System.out::println);

jpaStreamer.close();
}
 }

This will print:


185 SOLDIERS EVOLUTION
185 GANGS PRIDE
185 SWEET BROTHERHOOD
185 CHICAGO NORTH
185 HOME PITY
185 POND SEATTLE
185 CONTROL ANTHEM
185 DARN FORRESTER
185 WORST BANGER
184 SMOOCHY CONTROL
184 SONS INTERVIEW
184 SORORITY QUEEN
184 MOONWALKER FOOL
184 THEORY MERMAID

As can be seen, queries are simple, concise, completely type-safe and follow the Java Stream standard API. No need to learn new stuff.

The code above will create the following SQL (shortened for brevity):

select
film0_.film_id as film_id1_1_,
film0_.length as length4_1_,
film0_.title as title10_1_,
/* more columns */
from
film film0_
order by
film0_.length desc limit ?

This means that most of the Java stream is actually executed on the database side. It is only the map() and forEach() operations (which cannot easily be translated to SQL) that are executed in the JVM. This is really cool!

Pre-Joining Columns

To avoid the “SELECT N + 1” problem, it is possible to configure streams to join in columns eagerly by providing a configuration object like this:

StreamConfiguration configuration = StreamConfiguration.of(Film.class)
.joining(Film$.actors)
.joining(Film$.language);

jpaStreamer.stream(configuration)
.filter(Film$.rating.in("G", "PG"))
.forEach(System.out::println);

This will create a Hibernate join under the hood and will only render a single SQL query where all the Film fields “List<Artist> artists” and “Language language” will be populated on the fly:


select
Film
from
Film as Film
left join
fetch Film.actors as generatedAlias0
left join
fetch Film.language as GeneratedAlias1
where
Film.rating in (
:param0, :param1
)

Conclusion

In this article, I have shown how you can avoid impedance mismatches between JPQL/HQL in Hibernate/JPA using the open-source library JPAstreamer. The Stream API allows you to compose type-safe and expressive database queries in standard Java without compromising the application performance. 

Feedback

The background to JPAStreamer is that we have developed the stream-based ORM-tool Speedment, and we have come across many developers that want to use Java streams but are constrained to use Hibernate in their applications. Therefore, we have now developed JPAstreamer, a JPA/Hibernate extension that handles Java Stream queries without the need to change the existing codebase. 

Take JPAStreamer for a spin and let me know what you like/dislike by dropping a message on Gitter!

Resources

Friday, November 15, 2019

Become a Master of Java Streams - Part 6: Creating a New Database Application Using Streams

Have you ever wanted to develop an "express" version of your database application? In this Hands-On Lab article, you will learn a truly easy and straightforward method. The entire Java domain model will be automatically generated for you. You just connect to your existing database and then start developing using Java streams. You will be able to create, for example, a new web application for your existing database in minutes.

This article is the last article in the series on How to Become a Master of Java Streams.

Part 1: Creating Streams
Part 2: Intermediate Operations
Part 3: Terminal Operations
Part 4: Database Streams
Part 5: Turn Joined Database Tables Into a Stream
Part 6: Creating a Database Application Using Streams

So far, you got to experience Speedment in the articles and through the exercises. For brevity, we did not include any descriptions on how to start from scratch but rather wanted you to get a glimpse of what using Java Streams with databases could look like. In this article, we’ll show you how to leverage Speedment for applications running against any of your databases. Setup only takes a few minutes but will save you vasts amounts of time due to the expressiveness of Streams and the provided type-safety.

Getting Started 

To help you configure your project, Speedment provides a project Initializer. Once you fill out the details of your project, it provides you with a zip-file containing a pom.xml with the needed dependencies and a Main.java starter.



The Speedment Initializer can be used to configure a Speedment project. 

Once you have clicked “download”, unzip the file and open the project in your IDE as a Maven project. In IntelliJ, the easiest way to do that is to choose File -> Open and then select the pom.xml-file in the unzipped project folder.

If you rather want to use Speedment in an existing project, configure your project via the Initializer to make sure you get all needed dependencies. Then simply merge the provided pom.xml with your current one and reimport Maven.

As you may recall from the previous articles, Speedment relies on an automatically generated Java domain model. Hence, before we can write our application, we need to generate the required classes. This is done using the Speedment Tool which is started by running mvn speedment:tool in the terminal or by running the same target via the IDE:s built-in Maven menu.

Firstly, you will be asked to register for a free license and connect to your database. A free license can be used for all open-source databases (unlimited use) and commercial databases (up to 500 MB and doesn’t require any billing information).
A free license can be used with all open-source databases (unlimited) and commercial databases (up to 500 MB and does not require billing information.) 

Once you complete the registration, you will be asked to provide credentials for your database (make sure you selected the correct DB-type in the initializer). Either use a local database of your own or run some tests with the Sakila database we used in the exercises.

Sakila Database Credentials 
Type: MariaDB
Host: 35.203.190.83
Port: 3306
Database name: sakila
User: sakila
Password: sakila
Fill out the database credentials to connect to your data source. (Note: Speedment never stores your database password). 

 A click on the “Connect”-button will launch the Speedment Tool. It presents the database structure to the left-hand side and settings for the selected table or column on the right-hand side. In this case, the default settings are sufficient meaning we can go ahead and press “Generate” (If your application doesn’t require all the tables and/or columns you can disable these before generating).

The Speedment Tool visualizes the data structure and allows customizations of the generated code.

Next, Speedment will analyze the database metadata and generate the entire Java domain model. Once this process is completed you are ready to write your application. If you check out the Main.java-file, you will find a project starter containing something like this:

public class Main {
    
    public static void main(final String... args) {

        Speedment app = new MyApplicationBuilder()
            .withUsername("your-dbms-username")
            .withPassword("your-dbms-password")
            .build();

        app.stop();

    }


}

From here, you are ready to build your application using the examples we have provided in the previous articles. Thereby, we can close the circle by fetching a Manager for the Film table (a handle to the content of the film table) by typing:

FilmManager films = app.getOrThrow(FilmManager.class);

Using the Manager we can now query our connected database as we have shown: 



List<Film> filmsTitleStartsWithA = films.stream()
  .filter(Film.TITLE.startsWith("A"))
  .sorted(Film.LENGTH)
  .collect(Collectors.toList());
 
filmsTitleStartsWithA: [
   FilmImpl { filmId=15, title=ALIEN CENTER, …, rating=NC-17, length = 46,
   FilmImpl { filmId=2, title=ACE GOLDFINGER, …, rating=G, length = 48,
… ]

Exercises 

This week there is no associated GitHub repo for you to play with. Instead, we encourage you to integrate Speedment in a new or an existing database application to try out your newly acquired skills.


Extra Exercise

When you are ready with your project, we encourage you to try out HyperStream, especially if you have a large database and want to increase the reading performance.

HyperStream goes beyond Stream and adds in-JVM-memory capabilities which boost application speed by orders of magnitude. You only need to add a few lines of code in your existing pom.xml and your Main.java file:

    .withBundle(InMemoryBundle.class) // add to the app builder

    ...

    // Load data from database into materialized view
    app.getOrThrow(DataStoreComponent.class) .load();

Read more in the user-guide. The Stream API remains the same but performance is vastly increased.

Conclusion

During the past six weeks, we have demonstrated the usefulness of the Java Stream API and how it can be leveraged for writing type-safe database applications in pure Java. If you wish to learn more about Speedment, check out the user guide which also contains a more thorough guide on Java Streams.

Lastly - thank you for taking interest in our article series, it has been truly great to see that many of you have been following along with the provided exercises. Happy coding!

Authors

Per Minborg
Julia Gustafsson

Resources

Further reading about Speedment Stream JOINs 
Speedment Manual 
Speedment Initializer 
Speedment on GitHub

Tuesday, November 12, 2019

Become a Master of Java Streams - Part 5: Turn Joined Database Tables into a Stream

Is it possible to turn joined database tables into a Java Stream? The answer is yes. Since we got this question so many times, we decided to throw in another hands-on-lab article explaining how to perform more advanced Stream Joins. So here you are, the fifth article out of six, complemented by a GitHub repository containing instructions and exercises to each unit.

Part 1: Creating Streams
Part 2: Intermediate Operations
Part 3: Terminal Operations
Part 4: Database Streams
Part 5: Turn Joined Database Tables into Streams
Part 6: Creating a Database Application Using Streams

Stream JOINs

In the last article, we pointed out the great resemblance between Streams and SQL constructs. Although, the SQL operation JOIN lacks a natural mapping in the general case. Therefore, Speedment leverages its own JoinComponent to join up to 10 tables (using INNER JOIN, RIGHT JOIN, LEFT JOIN or CROSS JOIN) in a type-safe way. Before we introduce the JoinComponent in more depth, we will elaborate on the similarities between individual tables and joins.

We previously used a Speedment Manager as a handle to a database table. This process is visualized below:

A Manager acts as a handle to a database table and can act as a stream source. In this case, every row corresponds to an instance of Film.

Now that we wish to retrieve data from multiple tables, the Manager on its own is not sufficient. An SQL JOIN-query outputs a virtual table that combines data from multiple tables in different ways (e.g. depending on the join-type and WHERE-clauses). In Speedment, that virtual table is represented as a Join<T> object holding tuples of type T.

Join Component

To retrieve a Join-object we need the previously mentioned JoinComponent which uses a builder pattern. The resulting Join-objects are reusable and acts as handles to "virtual join tables", as described by this image:



The JoinComponent creates a Join-object which acts as a handle to a virtual table (the result of the join) and can act as a stream source. In this case, every row corresponds to an instance of Tuple2.
Now that we have introduced the notion of the JoinComponent we can start demonstrating how it is used.


Many-to-one

We start by looking at a Many-to-One relationship where multiple rows from a first table can match the same single row in a second table. For example, a single language may be used in many films. We can combine the two tables Film and Language using the JoinCompontent:
Join<Tuple2<Film, Language>> join = joinComponent
    .from(FilmManager.IDENTIFIER)
    .innerJoinOn(Language.LANGUAGE_ID).equal(Film.LANGUAGE_ID)
    .build(Tuples::of);

Basically, we start with the Film table and perform an INNER JOIN with the Language table on rows that have matching language_id:s.

We can then use the Join-object to stream over the resulting Tuples and print them all out for display. As always with Streams, no specific order of the elements is guaranteed even if the same join-element is reused.
 
join.stream()
    .forEach(System.out::println);

Tuple2Impl {FilmImpl { filmId = 1, title = ACADEMY DINOSAUR, ... }, LanguageImpl { 
languageId = 1, name = English, ... }}

Tuple2Impl {FilmImpl { filmId = 2, title = ACE GOLDFINGER, ... }, LanguageImpl {
languageId = 1, name = English, ... }}

Tuple2Impl {FilmImpl { filmId = 3, title = ADAPTATION HOLES, ... }, LanguageImpl {
languageId = 1, name = English, ... }}
…

Many-to-Many

A Many-to-Many relationship is defined as a relationship between two tables where many multiple rows from a first table can match multiple rows in a second table. Often a third table is used to form these relations. For example, an actor may participate in several films and a film usually have several actors.

The relation between films and actors in Sakila is described by the FilmActor table which references films and actors using foreign keys. Hence, if we would like to relate each Film entry to the actors who starred in that movie we need to join all three tables:
 
Join<Tuple3<FilmActor, Film, Actor>> join = joinComponent
    .from(FilmActorManager.IDENTIFIER)
    .innerJoinOn(Film.FILM_ID).equal(FilmActor.FILM_ID)
    .innerJoinOn(Actor.ACTOR_ID).equal(FilmActor.ACTOR_ID)
    .build(Tuples::of);

We start with the table describing the relation between the Film and Actor and perform and INNER JOIN with both Film and Actor on matching FILM_ID:s and ACTOR_ID:s respectively.


Collect Join Stream to Map

Our Join-object can now be used to create a Map that correlates a Film with a List of the starring Actor:s. Since the elements of our stream are Tuples we need to point to the desired entries. This is done using zero-indexed getters (get0() referencing FilmActor and so on).
 
Map<Film, List<Actor>> actorsInFilms = join.stream()
    .collect(
        groupingBy(Tuple3::get1,           
            mapping(Tuple3::get2, toList())   
        )
    );
Lastly we print the entries to display the name of the films and actors.
 
actorsInFilms.forEach((f, al) ->
    System.out.format("%s : %s%n",
        f.getTitle(),
        al.stream()
            .sorted(Actor.LAST_NAME)
            .map(a -> a.getFirstName() + " " + a.getLastName())
            .collect(joining(", ")
        )
     )
);
WONDERLAND CHRISTMAS : HARRISON BALE, CHRIS BRIDGES, HUMPHREY GARLAND, WOODY JOLIE, CUBA OLIVIER
BUBBLE GROSSE : VIVIEN BASINGER, ROCK DUKAKIS, MENA HOPPER
OPUS ICE : DARYL CRAWFORD, JULIA FAWCETT, HUMPHREY GARLAND, SEAN WILLIAMS
…

Filtering Tables

If we know initially that we are only interested in a subset of the Film entries, it is more efficient to get rid of these instances as we define the Join-object. This is done using the .where()-operator which is the equivalent to a filter() on a stream (and maps to the SQL keyword WHERE). As a filter it takes a Predicate that evaluates to true or false and should be expressed using Speedment Fields for optimization. Here we want to find the language of the films with titles beginning with an “A”:
 
Join<Tuple2<Film, Language>> join = joinComponent
    .from(FilmManager.IDENTIFIER)
        .where(Film.TITLE.startsWith(“A”))
    .innerJoinOn(Language.LANGUAGE_ID).equal(Film.LANGUAGE_ID)
    .build(Tuples::of);

If further filtering is needed, it is possible to stack any number of .where()-operations as they are combined with the SQL keyword AND under the hood.

Specialized Constructors

Sofar we have had to deal with the fairly abstract getters of the tuples (get0, get1 and so on). Although, upon building our Join-object we can provide any constructor to specialized objects. In the examples shown above, we have been interested in the title of the films and the name of the actors. That allows us to define our own object TitleActorName as such:
 
final class TitleActorName {
    
    private final String title;
    private final String actorName;

    TitleActorName(Film film, Actor actor) {
       this.title = film.getTitle();
       this.actorName = actor.getFirstName() + actor.getLastName();
    }
    public String title() {
        return title;
    }
    public String actorName() {
        return actorName;
    }
    @Override
    public String toString() {
        return "TitleLanguageName{" + "title=" + title + ", actorName=" + actorName + '}';
    }
}
We then provide the constructor of our own object to the Join builder and discard the linking FilmActor instance since it’s not used:

 
Join<TitleActorName> join = joinComponent
    .from(FilmActorManager.IDENTIFIER)
    .innerJoinOn(Film.FILM_ID).equal(FilmActor.FILM_ID)
    .innerJoinOn(Actor.ACTOR_ID).equal(FilmActor.ACTOR_ID)
    .build((fa, f, a) -> new TitleActorName(f, a));

This greatly improves the readability of any operations involving the resulting Join-object.

 
Map<String, List<String>> actorsInFilms = join.stream()
    .collect(
        groupingBy(TitleActorName::title,
            mapping(TitleActorName::actorName, toList())
        )
    );
    actorsInFilms.forEach((f, al) ->
        System.out.format("%s : %s%n", f, al)
    );

Simplifying Types

When a large number of tables are joined, the Java type can be tedious to write (e.g. Tuple5<...>). If you use a more recent version of Java, you can simply omit the type for the local variable like this:

var join = joinComponent
    .from(FilmManager.IDENTIFIER)
        .where(Film.TITLE.startsWith(“A”))
    .innerJoinOn(Language.LANGUAGE_ID).equal(Film.LANGUAGE_ID)
    .build(Tuples::of);
In this case, Java will automatically infer the type to Join<Tuple2<Film, Language>>

If you are using an older Java version, you can inline the join-declaration and the stream operator like this:

joinComponent
    .from(FilmManager.IDENTIFIER)
        .where(Film.TITLE.startsWith(“A”))
    .innerJoinOn(Language.LANGUAGE_ID).equal(Film.LANGUAGE_ID)
    .build(Tuples::of)
    .stream()
    .forEach(System.out::println);

Exercises

This week’s exercises will require combined knowledge from all the previous units and therefore acts as a great follow-up on the previous modules. There is still a connection to an instance of the Sakila database in the cloud so no setup of Speedment is needed. As usual, the exercises can be located in this GitHub repo. The content of this article is sufficient to solve the fifth unit which is called MyUnit5Extra. The corresponding Unit5Extra interface contains JavaDocs which describe the intended implementation of the methods in MyUnit5Extra.
 
public interface Unit5Extra {
/**
 * Creates and returns a new Map with Actors as keys and
 * a List of Films in which they appear as values.
 * <p>
 * The result might look like this:
 *
 * ActorImpl { actorId = 126, firstName = FRANCES, lastName = TOMEI, ... }=[FilmImpl { filmId = 21, title = AMERICAN CIRCUS, ...}, ...]
 * …
 *
 * @param joinComponent for data input
 * @return a new Map with Actors as keys and
 *         a List of Films in which they appear as values
 */
Map<Actor, List<Film>> filmographies(JoinComponent joinComponent);

The provided tests (e.g. Unit5ExtraTest) will act as an automatic grading tool, letting you know if your solution was correct or not.


Next Article

By now we hopefully managed to demonstrate how neat the Stream API is for database queries. The next article will move beyond the realm of movie rentals and allow you to write standalone database applications in pure Java for any data source. Happy coding!

Authors

Per Minborg
Julia Gustafsson

Resources

GitHub Opensource Project Speedment
Speedment Stream ORM Initializer
GitHub Repository "hol-streams"
Article Part 1: Creating Streams
Article Part 2: Intermediate Operations
Article Part 3: Terminal Operations

Wednesday, October 30, 2019

Become a Master of Java Streams - Part 4: Database Streams

SQL has always been a declarative language whereas Java for a long time has been imperative. Java streams have changed the game. Code your way through this hands-on-lab article and learn how Java streams can be used to perform declarative queries to an RDBMS database, without writing a single line of SQL code. You will discover, there is a remarkable similarity between the verbs of Java streams and SQL commands.

This article is the fourth out of five, complemented by a GitHub repository containing instructions and exercises to each unit.
Part 1: Creating Streams
Part 2: Intermediate Operations
Part 3: Terminal Operations
Part 4: Database Streams
Part 5: Creating a Database Application Using Streams

Database Streams 

When you familiarized yourself with the operations of Streams, you may have noticed a resemblance to the SQL constructs. Some of them have a more or less a direct mapping to Stream operations, such as LIMIT and COUNT. This resemblance is utilized by the open-source project Speedment to provide type-safe access to any relational database using pure Java.



This table shows how Speedment maps between SQL and Java Streams.

We are contributors to the Speedment open-source project and we will describe how Speedment allows us to use a database as the stream source and feed the pipeline with rows from any of the database tables.



As depicted in the visualization above, Speedment will establish a connection to the database and can then pass data to the application. There is no need to write any code for the database entries since Speedment analyses the underlying database and automatically generates all the required entity classes for the domain model. It saves a lot of time when you don’t have to write and maintain entity classes by hand for each table you want to use.

Sakila Database 

For the sake of this article, as well as the exercises, we use the MySQL example database Sakila as our data source. The Sakila database models an old-fashioned movie rentals business and therefore contains tables such as Film and Actor. An instance of the database is deployed in the cloud and is open for public access.

Speedment Manager 

In Speedment, the handle to a database table is a called a Manager. The managers are part of the automatically generated code.












A Manager acts as a handle to a database table and can act as a stream source. In this case, every row corresponds to an instance of Film. 


A Manager in Speedment is instantiated by calling:

FilmManager films = speedment.getOrThrow(FilmManager.class);

Note: speedment is an instance that can be obtained from an ApplicationBuilder (more on this topic in the next article).

If the FilmManager::stream is called, the result is a Stream to which we are free to apply any intermediate or terminal operations. For starters, we collect all rows in a list.
 
List<Film> allFilms = films.stream().collect(toList());
FilmImpl { filmId = 1, title = ACADEMY DINOSAUR, …
FilmImpl { filmId = 2, title = ACE GOLDFINGER, …
FilmImpl { filmId = 3, title = ADAPTATION HOLES, …
…

Filtering and Counting

Let’s look at a simple example that outputs the number of films having the rating “PG-13”. Just like a regular Stream, we can filter out the films with the correct rating, and then count these entries.
 
long pg13FilmCount = films.stream()
   .filter(Film.RATING.equal("PG-13"))
   .count();

pg13FilmCount: 195

One important property that follows with Speedment’s custom implementation of Streams is that the streams are able to optimize their own pipeline by introspection. It may look like the Stream will iterate over all rows of a table, but this is not the case. Instead, Speedment is able to translate the pipeline to an optimized SQL query that is passed on to the database. This means only relevant database entries are pulled into the Stream. Thus, in the example above, the stream will be automatically rendered to SQL similar to “SELECT … FROM film WHERE rating = ‘PG-13’ ”

This introspection requires that any use of anonymous lambdas (which do not contain any metadata that relates to the targeted column) are replaced with Predicates from Speedment Fields. In this case Film.RATING.equal(“PG-13”) returns a Predicate that will be tested on each Film and return true if and only if that Film has a Rating that is PG-13.

Although, this does not prevent us from expressing the predicate as:
 
    .filter(f -> f.getRating().equals(“PG-13”))

but this would force Speedment to fetch all the rows in the table and then apply the predicate, hence it is not recommended.

Finding the Longest Film

Here is an example that finds the longest film in the database using the max-operator with the Field Film.LENGTH:
 
Optional<Film> longestFilm = films.stream()
   .max(Film.LENGTH);

longestFilm: 
Optional[FilmImpl {filmId = 141, title = CHICAGO NORTH, length = 185, ...}]

Finding Three Short Films

Locating three short films (we defined short as <= 50 minutes) can be done by filtering away any films that are 50 minutes or shorter and picking the three first results. The predicate in the example looks at the value of the column “length” and determines if it is less than or equal to 50.

List<Film> threeShortFilms = films.stream()
 .filter(Film.LENGTH.lessOrEqual(50))
 .limit(3)
 .collect(toList());
threeShortFilms: [
    FilmImpl { filmId = 2, length = 48,..}, 
    FilmImpl { filmId = 3, length = 50, … }, 
    FilmImpl { filmId = 15, length = 46, ...}]

Pagination with Sorting

If we were to display all the films on a website or in an application, we would probably prefer to paginate the items, rather than loading (possibly) thousands of entries at once. This can be accomplished by combining the operation skip() and limit(). In the example below, we collect the content of the second page, assuming every “page” holds 25 entries. Recall that Streams do not guarantee a certain order of the elements, which means that we need to define an order with the sorted-operator for this to work as intended.
 
List<Film> filmsSortedByLengthPage2 = films.stream()
 .sorted(Film.LENGTH)
 .skip(25 * 1)
 .limit(25)
 .collect(toList());
filmsSortedByLengthPage2: 
[FilmImpl { filmId = 430, length = 49, …}, …]

Note: Finding the content of the n:th page is done by skipping (25 * (n-1)).
Note2: This stream will be automatically rendered to something like “SELECT ... FROM film ORDER BY length ASC LIMIT ? OFFSET ?, values:[25, 25]”

Films Starting with “A” Sorted by Length

We can easily locate any films starting with the capital letter “A” and sort them according to their length (with the shortest film first) like this:
 
List<Film> filmsTitleStartsWithA = films.stream()
 .filter(Film.TITLE.startsWith("A"))
 .sorted(Film.LENGTH)
 .collect(Collectors.toList());
filmsTitleStartsWithA: [
  FilmImpl { filmId=15, title=ALIEN CENTER, …, rating=NC-17, length = 46,
  FilmImpl { filmId=2, title=ACE GOLDFINGER, …, rating=G, length = 48,
… ]

Computing Frequency Tables of Film Length

We can also utilize the groupingBy-operator to sort the films in buckets depending on their lengths and count the total number of films in each bucket. This will create a so-called frequency table of film length.
 
Map<Short, Long> frequencyTableOfLength = films.stream()
 .collect(Collectors.groupingBy(
  Film.LENGTH.asShort(),
  counting()
 ));
frequencyTableOfLength: {46=5, 47=7, 48=11, 49=5, … }

Exercises

For this week’s exercises, you do not need to worry about connecting a database of your own. Instead, we have already provided a connection to an instance of the Sakila database in the cloud. As usual, the exercises can be located in this GitHub repo. The content of this article is sufficient to solve the fourth unit which is called MyUnit4Database. The corresponding Unit4Database Interface contains JavaDocs which describe the intended implementation of the methods in MyUnit4Database.

 
public interface Unit4Database {

   /**
    * Returns the total number of films in the database.
    *
    * @param films manager of film entities
    * @return the total number of films in the database
    */
   long countAllFilms(FilmManager films);
The provided tests (e.g. Unit4MyDatabaseTests) will act as an automatic grading tool, letting you know if your solution was correct or not.

Next Article

So far, we have only scraped the surface of database streams. The next article will allow you to write standalone database applications in pure Java. Happy coding!

Authors

Per Minborg
Julia Gustafsson

Resources

GitHub Opensource Project Speedment
Speedment Stream ORM Initializer
GitHub Repository "hol-streams"
Article Part 1: Creating Streams
Article Part 2: Intermediate Operations
Article Part 3: Terminal Operations

Friday, October 4, 2019

Become a Master of Java Streams - Part 1: Creating Streams

Declarative code (e.g. functional composition with Streams) provides superior code metrics in many cases. Code your way through this hands-on-lab article series and mature into a better Java programmer by becoming a Master of Java Streams.

The whole idea with Streams is to represent a pipeline through which data will flow and the pipeline’s functions operate on the data. This way, functional-style operations on Streams of elements can be expressed. This article is the first out of five where you will learn firsthand how to become a Master of Streams. We start with basic stream examples and progress with more complex tasks until you know how to connect standard Java Streams to databases in the Cloud.

Once you have completed all five articles, you will be able to drastically reduce your codebase and know how to write pure Java code for the entire applications in a blink.

Here is a summary of the upcoming articles:


Since we are firm believers in the concept of ”Learning by doing”, the series is complemented by a GitHub repository that contains Stream exercises split into 5 Units - each corresponding to the topic of an article. Instructions on how to use the source code are provided in the README-file.

What are Java Streams?

The Java Stream interface was first introduced in Java 8 and, together with lambdas, acts as a milestone in the development of Java since it contributes greatly to facilitating a declarative (functional) programming style. If you want to learn more about the advantages of declarative coding we refer you to this article.

A Java Stream can be visualized as a pipeline through which data will flow (see the image below). The pipeline’s functions will operate on the data by e.g. filtering, mapping and sorting the items. Lastly, a terminal operation can be performed to collect the items in a preferred data structure such as a List, an Array or a Map. An important thing to notice is that a Stream can only be consumed once.


A Stream Pipeline contains three main parts; the stream source, the intermediate operation(s) (zero to many) and a terminal operation.

Let’s have a look at an example to get a glimpse of what we will be teaching throughout this series. We encourage you to look at the code below and try to figure out what the print-statement will result in before reading the next paragraph.

List<String> list = Stream.of("Monkey", "Lion", "Giraffe","Lemur")
    .filter(s -&gt; s.startsWith("L"))
    .map(String::toUpperCase)
    .sorted()
    .collect(toList());
System.out.println(list);

Since the Stream API is descriptive and most often intuitive to use, you will probably have a pretty good understanding of the meaning of these operations regardless if you have encountered them before or not. We start off with a Stream of a List containing four Strings, each representing an African animal. The operations then filter out the elements that start with the letter “L”, converts the remaining elements to uppercase letters, sorts them in natural order (which in this case means alphabetical order) and lastly collects them into a List. Hence, resulting in the output [“LEMUR”, “LION”].

It is important to understand that Streams are “lazy” in the sense that elements are “requested” by the terminal operation (in this case the .collect() statement). If the terminal operation only needs one element (like, for example, the terminal operation .findFirst()), then at most one element is ever going to reach the terminal operation and the reminding elements (if any) will never be produced by the source. This also means that just creating a Stream is often a cheap operation whereas consuming it might be expensive depending on the stream pipeline and the number of potential elements in the stream.

In this case, the Stream Source was a List although many other types can act as a data source. We will spend the rest of this article describing some of the most useful source alternatives.


Stream Sources

Streams are mainly suited for handling collections of objects and can operate on elements of any type T. Although, there exist three special Stream implementations; IntStream, LongStream, and DoubleStream which are restricted to handle the corresponding primitive types.

An empty Stream of any of these types can be generated by calling Stream.empty() in the following manner:

Stream<T>     Stream.empty()
IntStream     IntStream.empty()
LongStream    LongStream.empty()
DoubleStream  DoubleStream.empty()

Empty Streams are indeed handy in some cases, but the majority of the time we are interested in filling our Stream with elements. This can be accomplished in a large number of ways. We will start by looking at the special case of an IntStream since it provides a variety of useful methods.


Useful IntStreams

A basic case is generating a Stream over a small number of items. This can be accomplished by listing the integers using IntStream.of(). The code below yields a simple stream of elements 1, 2 and 3.
IntStream oneTwoThree = IntStream.of(1, 2, 3);
Listing all elements manually can be tedious if the number of items grows large. In the case where we are interested in values in a certain range, the command .rangeClosed() is more effective. The operation is inclusive, meaning that the following code will produce a stream of all elements from 1 to 9.
IntStream positiveSingleDigits = IntStream.rangeClosed(1, 9);

An even more powerful command is .iterate() which enables greater flexibility in terms of what numbers to include. Below, we show an example of how it can be used to produce a Stream of all numbers that are powers of two.
IntStream powersOfTwo = IntStream.iterate(1, i -> i * 2);
There are also several perhaps more unexpected ways of producing a Stream. The method chars() can be used to Stream over the characters in a String, in this case, the elements “A”, “B” and “C”.
IntStream chars = "ABC".chars();
There is also a simple way to generate a Stream of random integers.
IntStream randomInts = new Random().ints();

Stream an Array

Streaming existing data collections is another option. We can stream the elements of an existing Array or choose to list items manually using Stream.of() as previously shown and repeated below.
String[] array = {"Monkey", "Lion", "Giraffe", "Lemur"};
Stream<String> stream2 = Stream.of(array);
Stream<String> stream = Stream.of("Monkey", "Lion", "Giraffe", "Lemur");

Stream from a Collection

It is also very simple to stream any Collection. The examples below demonstrate how a List or Set can be streamed with the simple command .stream().
List<String> list = Arrays.asList("Monkey", "Lion", "Giraffe", "Lemur");
Stream<String> streamFromList = list.stream();
Set<String> set = new HashSet<>(list);
Stream<String> streamFromSet = set.stream();

Stream from a Text File

Sometimes it can also be useful to stream the contents of a text-file. The following command will provide a Stream<String> that holds every line from the referenced file as a separate element.

Stream<String> lines = Files.lines(Paths.get("file.txt"));


Exercise

Now that we have familiarized you with some of the ways of creating a Stream, we encourage you to clone this GitHub repo and start practicing. The content of the article will be enough to solve the first Unit which is called Create. The Unit1Create interface contains JavaDocs which describes the intended implementation of the methods in Unit1MyCreate.
public interface Unit1Create {
 /**
  * Creates a new Stream of String objects that contains
  * the elements "A", "B" and "C" in order.
  *
  * @return a new Stream of String objects that contains
  *   the elements "A", "B" and "C" in order
  */
  Stream<String> newStreamOfAToC();
The provided tests (e.g. Unit1MyCreateTest) will act as an automatic grading tool, letting you know if you solution was correct or not.


If you have not done so yet, go ahead and solve the work items in the Unit1MyCreate class. “Gotta catch ‘em all”.

In the next article, we will continue to describe several intermediate operations that can be applied to these Streams and that will convert them into other Streams. See you soon!

Authors 

Per Minborg
Julia Gustafsson

Monday, September 2, 2019

Java Performance: For-looping vs. Streaming

Is counting upwards or downwards in a for-loop the most efficient way of iterating? Sometimes the answer is neither. Read this post and understand the impact of different iteration varieties.

Iteration Performance

There are many views on how to iterate with high performance. The traditional way of iterating in Java has been a for-loop starting at zero and then counting up to some pre-defined number:

private static final int ITERATIONS = 10_000;

@Benchmark
public int forUp() {
    int sum = 0;
    for (int i = 0; i < ITERATIONS; i++) {
        sum += i;
    }
    return sum;
}
Sometimes, we come across a for-loop that starts with a predetermined non-negative value and then it counts down instead. This is fairly common within the JDK itself, for example in the class String. Here is an example of solving the previous problem by counting down instead of up.

@Benchmark
public int forDown() {
    int sum = 0;
    for (int i = ITERATIONS; i-- > 0;) {
        sum += i;
    }
    return sum;
}

I think the rationale here is that checking how values relate to zero is potentially more efficient than testing how values relate to any other arbitrary value. In fact, all CPUs that I know about have machine code instructions that can check how a given value relates to zero. Another idea is that the count down idiom given above appears to only inspect the loop variable one time (it simultaneously checks the value and then decreases it) as opposed to the regular example at the top. I suspect that this has little or no influence on today’s efficient JIT compiler who will be able to optimize the first iteration just as good as the second. It might have an impact when the code runs in interpretation mode but this is not examined in this article.

Another way of doing the same thing using an IntStream looks like this:

@Benchmark
public int stream() {
    return IntStream.range(0, ITERATIONS)
        .sum();
}

If more performance is needed for large iterations, it is relatively easy to make the stream parallel by just adding a .parallel() operator to the stream. This is not examined in this article.

Performance under Graal VM

Running these tests under GraalVM (rc-11, with the new C2 compiler that ships with GraallVM) on my laptop (MacBook Pro mid 2015, 2.2 GHz Intel Core i7) gives the following:

Benchmark              Mode  Cnt       Score       Error  Units
ForBenchmark.forDown  thrpt    5  311419.166 ±  4201.724  ops/s
ForBenchmark.forUp    thrpt    5  309598.916 ± 12998.579  ops/s
ForBenchmark.stream   thrpt    5  312360.089 ±  8291.792  ops/s

It might come as a surprise for some that the stream solution is the fastest one, albeit by a margin that is well within error margins.

In a previous article, I presented some code metric advantages with streams and Declarative programming compared to traditional Imperative code. I have not tested performance for cold code sections (i.e. before the JIT kicks in).

Clever Math

From math, we recall that the sum of consecutive numbers starting at zero is N*(N+1)/2 where N is the highest number in the series. Running this benchmark:

@Benchmark
public int math() {
    return ITERATIONS * (ITERATIONS + 1) / 2;
}

gives us a performance increase of over 1,000 times over the previous implementations:

Benchmark           Mode  Cnt          Score          Error  Units
ForBenchmark.math  thrpt    5  395561077.984 ± 11138012.141  ops/s

The more iterations, the more gain. Cleverness sometimes trumps brute force.

Ultra-fast Data Streams

With Speedment HyperStream, it is possible to get similar performance with data from databases. Read more here on HyperStream.

Conclusions

On some commonly used hardware/JVMs, it does not matter if we iterate upwards or downwards in our for-loops. More modern JVMs are able to optimize stream iterations so they have equivalent or even better performance than for-loops.

Stream code is generally more readable compared to for-loops in my opinion and so, I believe streams are likely to be the de facto iteration contrivance in some future.

Database content can be streamed with high performance using Speedment HyperStream.

Thursday, August 15, 2019

Java: An Optional Implementation of Optional

The class java.util.Optional is implemented as a single immutable concrete class that internally handles two cases; one with an element and one without. Wouldn't it have been a better choice to let Optional be an interface and have two different implementations implement that interface instead? After all, that is what we generally are taught to do in an object-oriented language.

In this article, we will learn about some of the potential arguments for the current Optional implementation. We will also learn why Streams are implemented in a different way, enabling Streams to be obtained from files or even database tables.

The Real Optional Implementation

The real java.util.Optional::get is implemented as shown hereunder:
public T get() {
        if (value == null) {
            throw new NoSuchElementException("No value present");
        }
        return value;
    }
As can be seen, there are two code paths; one where the value is null (no element and an exception is thrown) and one when the value is something else (the value is returned).

An Optional Optional Implementation

Let’s pretend that we would go back in a time machine and were tasked to implement Optional once again. I think it is likely that many of us would come up with an initial solution much like the one below (I have named the hypothetical interface Option so we can distinguish it from the “real” one) with two distinct implementations (here EmptyOption and PresentOption):

public interface Option<T> {

    T get();

    boolean isPresent();

    public <U> Option<U> map(Function<? super T, ? extends U> mapper);

    static <T> Option<T> empty() { return (Option<T>) EmptyOption.EMPTY; }

    static <T> Option<T> of(T value) { return new PresentOption<>(value); }

    static <T> Option<T> ofNullable(T value) {
        return value == null ? empty() : of(value);
    }

}

final class EmptyOption<T> implements Option<T> {

    static final EmptyOption<?> EMPTY = new EmptyOption<>();

    private EmptyOption() {}

    @Override public T get() { throw new NoSuchElementException(); }

    @Override public boolean isPresent() { return false; }

    @Override
    public <U> Option<U> map(Function<? super T, ? extends U> mapper) {
        requireNonNull(mapper);
        return (Option<U>) EMPTY;
    }
}

final class PresentOption<T> implements Option<T> {

    private final T value;

    PresentOption(T value) { this.value = requireNonNull(value); }

    @Override public T get() { return value; }

    @Override
    public boolean isPresent() { return true; }

    @Override
    public <U> Option<U> map(Function<? super T, ? extends U> mapper) {
        requireNonNull(mapper);
        return Option.ofNullable(mapper.apply(value));
    }
}

Only a few methods are shown for brevity but the principle remains the same: distinct implementations for the case where an element is present and when it is not. This gives a much clearer code and also opens up the possibility for anyone to implement optionals.

Analysis

I am confident that this type of solution was evaluated by the JDK team at the time Optional was conceived and I think it was a well-informed decision not to opt for this solution. Optional was primarily intended to “wrap” return values to protect from NPEs and other drawbacks of returning raw null values. I also think the design goal was that there should be little to negligible performance impact using Optional.

In the following, I speculate in some of the arguments to elect the present Optional implementation over the one coined above.

Profile Pollution

The JIT compiler compiles the Java byte code on-demand to improve performance over interpreting the byte code.

In order to do this efficiently, the JIT compiler is able to gather statistics for every known method. Each method can have a MethodData object that contains metrics on how the method is used and such an object is created once the JVM thinks the method is “warm” enough (i.e. has been called sufficiently in some sense).

The process of creating and maintaining MethodData is called “profiling”.

“Profile Pollution” occurs when the method is used substantially different between calls, including, but not limited to, providing alternating non-null/null elements and calling different polymorph methods (e.g. a parameter is generic of type T and the called method invokes T::equals). A cornerstone feature of Java is its ability to invoke methods dynamically. Thus, when Option::get is invoked, either EmptyOption::get or PresentOption::get is ultimately invoked depending on which implementation is present at the time of invocation.

Once the method has been invoked some 10,000 times, the JIT compiler is using the MethodData to create an efficient compiled code snippet that executes in the best way given the statistics gathered so far.

So, if elements are present all the time (using PresentOption) and the code is compiled with that in mind, but then there is an EmptyOption suddenly appearing, the code must “back out” and take a much slower code path.

With Optional in just one final class, there can never be any other implementation of the Optional methods and thus, no profile pollution due to different implementations. The JIT can make a deterministic and reasonably fast compiled code determination.

But wait, wouldn’t it be possible for the JVM to check all classes at startup and determine that there were, in fact, just two implementing classes of the Option and then it could figure the whole thing out? Well, no. We are free to add classes at any time so there would be no way of safely enumerating all possible implementations of a particular interface. At least not until we have real sealed classes in Java.

API pollution

If people were free to write custom implementations of Optional, then these implementations would most likely be suffering from design flaws/deviations compared to the built-in Optional. Also, people would likely let their own types implement the interface Optional adding to the burden of the JIT compiler/profiler and will thus tempt people to use composite types (e.g. Foo implements Bar, Optional<Bazz>) which was not intended.

Also, Optional is now an integral part of Java and as such, it can be made to efficiently evolve with the JDK itself including, perhaps, inline classes and other new upcoming Java features.

Optional vs. Streams

As opposed to Optional, java.util.stream.Stream and the specialized versions, like IntStream, are indeed interfaces. Why is not Stream a concrete single final class just like Optional?

Well, Streams have a completely different set of requirements. Streams can be obtained from a Collection or an array but there are far more powerful ways of obtaining a Stream. It is possible to acquire a Stream from a file, a socket, a random generator and even from tables in a database. These features would not be possible to implement if Stream was sealed.

Speedment Stream is an example of a library that allows standard Java Streams to be obtained from virtually any database. Read more about Speedment Stream here.

Conclusion

Optional is sealed and there are good reasons why. The internal implementation of Optional is less clear but that is a price worth paying with the benefits of better performance and clearer user code.

Streams are non-sealed interfaces that can be implemented by anyone and can be used to obtain elements from various sources including files and database tables. Speedment Stream ORM can be used to get Streams from database tables.

Download Speedment Stream here.

Monday, May 13, 2019

Java: How to Become More Productive with Hazelcast in Less Than 5 Minutes

Java: How to Become More Productive with Hazelcast in Less Than 5 Minutes

What if you want to use a Hazelcast In-Memory Data Grid (IMDG) to speed up your database applications, but you have hundreds of tables to handle? Manually coding all Java POJOs and serialization support would entail weeks of work and when done, maintaining that domain model by hand would soon turn into a nightmare. Read this article and learn how to save time and do it in 5 minutes.

Now there is a graceful way to manage these sorts of requirements. The Hazelcast Auto DB Integration Tool allows connection to an existing database which can generate all these boilerplate classes automatically. We get true POJOs, serialization support, configuration, MapStore/MapLoad, ingest and more without having to write a single line of manual code. As a bonus, we get Java Stream support for Hazelcast distributed maps.

Using the Tool

Let us try an example. As in many of my articles, I will be using the Sakila open-source example database. It can be downloaded as a file or as a Docker instance. Sakila contains 16 tables and a total of 90 columns in those tables. It also includes seven views with additional columns.

To start, we use the Hazelcast Auto DB Integration Initializer and a trial license key.


Fill in the values as shown above and press “Download” and your project is saved to your computer. Then, follow the instructions on the next page explaining how to unzip, start the tool and get the trial license.

Next, we connect to the database:



The tool now analyses the schema metadata and then visualizes the database schema in another window:



Just press the “Generate” button and the complete Hazelcast domain model will be generated automatically within 2 or 3 seconds.



Now, we are almost ready to write our Hazelcast IMDG application. We need to create a Hazelcast IMDG to store the actual data in first.

Architecture

This is how the architecture looks like where the Application talks to the Hazelcast IMDG which, in turn, gets its data from the underlying Database:





The code generated by the tool need only be present in the Application and not in the Hazelcast IMDG.

Creating a Hazelcast IMDG

Creating a Hazelcast IMDG is easy. Add the following dependency to your pom.xml file:

<dependency>
     <groupId>com.hazelcast</groupId>
     <artifactId>hazelcast</artifactId>
     <version>3.11</version>
</dependency>

Then, copy the following class to your project:

public class Server {

    public static void main(String... args) throws InterruptedException {
        final HazelcastInstance instance = Hazelcast.newHazelcastInstance();
        while (true) {
            Thread.sleep(1000);
        }
    }

}

Run this main method three times to create three Hazelcast nodes in a cluster. More recent versions of IDEA requires “Allow parallel run” to be enabled in the Run/Debug Configurations. If you only run it once, that is ok too. The example below will still work even though we would just have one node in our cluster.

Running the main method tree times will produce something like this:

Members {size:3, ver:3} [
 Member [172.16.9.72]:5701 - d80bfa53-61d3-4581-afd5-8df36aec5bc0
 Member [172.16.9.72]:5702 - ee312d87-abe6-4ba8-9525-c4c83d6d99b7
 Member [172.16.9.72]:5703 - 71105c36-1de8-48d8-80eb-7941cc6948b4 this
]
Nice! Our three-node-cluster is up and running!

Data Ingest

Before we can run any business logic, we need to ingest data from our database into the newly created Hazelcast IMDG. Luckily, the tool does this for us too. Locate the generated class named SakilaIngest and run it with the database password as the first command line parameter or modify the code so it knows about the password. This is what the generated class looks like.

public final class SakilaIngest {
    
    public static void main(final String... argv) {
        if (argv.length == 0) { 
            System.out.println("Usage: " + SakilaIngest.class.getSimpleName() + " database_password");
         } else {
            try (Speedment app = new SakilaApplicationBuilder()
                .withPassword(argv[0]) // Get the password from the first command line parameter
                .withBundle(HazelcastBundle.class)
                .build()) {
            
                IngestUtil.ingest(app).join();
            }
        }
    }
}
When run, the following output is shown (shortened for brevity):

...
Completed          599 row(s) ingest of data for Hazelcast Map sakila.sakila.customer_list
Completed            2 row(s) ingest of data for Hazelcast Map sakila.sakila.sales_by_store
Completed       16,049 row(s) ingest of data for Hazelcast Map sakila.sakila.payment
Completed       16,044 row(s) ingest of data for Hazelcast Map sakila.sakila.rental
Completed          200 row(s) ingest of data for Hazelcast Map sakila.sakila.actor_info

We now have all data from the database in the Hazelcast IMDG. Nice!

Hello World

Now that our grid is live and we have ingested data, we have access to populated Hazelcast maps. Here is a program that prints all films of length greater than one hour to the console using the Map interface:
public static void main(final String... argv) {
        try (Speedment app = new SakilaApplicationBuilder()
            .withPassword("your-db-password-goes-here")
            .withBundle(HazelcastBundle.class)
            .build()) {

            HazelcastInstance hazelcast = app.getOrThrow(HazelcastInstanceComponent.class).get();

            IMap<Integer, Film> filmMap = hazelcast.getMap("sakila.sakila.film");
            filmMap.forEach((k, v) -> {
                if (v.getLength().orElse(0) > 60) {
                    System.out.println(v);
                }
            });

        }
    }

The film length is an optional variable (i.e., nullable in the database) so it gets automatically mapped to an OptionalLong. It is possible to set this behavior to “legacy POJO” that returns null if that is desirable in the project at hand.

There is also an additional feature with the tool: We get Java Stream support! So, we could write the same functionality like this:

public static void main(final String... argv) {
    try (Speedment app = new SakilaApplicationBuilder()
        .withPassword("your-db-password-goes-here")
        .withBundle(HazelcastBundle.class)
        .build()) {

        FilmManager films = app.getOrThrow(FilmManager.class);
            
        films.stream()
            .filter(Film.LENGTH.greaterThan(60))
            .forEach(System.out::println);

    }

Under the Hood

The tool generates POJOs that implements Hazelcast’s “Portable” serialization support. This means that data in the grid is accessible from applications written in many languages like Java, Go, C#, JavaScript, etc.

The tool generates the following Hazelcast classes:

POJO

One for each table/view that implements the Portable interface.

Serialization Factory

One for each schema. This is needed to efficiently create Portable POJOs when de-serializing data from the IMDG in the client.

MapStore/MapLoad

One for each table/view. These classes can be used by the IMDG to load data directly from a database.

Class Definition

One for each table/view. These classes are used for configuration.

Index utility method

One per project. This can be used to improve the indexing of the IMDG based on the database indexing.

Config support

One per project. Creates automatic configuration of serialization factories, class definitions, and some performance setting.

Ingest support

One per project. Template for ingesting data from the database into the Hazelcast IMDG.

The tool also contains other features such as support for Hazelcast Cloud and Java Stream support.

A particularly appealing property is that the domain model (e.g., POJOs and serializers) does not need to be on the classpath of the servers. They only need to be on the classpath on the client side. This dramatically simplifies the setup and management of the grid. For example, if you need more nodes, add a new generic grid node and it will join the cluster and start participating directly.

Hazelcast Cloud

Connections to Hazelcast Cloud instances can easily be configured using the application builder as shown in this example:

Speedment hazelcastApp = new SakilaApplicationBuilder()
            .withPassword(“<db-password>")
            .withBundle(HazelcastBundle.class)
            .withComponent(HazelcastCloudConfig.class, 
                () -> HazelcastCloudConfig.create(
                            "<name of cluster>",
                            "<cluster password>",
                            "<discovery token>"
                )
            )
            .build();

Savings

I estimate that the tool saved me several hours (if not days) of boilerplate coding just for the smaller example Sakila database. In an enterprise-grade project with hundreds of tables, the tool would save a massive amount of time, both in terms of development and maintenance.

Now that you have learned how to create code for your first exemplary project and have set up all the necessary tools, I am convinced that you could generate code for any Hazelcast database project in under 5 minutes.

Resources

Sakila: https://dev.mysql.com/doc/index-other.html or https://hub.docker.com/r/restsql/mysql-sakila
Initializer: https://www.speedment.com/hazelcast-initializer/
Manual: https://speedment.github.io/speedment-doc/hazelcast.html

Monday, April 8, 2019

Java Stream: Is a Count Always a Count?

Java Stream: Is a Count Always a Count?

It might appear obvious that counting the elements in a Stream takes longer time the more elements there are in the Stream. But actually, Stream::count can sometimes be done in a single operation, no matter how many elements you have. Read this article and learn how.

Count Complexity

The Stream::count terminal operation counts the number of elements in a Stream. The complexity of the operation is often O(N), meaning that the number of sub-operations is proportional to the number of elements in the Stream.

In contrast, the List::size method has a complexity of O(1) which means that regardless of the number of elements in the List, the size() method will return in constant time. This can be observed by running the following JMH benchmarks:
@State(Scope.Benchmark)
public class CountBenchmark {

    private List<Integer> list;

    @Param({"1", "1000", "1000000"})
    private int size;

    @Setup
    public void setup() {
        list = IntStream.range(0, size)
            .boxed()
            .collect(toList());
    }

    @Benchmark
    public long listSize() {
        return list.size();
    }

    @Benchmark
    public long listStreamCount() {
        return list.stream().count();
    }

    public static void main(String[] args) throws RunnerException {
        Options opt = new OptionsBuilder()
            .include(CountBenchmark.class.getSimpleName())
            .mode(Mode.Throughput)
            .threads(Threads.MAX)
            .forks(1)
            .warmupIterations(5)
            .measurementIterations(5)
            .build();

        new Runner(opt).run();

    }

}

This produced the following output on my laptop (MacBook Pro mid 2015, 2.2 GHz Intel Core i7):

Benchmark                        (size)   Mode  Cnt          Score           Error  Units
CountBenchmark.listSize               1  thrpt    5  966658591.905 ± 175787129.100  ops/s
CountBenchmark.listSize            1000  thrpt    5  862173760.015 ± 293958267.033  ops/s
CountBenchmark.listSize         1000000  thrpt    5  879607621.737 ± 107212069.065  ops/s
CountBenchmark.listStreamCount        1  thrpt    5   39570790.720 ±   3590270.059  ops/s
CountBenchmark.listStreamCount     1000  thrpt    5   30383397.354 ±  10194137.917  ops/s
CountBenchmark.listStreamCount  1000000  thrpt    5        398.959 ±       170.737  ops/s


As can be seen, the throughput of List::size is largely independent of the number of elements in the List whereas the throughput of Stream::count drops of rapidly as the numbers of elements grow. But, is this really always the case for all Stream implementation per se?

Source Aware Streams

Some stream implementations are actually aware of their sources and can take appropriate shortcuts and merge stream operations into the stream source itself. This can improve performance massively, especially for large streams. The Speedment ORM tool allows databases to be viewed as Stream objects and these streams can optimize away many stream operations like the Stream::count operation as demonstrated in the benchmark below. I have used the open-source Sakila exemplary database as data input. The Sakila database is all about rental films, artists etc.

@State(Scope.Benchmark)
public class SpeedmentCountBenchmark {

    private Speedment app;
    private RentalManager rentals;
    private FilmManager films;

    @Setup
    public void setup() {
        app =  new SakilaApplicationBuilder()
            .withBundle(DataStoreBundle.class)
            .withLogging(ApplicationBuilder.LogType.STREAM)
            .withPassword(ExampleUtil.DEFAULT_PASSWORD)
            .build();

        app.get(DataStoreComponent.class).ifPresent(DataStoreComponent::load);

        rentals = app.getOrThrow(RentalManager.class);
        films = app.getOrThrow(FilmManager.class);

    }

    @TearDown
    public void tearDown() {
        app.close();
    }


    @Benchmark
    public long rentalsCount() {
        return rentals.stream().count();
    }


    @Benchmark
    public long filmsCount() {
        return films.stream().count();
    }


    public static void main(String[] args) throws RunnerException {
        Options opt = new OptionsBuilder()
            .include(SpeedmentCountBenchmark.class.getSimpleName())
            .mode(Mode.Throughput)
            .threads(Threads.MAX)
            .forks(1)
            .warmupIterations(5)
            .measurementIterations(5)
            .build();

        new Runner(opt).run();

    }

}
When run, the following output will be produced:

Benchmark                              Mode  Cnt         Score          Error  Units
SpeedmentCountBenchmark.filmsCount    thrpt    5  71037544.648 ± 75915974.254  ops/s
SpeedmentCountBenchmark.rentalsCount  thrpt    5  69750012.675 ± 37961414.355  ops/s


The “rental” table contains over 10,000 rows whereas the “film” table only contains 1,000 rows. Nevertheless, their Stream::count operations complete in almost the same time. Even if a table would contain a trillion rows, it would still count the elements in the same elapsed time. Thus, the Stream::count implementation has a complexity that is O(1) and not O(N).

Note: The benchmark above were run with Speedment's “DataStore” in-JVM-memory acceleration. If run with no acceleration directly against a database, the response time would depend on the underlying database’s ability to execute a “SELECT count(*) FROM film” query.

Summary

It is possible to create Stream implementation that counts their elements in a single operation rather than counting each and every element in the stream. This can improve performance significantly, especially for streams with many elements.

Resources

Speedment Stream ORM Initializer: https://www.speedment.com/initializer/
Sakila: https://dev.mysql.com/doc/index-other.html or https://hub.docker.com/r/restsql/mysql-sakila

Tuesday, December 18, 2018

Java: Aggregate Data Off-Heap

Java: Aggregate Data Off-Heap

Explore how to create off-heap aggregations with a minimum of garbage collect impact and
maximum memory utilization.

Creating large aggregations using Java Map, List and Object normally creates a lot of heap memory overhead. This also means that the garbage collector will have to clean up these objects once the aggregation goes out of scope.

Read this short article and discover how we can use Speedment Stream ORM to create off-heap aggregations that can utilize memory more efficiently and with little or no GC impact.

Person

Let’s say we have a large number of Person objects that take the following shape:

public class Person {
    private final int age;
    private final short height;
    private final short weight;        
    private final String gender;
    private final double salary;
    …
    // Getters and setters hidden for brievity
}

For the sake of argument, we also have access to a method called persons() that will create a new Stream with all these Person objects.

Salary per Age

We want to create the average salary for each age bucket. To represent the results of aggregations we will be using a data class called AgeSalary which associates a certain age with an average salary.

public class AgeSalary {
     private int age;
     private double avgSalary;
     … 
    // Getters and setters hidden for brievity
}

Age grouping for salaries normally entails less than 100 buckets being used and so this example is just to show the principle. The more buckets, the more sense it makes to aggregate off-heap.

Solution

Using Speedment Stream ORM, we can derive an off-heap aggregation solution with these three steps:

Create an Aggregator

var aggregator = Aggregator.builderOfType(Person.class, AgeSalary::new)
    .on(Person::age).key(AgeSalary::setAge)
    .on(Person::salary).average(AgeSalary::setAvgSalary)
    .build();

The aggregator can be reused over and over again.

Compute an Aggregation

var aggregation = persons().collect(aggregator.createCollector());

Using the aggregator, we create a standard Java stream Collector that has its internal state completely off-heap.

Use the Aggregation Result

aggregation.streamAndClose()
    .forEach(System.out::println);

Since the Aggregation holds data that is stored off-heap, it may benefit from explicit closing rather than just being cleaned up potentially much later. Closing the Aggregation can be done by calling the close() method, possibly by taking advantage of the AutoCloseable trait, or as in the example above by using streamAndClose() which returns a stream that will close the Aggregation after stream termination.

Everything in a One-Liner

The code above can be condensed to what is effective a one-liner:

persons().collect(Aggregator.builderOfType(Person.class, AgeSalary::new)
    .on(Person::age).key(AgeSalary::setAge)
    .on(Person::salary).average(AgeSalary::setAvgSalary)
    .build()
    .createCollector()
).streamAndClose()
    .forEach(System.out::println);

There is also support for parallel aggregations. Just add the stream operation Stream::parallel and aggregation is done using the ForkJoin pool.

Resources

Download Speedment here

Read more about off-heap aggregations here