Quantcast
Channel: Java – blog@CodeFX
Viewing all 68 articles
Browse latest View live

Repackaging Exceptions In Streams

$
0
0

Java 8 is a couple of years old but there are still use cases, not even edge cases, that the community did not yet develop a good arsenal of solutions for. How to handle checked exceptions in stream pipelines is one such problem. The functional interfaces various Stream operations accept do not allow implementations to throw checked exceptions but many methods we might want to call do. Obviously, there’s a tension here, which many developers have come across.

I want to explore this topic in a short series of posts:

Repackaging Exceptions In Streams
Repackage exceptions in order to throw them without the compiler complaining about it.
Handling Exceptions In Streams
Catch and handle exceptions on the spot, possibly by deferring error handling.
Throwing Exceptions From Streams
How to handle deferred errors by throwing an exception after all.

My main goal is to propose various solutions and, ideally, to establish a common terminology that makes discussions easier. I will also comment on my suggestions, adding my own assessment of how useful I find them – this is secondary, though, and I hope that it does not distract from the main goal: getting the ideas out there.

This first post will look into repackaging exceptions so that the compiler stops complaining.

Setting the Scene

The underlying scenario is something every frequent user of streams has encountered in one form or other: A method you would like to use in one of stream’s intermediate operations throws a checked exceptions.

In this post, I will assume that you are trying to parse a stream of strings to a stream of users:

Stream<User> parse(Stream<String> strings) {
    return strings.map(User::parse);
}

(If you’re not down with having streams as parameters or return values, assume the entire stream pipeline would be within the method’s scope. The following techniques apply either way but some of the assessments would be different if you handled the entire stream on the spot.)

Unfortunately,

User::parse
can throw a
ParseException
:

public class User {

    public static User parse(String userString) throws ParseException {
        // ...
    }

}

This leads to the compiler complaining about “Unhandled exception: java.text.ParseException” for the method reference

User::parse
. What to do now?

Before we look into solutions for this problem I want to point something out: I do not regard the Stream API’s incompatibility with checked exceptions as something that could’ve been overcome with a different design. At some point I may write a longer post explaining that, but the short version is this: If the functional interface methods could throw checked exceptions, there would be no pleasant way to combine that with streams’ laziness as it is the terminal operation that will eventually throw that exception.

But we can make good use of a function that can throw exceptions, so let’s introduce that interface while we’re at it:

@FunctionalInterface
interface CheckedFunction<T, R, EX extends Exception> {

    R apply(T element) throws EX;

}

This allows us to assign

User::parse
to a
CheckedFunction<String, User, ParseException>
. Note that the type of the exception is generic, which will come in handy later.

Repackaging Exceptions In Streams

So do you really have to handle the exceptions? Could you not just, I don’t know, make the problem go away? The surprising answer is “Yes, you can.” Whether you should remains to be seen…

Wrap In Unchecked Exception

Given a function that throws a checked exception, it is pretty easy to transform it into one that throws an unchecked one instead:

Stream<User> parse(Stream<String> strings) {
    return strings
            .map(uncheckException(User::parse))
}

<T, R> Function<T, R> uncheckException(
        CheckedFunction<T, R, Exception> function) {
    return element -> {
        try {
            return function.apply(element);
        } catch (Exception ex) {
            // thanks to Christian Schneider for pointing out
            // that unchecked exceptions need not be wrapped again
            if (ex instanceof RuntimeException)
                throw (RuntimeException) ex;
            else
                throw new RuntimeException(ex);
        }
    };
}

This is actually not too bad. And if you prefer unchecked exceptions anyway, then this is all the more enticing. If, on the other hand, you value the distinction between checked exceptions (for things you expect can go wrong, like bad input for example) and unchecked exceptions (for implementation errors), then this will sent shivers down your spine.

In any case the final consumer of the stream has to be aware that the exception could be thrown, which at this point needs to be communicated with tests or documentation, both easier to ignore than the compiler. It feels a little like hiding a bomb in the stream.

Finally, note that this aborts the stream as soon as the first error occurs – something that might or might not be ok. Deciding whether it is ok can be tough if the method returns a stream instead of consuming it because different callers might have different requirements.

Sneaky-Throw Exception

Another way to fix this whole thing, is to “sneaky-throw” the exception. This technique uses generics to confuse the compiler and

@SuppressWarnings
to silence its remaining complaints.

Stream<User> parse(Stream<String> strings) {
    return strings
            .map(hideException(User::parse));
}

<T, R> Function<T, R> hideException(
        CheckedFunction<T, R, Exception> function) {
    return element -> {
        try {
            return function.apply(element);
        } catch (Exception ex) {
            return sneakyThrow(ex);
        }
    };
}

@SuppressWarnings("unchecked")
<E extends Throwable, T> T sneakyThrow(Throwable t) throws E {
    throw (E) t;
}

Err, what? As promised, the method

sneakyThrow
uses generics to trick the compiler into throwing a checked exception without declaring it. Then
hideException
uses that to catch any exception the
CheckedFunction
might throw and rethrows it sneakily. (In case your using Lombok, have a look at its
@SneakyThrows
annotation
.)

I consider this a very risky move. For one, it still hides a bomb in the stream. It goes much further, though, and makes that bomb extra hard to defuse properly. Did you ever try to catch a checked exception that is not declared with a

throws
clause?

try {
    userStrings.stream()
            .map(hideException(User::parse));
            .forEach(System.out::println);
// compile error because ParseException
// is not declared as being thrown
} catch (ParseException ex) {
    // handle exception
}

Won’t work because the compiler operates under the assumption that none of the methods actually throw a

ParseException
. Instead you’d have to catch
Exception
, filter out
ParseException
and rethrow everything else.

Wow, that sucks!

Unfortunately this technique shows up in a StackOverflow answer that ranks extremely well on Google when looking for Java stream exception handling. In all fairness, the answer contains a disclaimer but I am afraid it might get ignored too often:

Needless to say, this should be handled with care and everybody on the project must be aware that a checked exception may appear where it is not declared.

But as we have seen there is no good way to declare / catch such an exception, so I would have worded that a little stronger:

It’s a nice experiment but never actually do it! If you really want to throw, wrap in a runtime exception.

Lift Exception

The problem with sneaky-throw was that it surprises consumers of the stream and makes it hard to handle that exception even once they overcame that surprise. For the latter, at least, there is a way out. Consider this function:

<T, R, EX extends Exception> Function<T, R> liftException(
        CheckedFunction<T, R, EX> function) throws EX {
    return hideException(function);
}

It does exactly the same as

hideException
but it declares that it throws
EX
. Why would that be helpful? Because this way you can use it to make the compiler understand that a checked exception might get thrown:

Stream<User> parse(Stream<String> strings) {
    return strings
            // does not compile because `liftException`
            // throws ParseException but it is unhandled
            .map(liftException(User::parse));
}

The problem is, and the body of

liftException
makes that abundantly clear, that it does of course not throw an exception. So in an example like this, where we see only part of the pipeline, it arguably makes the situation even more confusing. Now, callers of
parse
might put it into a try-catch block, expecting to have handled the exception well (if the don’t think too hard about it), and then still get surprised when the terminal operation throws that very exception (remember it is hidden with
sneakyThrow
).

If you are someone who never returns streams, though,

liftException
can be pretty useful. With it, some call in your stream pipeline declares to throw a checked exception so you can put it all into a try-catch block:

try {
    userStrings.stream()
            .map(liftException(User::parse));
            .forEach(System.out::println);
} catch (ParseException ex) {
    // handle exception
}

Alternatively, the method containing the pipeline could declare that it throws the exception:

List<User> parse(List<String> userStrings) throws ParseException {
    return userStrings.stream()
            .map(liftException(User::parse));
            .collect(toList());
}

But as I said before, I think this only works well if you never return streams. Because if you do, even only occasionally, there is a risk that your or a colleague takes the pipeline apart during a refactoring, arming the bomb that is an undeclared checked exception, hidden in a stream.

There is another drawback that Sebastian Millies pointed out, namely that the interfaces and methods used so far only allow a single exception. As soon as a method declares more than one checked exception, things get problematic. Either you let Java derive a common supertype (likely to be

Exception
) or you declare additional
CheckedFunction
interfaces and
liftException
methods for more than one exception. Both not exactly great options.

Reflection

Given a method that throws a checked exception I have shown you two and a half different ways to use them in a stream if the exception needs to be thrown immediately:

Note that all of these approaches mean that the stream pipeline will stop processing then and there, yielding no results unless those achieved by side effects. I find often that is not what I want to do, though (because I do like returning streams). The next article tackles this by investigating how to handle exceptions on the spot, without aborting the pipeline.

The post Repackaging Exceptions In Streams appeared first on blog@CodeFX.


Optional Dependencies in the Java Platform Module System

$
0
0

The Java Platform Module System (JPMS) has a strong opinion on dependencies: By default, they need to be required (to be accessible) and then they need to be present both at compile and at run time. This does not work with optional dependencies, though, where code is written against artifacts that are not necessarily present at run time. Fortunately, the JPMS has a

requires static
clause that can be used in these exact situations.

I will show you a couple of examples in which the default behavior’s strictness leads to problems and then introduce the module system’s solution to optional dependencies:

requires static
. Coding against them is not trivial, though, so we will have a close look at that as well.

Overview

Some examples build on the

optional-dependencies
branch of a small demo application, called the Service Monitor.

The Conundrum Of Unrequired Dependencies

To nail down where exactly the strictness of regular

requires
clauses leads to problems, I want to start with two examples. While similar in some aspects there are differences that become important later when we discuss how we code against potentially missing dependencies.

The Utility Library

Let’s start with an imaginary library we’re maintaining, uber.lib, that integrates with a handful of other libraries. Its API offers functionality that builds on them and thus exposes their types. We’ll play this through with the example of com.google.guava, which in our hypothetical scenario was already turned into a Java module that uber.lib wants to code against.

As maintainers of uber.lib we assume that nobody who is not already using Guava is ever going to call the Guava portion of our library. This makes sense in certain cases: Why would you call a method in uber.lib that creates a nice report for a

com.google.common.graph.Graph
instance if you don’t have such a graph?

For uber.lib that means that it can function perfectly without com.google.guava: If Guava makes it into the module graph, clients might call into that portion of the uber.lib API. If it doesn’t, they won’t and the library will be fine as well. We can say that uber.lib never needs the dependency for its own sake.

With regular dependencies optional relationships can not be implemented.

With regular

requires
clauses, such an optional relationship can not be implemented, though. According to the rules for readability and accessibility, uber.lib has to require com.google.guava to compile against its types but this forces all clients to always have Guava on the module path when launching their application.

If uber.lib integrates with a handful of libraries, it would make clients depend on all of them even though they might never use more than one.
That’s not a nice move from us.

The Fancy Statistics Library

The second example comes from the demo application, which contains a module monitor.statistics. Let’s assume there was some advanced statistics library containing a module stats.fancy that monitor.statistics wants to use but which could not be present on the module path for each deployment of the application. (The reason for that is irrelevant but let’s go with a license that prevents the fancy code from being used “for evil” but, evil masterminds that we are, we occasionally want to do just that.)

We would like to write code in monitor.statistics that uses types from the fancy module but for that to work we need to depend on it with a

requires
clause. If we do that, though, the module system would not let the application launch if stats.fancy is not present.

The conundrum of optional dependencies

Deadlock. Again.

Optional Dependencies With ‘requires static’

When a module needs to be compiled against types from another module but does not want to depend on it at run time, it can use a

requires static
clause. If
foo requires static bar
, the module system behaves different at compile and run time:
  • At compile time, bar must be present or there will be an error. During compilation bar is readable by foo.
  • At run time, bar might be absent and that will cause neither error nor warning. If it is present, it is readable by foo.

We can immediately put this into action and create an optional dependency from monitor.statistics to stats.fancy:

module monitor.statistics {
    requires monitor.observer;
    requires static stats.fancy;
    exports monitor.statistics;
}

If stats.fancy is missing during compilation, we get an error when the module declaration is compiled:

monitor.statistics/src/main/java/module-info.java:3:
    error: module not found: stats.fancy
        requires static stats.fancy;
                             ^
1 error

At launch time, though, the module system does not care whether stats.fancy is present or not.

Similarly, the module descriptor for uber.lib declares all dependencies as optional:

module uber.lib {
    requires static com.google.guava;
    requires static org.apache.commons.lang;
    requires static org.apache.commons.io;
    requires static io.javaslang;
    requires static com.aol.cyclops;
}

Now that we know how to declare optional dependencies, two questions remain to be answered:

  • Under what circumstances will it be present?
  • How can we code against an an optional dependency?

We will answer both questions next.

Resolution Of Optional Dependencies

Module resolution is the process that, given an initial module and a universe of observable modules, builds a module graph by resolving

requires
clauses. When a module is being resolved, all modules it requires must be found in the universe of observable modules. If they are, they are added to the module graph; otherwise an error occurs. It is important to note that modules that did not make it into the module graph during resolution are not available later during compilation or execution, either.

At compile time, module resolution handles optional dependencies just like regular dependencies. At run time, though,

requires static
clauses are mostly ignored. When the module system encounters one it does not try to fulfill it, meaning it does not even check whether the named module is present in the universe of observable modules.

A module that is only an optional dependency will not be available at run time.

As a consequence even if a module is present on the module path (or in the JDK for that matter), it will not be added to the module graph just because of an optional dependency. It will only make it into the graph if it is also a regular dependency of some other module that is being resolved or because it was added explicitly with the command line flag

--add-modules
.

Maybe you stumbled across the phrase that optional dependencies “are mostly ignored”. Why mostly? Well, one thing the module system does is if an optional dependency makes it into a graph, a readability edge is added. This ensures that if the optional module is present, its types can be accessed straight away.

Coding Against Optional Dependencies

Optional dependencies require a little more thought when writing code against them because this is what happens when monitor.statistics uses types in stats.fancy but the module isn’t present at run time:

Exception in thread "main" java.lang.NoClassDefFoundError:
    stats/fancy/FancyStats
        at monitor.statistics/monitor.statistics.Statistician
            .<init>(Statistician.java:15)
        at monitor/monitor.Main.createMonitor(Main.java:42)
        at monitor/monitor.Main.main(Main.java:22)
Caused by: java.lang.ClassNotFoundException: stats.fancy.FancyStats
        ... many more

Oops. We usually don’t want our code to do that.

Generally speaking, when the code that is currently being executed references a type, the Java Virtual Machine checks whether it is already loaded. If not, it tells the class loader to do that and if that fails, the result is a

NoClassDefFoundError
, which usually crashes the application or at least fails out of the chunk of logic that was being executed.

With optional dependencies we opt out of the checks that make the module system safe.

This is something JAR hell was famous for and that the module system wants to overcome by checking declared dependencies when launching an application. But with

requires static
we opt out of that check, which means we can end up with a
NoClassDefFoundError
after all. What can we do against that?

Established Dependency

Before looking into solutions, though, we need to see whether we really have a problem. In the case of uber.lib we expect to only use types from an optional dependency if the code calling into the library already uses them, meaning class loading already succeeded.

In other words, when uber.lib gets called all required dependencies must be present or the call would not have been possible. So we don’t have a problem after all and don’t need to do anything.

An optional depndency might always be present when the calling code gets executed

Internal Dependency

The general case is different, though. It might very well be the module with the optional dependency that first tries to load classes from it, so the risk of a

NoClassDefFoundError
is very real.

An optional dependency might only be used internally

One solution for this is to make sure that all possible calls into the module with the optional dependency have to go through a checkpoint before accessing the dependency. That checkpoint has to evaluate whether the dependency is present and send all code that arrives at it down a different execution path if it isn’t.

It might be necessary to check presence of an optional dependency

The module system offers a way to check whether a module is present. I explained in my newsletter how to get there and why I use the new stack-walking API, so here you’ll just have to trust me when I say that this is the way to go:

public class ModuleUtils {

    public static boolean isModulePresent(String moduleName) {
        return StackWalker
                .getInstance(RETAIN_CLASS_REFERENCE)
                .walk(frames -> frames
                        .map(StackFrame::getDeclaringClass)
                        .filter(declaringClass ->
                                declaringClass != ModuleUtils.class)
                        .findFirst()
                        .orElse((Class) ModuleUtils.class));
                .getModule();
                .getLayer()
                .findModule(moduleName)
                .isPresent();
        // chain all the methods!
    }

}

(In a real application it might make sense to cache the value as to not always repeat the same check.)

Calling this method with an argument like

"stats.fancy"
will return whether that module is present. If called with the name of a regular dependency (simple
requires
clause), the result will always be
true
because otherwise the module system would not have let the application launch. If called with the name of an optional dependency (
requires static
clause), the result will either be
true
or
false
.

If an optional dependency is present, the module system established readability and so it is safe to go down an execution path that uses types from the module. If it is absent, choosing such a path would lead to a

NoClassDefFoundError
, so a different one has to be found.

Summary

Sometimes you want to write code against a dependency that might not always be present at run time. To make the dependency’s types available at compile time but not enforce its presence at launch time, the module system offers the

requires static
clause. Note, though, that a module does not get picked up during resolution if it is only referenced this way and that special care needs to be taken to make sure code does not crash if the optional dependency is absent at run time.

To learn more about the module system check out the JPMS tag or get my book The Java 9 Module System (with Manning). If you’re interested in the historical perspective, check the Project Jigsaw tag.

The post Optional Dependencies in the Java Platform Module System appeared first on blog@CodeFX.

Java 9 Resources – Talks, Articles, Repos, Blogs, Books And Courses

$
0
0

I wrote this article for SitePoint's Java channel, where you can find a lot of interesting articles about our favorite programming language. Check it out!

You can tell that Java 9 draws near because the number of posts and talks about it skyrocketed in the recent months. I want to recommend existing talks and articles where you can learn about Java 9 but also further resources, where new, high-quality content will pop up.

Talks

If you’re the kind of person who likes to watch talks, there are quite a few I can recommend. For a great high-level overview and conceptual intro to the module system, watch Java 9: Make Way for Modules! (Mark Reinhold; 40 min). Going deeper into the module system, the JDK team has an entire series of talks:

As a follow-up, there was an Ask the Architect session at JFokus where Mark Reinhold answers all kinds of questions, among them some about Java 9 (transitive dependencies, version conflicts, state of JavaFX, ahead-of-time compilation; 23 min).

With Java 9 coming closer, people started presenting on non-modularity features that Java 9 has to offer. Simon Ritter races through 55 new features in JDK 9 (50 min) and this is me talking a little bit about the module system before going into the new language features, a few new APIs (collection factories, reactive streams aka Flow API, stack-walking, multi-release JARs) and performance (50 min). If you want to dive deeper, there is a talk by Aleksey Shipilëv on compact strings and indified string concatenation, which I highly recommend (60 min). Monica Beckwith explains about G1 but be warned, you better have your GC expertise down before giving this a try (55 min).

There are also a number of great talks that are much more practical. To learn about how Maven deals with with Java 9, watch Robert Scholte talk about Unicode encoding, version strings, cross compilation, multi-release JARS, and then of course Jigsaw with its impact on how Maven works but also what it has to offer (50 min). Don’t miss live-coding queen Trisha Gee working on a Java 9 project with IntelliJ, where she demonstrates various features of both the JVM and the IDE (30 min). If you’re interested to see what a migration to Java 9 modules might look like, watch Rabea Gransberger live-refactor a small demo project (15 min). Of course there is no way to talk about live-coding without mentioning Venkat Subramaniam, who shows off modules and JShell in a mammoth 150 minute live session.

For shorter bits there are a couple of interviews the Voxxed folks recorded:

Repositories

Of course people started writing code and there are a few interesting repositories that demonstrate Java 9 features:

Java 9 Resources

Articles

There are countless articles about Java 9, so there is simply no way to do everyone justice. Here’s my try of listing the more important ones:

For a longer list of posts, check out Baeldung’s Java 9 site.

I want to end whit list with a teaser: One particular interesting part about Java 9 modularity is whether build tools will generate module declarations. Next Monday Robert Scholte, chairman of the Apache Maven project and principal developer of Maven’s Java 9 compatibility and features, will tell us whether Maven can do that for you. Spoiler: It doesn’t look good.

Blogs

There are a few blogs and sites that regularly publish about Java 9. Most of them have the decency to tag those posts, so you don’t have to go searching. Company blogs/sites:

Books and Courses

If you want to go really deep and prepare yourself for actually using the new stuff, you might want to go for a book or online course. These are the ones I know of:

And because this is my blog I will take the freedom to make sure you don’t overlook that the last book on that list is by me, which of course makes it the best one. 😜 Early access is open, so you can get it now!

The post Java 9 Resources – Talks, Articles, Repos, Blogs, Books And Courses appeared first on blog@CodeFX.

Java 9 Migration Guide: The Seven Most Common Challenges

$
0
0

I’m sure you’ve heard that updating to Java 9 is no walk in the park, maybe even that it’s an incompatible update and that a migration makes no sense for large code bases. After doing exactly that, migrating an old and fairly large code base, I can tell you that it’s not that bad. It’s more work than bumping to Java 8, true, but it’s time well spent. More than anything else, the migration uncovered some small and a few not so small problems that needed fixing regardless of the migration itself and we took the opportunity to do just that.

I collected a few surprising details over at java9.wtf but condensed the seven largest issues into this Java 9 migration guide. It’s as much a post as it is a resource to come back to, so put it on speed dial and search it when you have a concrete problem. Also note that while you need to know a bit about the module system (here’s a hands-on guide), this is not about modularizing your application – it is only about getting it to compile and run on Java 9.

Overview

This is a list of the seven most likely problems to trip you up during a migration to Java 9:

Each section explains the problem, the most common symptoms that help you identify it and a set of possible fixes (usually command line options for

java
or
javac
). A future post will tie individual fixes into a larger migration strategy and make some recommendations based on my experiences.

Illegal Access To Internal APIs

One of the module system’s biggest selling points is strong encapsulation. It makes sure non-public classes as well as classes from non-exported packages are inaccessible from outside the module. First and foremost, this of course applies to the platform modules shipped with the JDK, where only

java.*
and
javax.*
packages are fully supported. Most
com.sun.*
and
sun.*
packages, on the other hand, are internal and hence inaccessible by default.

While the Java 9 compiler behaves exactly as you would expect and prevents illegal access, the same is not true for the run time. To offer a modicum of backwards compatibility it eases migration and improves the chances of applications built on Java 8 to run on Java 9 by granting access to internal classes. If reflection is used for the access, a warning is emitted.

Symptoms

During compilation against Java 9 you see compile errors similar to the following:

error: package com.sun.java.swing.plaf.nimbus is not visible
import com.sun.java.swing.plaf.nimbus.NimbusLookAndFeel;
                              ^
    (package com.sun.java.swing.plaf.nimbus is declared
    in module java.desktop, which does not export it)
1 error

Warnings emitted for reflection look as follows:

Static access to [Nimbus Look and Feel]
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by j9ms.internal.Nimbus
    (file:...) to constructor NimbusLookAndFeel()
WARNING: Please consider reporting this
    to the maintainers of j9ms.internal.Nimbus
WARNING: Use --illegal-access=warn to enable warnings
    of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Reflective access to [Nimbus Look and Feel]

Fixes

The most obvious and sustainable fix for dependencies on internal APIs is to get rid of them. Replace them with maintained APIs and you paid back some high-risk technical debt.

If that can’t be done for whatever reason, the next best thing is to acknowledge the dependencies and inform the module system that you need to access it. To that end you can use two command line options:

  • The option
    --add-exports $module/$package=$readingmodule
    can be used to export
    $package
    of $module to $readingmodule. Code in $readingmodule can hence access all public types in
    $package
    but other modules can not. When setting $readingmodule to
    ALL-UNNAMED
    , all modules in the module graph and code from the class path can access that package. During a migration to Java 9, you will always use that placeholder. The option is available for the
    java
    and
    javac
    commands.
  • This covers access to public members of public types but reflection can do more than that: With the generous use of
    setAccessible(true)
    it allows interaction with non-public classes, fields, constructors, and methods (sometimes called deep reflection), which even in exported packages are still encapsulated. The
    java
    option
    --add-opens
    uses the same syntax as
    --add-exports
    and opens the package to deep reflection, meaning all of its types and their members are accessible regardless of their visibility modifiers.

You obviously need

--add-exports
to appease the compiler but gathering
--add-exports
and
--add-opens
for the run time has advantages as well:
  1. the run time’s permissive behavior will change in future Java releases, so you have to do that work at some point anyway
  2. --add-opens
    makes the warnings for illegal reflective access go away
  3. as I will show in a minute, you can make sure no new dependencies crop up by making the run time actually enforce strong encapsulation

Going Further

Compiling against Java 9 helps hunting down dependencies on internal APIs in the project’s code base. But the libraries and frameworks your project uses are just as likely to make trouble.

JDeps is the perfect tool to find compile dependencies on JDK-internal APIs in your project and your dependencies. If you’re not familiar with it, I’ve written a tutorial that gets you started. Here’s how to use it for the task at hand:

jdeps --jdk-internals -R --class-path 'libs/*' $project

Here,

libs
is a folder containing all of your dependencies and
$project
your project’s JAR. Analyzing the output is beyond this article’s scope but it’s not that hard – you’ll manage.

Finding reflective access is a little tougher. The run time’s default behavior is to warn you once for the first illegal access to a package, which is insufficient. Fortunately, there’s the

--illegal-access=$value
option, where
$value
can be:
  • permit
    : Access to all JDK-internal APIs is permitted to code on the class path. For reflective access, a single warning is issued for the first access to each package. (Default in Java 9.)
  • warn
    : Behaves like
    permit
    but a warning is issued for each reflective access.
  • debug
    : Behaves like
    warn
    but a stack trace is included in each warning.
  • deny
    : The option for those who believe in strong encapsulation:
    All illegal access is forbidden by default.

Particularly

deny
is very helpful to hunt down reflective access. It is also a great default value to set once you’ve collected all required
--add-exports
and
--add-opens
options. This way, no new dependencies can crop up without you noticing it.

Dependencies On Java EE Modules

There’s a lot of code in Java SE that’s actually Java EE related. It ended up in these six modules:

  • java.activation with
    javax.activation
    package
  • java.corba with
    javax.activity
    ,
    javax.rmi
    ,
    javax.rmi.CORBA
    , and
    org.omg.*
    packages
  • java.transaction with
    javax.transaction
    package
  • java.xml.bind with all
    javax.xml.bind.*
    packages
  • java.xml.ws with
    javax.jws
    ,
    javax.jws.soap
    ,
    javax.xml.soap
    , and all
    javax.xml.ws.*
    packages
  • java.xml.ws.annotation with
    javax.annotation
    package

For various compatibility reasons (one of them being split packages, which we will look at next), code on the class path does not see these modules by default, which leads to compile or run time errors.

Symptoms

Here’s a compile error for a class using

JAXBException
from the java.xml.bind module:

error: package javax.xml.bind is not visible
import javax.xml.bind.JAXBException;
                ^
    (package javax.xml.bind is declared in module java.xml.bind,
        which is not in the module graph)
1 error

If you get it past the compiler but forget to massage the run time, you’ll get a

NoClassDefFoundError
:

Exception in thread "main" java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException
    at monitor.Main.main(Main.java:27)
Caused by: java.lang.ClassNotFoundException: javax.xml.bind.JAXBException
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:185)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:496)
    ... 1 more

Fixes

Once you modularized your code, you can declare a regular dependency in the module’s declaration. Until then,

--add-modules $module
comes to your rescue, which makes sure
$module
is available and can be added to both
java
and
javac
. If you add java.se.ee, you’ll have access to all Java EE modules.

Split Packages

This one is a little tricky… To enforce consistency a module is not allowed to read the same package from two different modules. The actual implementation is stricter, though, and no two modules are allowed to even contain the same package (exported or not). The module system operates under that assumption and whenever a class needs to be loaded, it looks up which module contains that package and goes looking for the class in there (which should boost class loading performance).

To safeguard the assumption the module system checks that no two named modules split a package and barfs if it finds any that do. During migration you’re not quite in that situation, though. Your code comes from the class path, which puts it into the so-called unnamed module. To maximize compatibility it is not scrutinized and no module-related checks are applied to it.

Now, in the case of split packages, this means a split between a named module (e.g. in the JDK) and the unnamed module is not discovered. Which may sound very fortunate, is the opposite if you mix in the class loading behavior: If a package is split between a module and the class path, for classes from that package class loading will always and only look into the module. This means classes in the class path portion of the package are effectively invisible.

Symptoms

The symptom is that a class from the class path can not be loaded even though it’s definitely there, leading to compile errors like this:

error: cannot find symbol
    symbol:   class Nonnull
    location: package javax.annotation

Or, at run time, to

NoClassDefFoundError
s like above.

One example where this can occur is with the various JSR-305 implementations. A project using, for example, the annotations

javax.annotation.Generated
(from java.xml.ws.annotation) and
java.annotation.Nonnull
(from com.google.code.findbugs:jsr305) will have trouble compiling. It is either missing the Java EE annotations or, when the module is added like described above, will encounter a split package and not see the JSR 305 module.

Fixes

The migration path will be different, depending on the artifact that splits the JDK package. In some cases it might be more than just some classes that go into a random JDK package but a replacement for an entire JDK module, for example because it overrides an endorsed standard. In that case, you are looking for the

--upgrade-module-path $dir
option – modules found in
$dir
are used to replace upgradeable modules in the run time.

If you indeed just have a couple of classes that split a package, the long-term solution is to remove the split. In case that is not possible in the short-term, you can patch the named module with the content from the class path. The option

--patch-module $module=$artifact
will merge all classes from
$artifact
into
$module
, putting all portions of the split package into the same module, thus removing the split.

There are a few things to look out for, though. First of all, the patched module must actually make it into the module graph, for which it might be necessary to use

--add-modules
. Then, it must have access to all the dependencies that it needs to run successfully. Since named modules can not access code from the class path, this might make it necessary to start creating some automatic modules, which goes beyond the scope of this post.

Going Further

Finding split package by try and error is pretty unnerving. Fortunately JDeps reports them, so if you analyze your project and its dependencies, the first lines of output will report split packages. You can use the same command as above:

jdeps --jdk-internals -R --class-path '$libs/*' project.jar

Casting To URL Class Loader

The class loading strategy that I just described is implemented in a new type and in Java 9 the application class loader is of that type. That means it is not a

URLClassLoader
, anymore, so the occasional
(URLClassLoader) getClass().getClassLoader()
sequence will no longer execute. This is another typical example where Java 9 is backwards compatible in the strict sense (because that it’s a
URLCassLoader
was never specified) but which can nonetheless cause migration challenges.

Symptoms

This one is very obvious. You’ll get a

ClassCastException
complaining that the new
AppClassLoader
is no
URLClassLoader
:

Exception in thread "main" java.lang.ClassCastException:
    java.base/jdk.internal.loader.ClassLoaders$AppClassLoader
    cannot be cast to java.base/java.net.URLClassLoader
        at monitor.Main.logClassPathContent(Main.java:46)
        at monitor.Main.main(Main.java:28)

Fixes

The class loader was probably cast to access methods specific to

URLClassLoader
. If so, your chances to do a migration with only small changes are slim. The only supported (and hence accessible) super types of the new
AppClassLoader
are
SecureClassLoader
and
ClassLoader
and only few methods were added here in 9. Still, have a look, they might do what you’re looking for.

Rummaging Around In Run-Time Images

With the JDK being modularized the layout of the run time image fundamentally changed. Files like

rt.jar
,
tools.jar
, and
dt.jar
are gone; the JDK classes are now bundled into
jmod
files (one per module), a purposely unspecified file format that allows future optimizations without regards to backwards compatibility. Furthermore the distinction between JRE and JDK is gone.

All of this has been unspecified but that doesn’t mean that there’s no code out there depending on these details. Particularly tools like IDEs (although these have mostly been updated already) will have compatibility problems with these changes and will stop working in unpredictable ways unless they’re updated.

As a consequence of these changes, the URL you get for system resources, e.g. from

ClasLoader::getSystemResource
, changed. It used to be of the following form:
jar:file:$javahome/lib/rt.jar!$path
, where
$path
is something like
java/lang/String.class
. It now looks like
jrt:/$module/$path
. Of course all APIs that create or consume such URLs were updated but non-JDK code handcrafting these URLs will have to be updated for Java 9.

Furthermore, the

Class::getResource*
and
ClassLoader::getResource*
methods no longer read JDK-internal resources. Instead use
Module::getResourceAsStream
to access module-internal resources or create a JRT file system as follows:

FileSystem fs = FileSystems.getFileSystem(URI.create("jrt:/"));
fs.getPath("java.base", "java/lang/String.class"));

Boot Class Path

I’m in murky waters here because I never used the

-Xbootclasspath
option, which is mostly removed. Apparently its features are replaced by various new command line options (paraphrasing from JEP 220 here):
  • the
    javac
    option
    --system
    can be used to specify an alternate source of system modules
  • the
    javac
    option
    --release
    can be used to specify an alternate platform version
  • the
    java
    option
    --patch-module
    option, mentioned above, can be used to inject content into modules in the initial module graph

New Version Strings

After more than 20 years, Java has finally and officially accepted that it’s no longer on version

1.x
. Hooray! So from Java 9 on, the system property
java.version
and its siblings no longer start with
1.x
but with
x
, i.e.
9
in Java 9.

Symptoms

There are no clear-cut symptoms – pretty much everything could go wrong if some utility function determines the wrong version. It’s not too hard to find, though. A full text search for the following strings should lead to all version-string-specific code:

java.version
,
java.runtime.version
,
java.vm.version
,
java.specification.version
,
java.vm.specification.version
.

Fixes

If you are willing to raise your project’s requirements to Java 9, you can eschew the whole system property prodding and parsing and instead use the new

Runtime.Version
type, which makes all of this much easier. If you want to stay compatible to pre Java 9, you could still use the new API by creating a multi-release JAR. If that’s also out of the question, it looks like you actually have to write some code (uch!) and branch based on the major version.

Summary

Now you know how to use internal APIs (

--add-export
and
--add-opens
), how to make sure Java EE modules are present (
--add-modules
), and how to deal with split packages (
--patch-module
). These are the most likely problems you’ll encounter during a migration. Less common and also less easy to fix without access to the problematic code are casts to
URLClassLoader
, problems due to the new runtime image layout and resource URLs, the removed
-Xbootclasspath
, and new version strings.

Knowing how to fix these will give you very good chances to overcome all your migration challenges and make your application compile and run on Java 9. If not, take a look at JEP 261’s Risks and Assumptions sections, which lists a few other potential pitfalls.

If you’re a little overwhelmed by all this, wait for my next posts, which gives some advice on how to string these individual fixes into a comprehensive migration strategy, for example by including build tools and continuous integration. Or get my book, where I explain all of this and more.

The post Java 9 Migration Guide: The Seven Most Common Challenges appeared first on blog@CodeFX.

Planning Your Java 9 Update

$
0
0

You’ve read the Java 9 migration guide and now you’re ready to start your Java 9 update? Great! Here are some tips on how to get a sense of what’s awaiting you.

Looking For Trouble

There are two obvious steps to take to gather the first data points for how challenging your update will be:

  • Run your regular build entirely on Java 9, ideally in a way that lets you gather all errors instead of stopping at the first.
  • If you’re developing an application, build it as you do normally (meaning, not yet on Java 9) and then run it on Java 9. Consider using
    --illegal-access=debug
    or
    deny
    to get more information on illegal access.

Carefully analyze the output, take note of new warnings and errors and try to link them to what you know about migration challenges. Also look out for warnings or errors due to removed command line options.

It is a good idea to apply some quick fixes like adding exports or Java EE modules. This allows you to see the tougher problems that may be hiding behind benign ones. In this phase, no fix is too dirty or too hacky – anything that gets the build to throw a new error is a victory. If you get too many compile errors, you could compile with Java 8 and just run the tests on Java 9.

Then run JDeps on your project and your dependencies. Analyze dependencies on JDK-internal APIs and take not of any Java EE modules. Also look for split packages between platform modules and application JARs. A good way to get started are the following two JDeps calls, where all your project’s dependencies are in the

libs
folder:

jdeps --jdk-internals -R --class-path 'libs/*' project.jar
jdeps -s -R --class-path 'libs/*' project.jar

Finally, search your code base for calls to

AccessibleObject::setAccessible
, casts to
URLClassLoader
, parsing of
java.version
system properties
, or handcrafting resource URLs. Put everything you found on one big list – now it’s time to analyze it.

How Bad Is It?

The problems you’ve found should fall into the two categories “I’ve seen it in before” and “What the fuck is going on?”. For the former, split it up further into “Has at least a temporary fix” and “Is a hard problem.” Particularly hard problems are removed APIs and package splits between platform modules and JARs that do not implement an endorsed standard or a standalone technology.

It’s very important not to confuse prevalence with importance. You might get about a thousand errors because a Java EE module is missing, but fixing that is trivial. You’re in big trouble, though, if your core feature depends on that one cast of the application class loader to

URLClassLoader
. Or you might have a critical dependency on a removed API but because you’ve designed your system well, it just causes a few compile errors in one subproject.

A good approach is to ask yourself for each specific problem that you don’t know a solution for off the top of your head: “How bad would it be if I cut out the troublesome code and everything that depends on it?” How much would that hurt your project?

In that vein, would it be possible to temporarily deactivate the troublesome code? Tests can be ignored, features toggled with flags. Get a sense for the how feasible it is to delay a fix and run the build or the application without it.

When you’re all done you should have a list of issues in these three categories:

  • a known problem with an easy fix
  • a known, hard problem
  • an unknown problem, needs investigation

For problems in the last two categories, you should know how dangerous they are for your project and how easy you could get by without fixing them right now.

On Estimating Numbers

Chances are, somebody wants you to make an estimate that involves some hard numbers – maybe in hours, maybe in cash. That’s tough in general, but here it is particularly problematic.

A Java 9 migration makes you face the music of decisions long past. Your project might be tightly coupled to an outdated version of that Web framework you wanted to update for years, or it might a have accrued a lot of technical debt around that unmaintained library. And unfortunately both stop working on Java 9. What you have to do now is pay back some technical debt and everybody knows that the fees and interest can be hard to estimate. Finally, just like a good boss battle, the critical problem, the one that costs you the most to fix, could very well be hidden behind a few other troublemakers, so you can’t see it until you’re in too deep.

I’m not saying these scenarios are likely just that they’re possible, so be careful with guessing how long it might take you to migrate to Java 9.

Jumping Into Action

The next step is to actually start working on the issues you collected. I recommend to not do that in isolation, but to set your build tool and continuous integration up to build your project with Java 9 and then start solving them one by one. More on that in another post.

The post Planning Your Java 9 Update appeared first on blog@CodeFX.

Five Command Line Options To Hack The Java 9 Module System

$
0
0

The Java Platform Module System not only comes with an entire set of new rules to abide by, it also introduces a host of command line options to break them. Whether you need to access internal APIs, add unforeseen modules, or extend modules with classes of your own, they have you covered. In this post I want to go over the five most important command line options that you will need to get your project to compile, test, and run in the face of various migration challenges.

Beyond presenting a few specific options, I close with some general thoughts on command line options and particularly their pitfalls.

By the way, I use

$var
as placeholders that you have to replace with the module, package, or JAR names that fix your problem.

Five Critical Command Line Options

This post covers

--add-exports
,
--add-opens
,
--add-modules
,
--add-reads
, and
--patch-module
. Let’s get it on!

Accessing Internal APIs With
--add-exports

The command line option

--add-exports $module/$package=$readingmodule
exports
$package
of $module to $readingmodule. Code in $readingmodule can hence access all public types in
$package
but other modules can not. (The option is available for the
java
and
javac
commands.)

When setting $readingmodule to

ALL-UNNAMED
, all modules in the module graph and code from the class path can access that package. When accessing internal APIs during a migrating to Java 9, you will always use that placeholder – only once your own code runs in modules does it really make sense to limit exports to specific modules.

As an example, assume you have a class that uses

com.sun.java.swing.plaf.nimbus.NimbusLookAndFeel
– during compilation you would get the following error:

error: package com.sun.java.swing.plaf.nimbus is not visible
import com.sun.java.swing.plaf.nimbus.NimbusLookAndFeel;
                              ^
  (package com.sun.java.swing.plaf.nimbus is declared
   in module java.desktop, which does not export it)
1 error

The troublesome class imports

NimbusLookAndFeel
from the encapsulated package
com.sun.java.swing.plaf.nimbus
. Note how the error message points out the specific problem, including the module that contains the class.

This clearly doesn’t work out of the box on Java 9, but what if we want to keep using it? Then we’d likely be making a mistake because there’s a standardized alternative in

javax.swing.plaf.nimbus
, but for the sake of this example let’s say we still want to use this one – maybe to interact with legacy code that can not be changed.

All we have to do to successfully compile against

com.sun.java.swing.plaf.nimbus.NimbusLookAndFeel
is to add
--add-exports java.desktop/com.sun.java.swing.plaf.nimbus=ALL-UNNAMED
to the compiler command. If we do that manually, it would like as follows:

javac
    --add-exports java.desktop/com.sun.java.swing.plaf.nimbus=ALL-UNNAMED
    --class-path $dependencies
    -d $target-folder
    $source-files

Adding exports on the command line only changes the one compilation

This way, code happily compiles against encapsulated classes. But it is important to realize that we’ve only pushed the problem to run time! Adding this export on the command line really only changes the one compilation – there is no information put into the resulting bytecode that would allow that class to access that package during execution. So we still have to figure out how to make it work at run time.

Reflectively Accessing Internal APIs With
--add-opens

The

java
option
--add-opens $module/$package=$reflectingmodule
can be used to open
$package
of $module for deep reflection to $reflectingmodule. Code in $reflectingmodule can hence reflectively access all types and members in
$package
but other modules can not.

When setting $reflectingmodule to

ALL-UNNAMED
, all modules in the module graph and code from the class path can reflectively access that package.When accessing internal APIs during a migrating to Java 9, you will always use that placeholder – only once your own code runs in modules does it really make sense to limit exports to specific modules.

A common case are dependency injection libraries like Guice that use the class loader’s internal API, which results in errors like the following:

Caused by: java.lang.reflect.InaccessibleObjectException:
Unable to make ClassLoader.defineClass accessible:
module java.base does not "opens java.lang" to unnamed module

Note how the error message points out the specific problem, including the module that contains the class. To make this work we simply need to open the package containing the class:

java
    --add-opens java.base/java.lang=ALL-UNNAMED
    --class-path $dependencies
    -jar $appjar

Adding Classes To Modules With
--patch-module

The compiler and runtime option

--patch-module $module=$artifact
merges all classes from
$artifact
into $module. There are a few things to look out for, but let’s see an example before we get to them.

When discussing split packages during migration, we looked at the example of a project that uses the annotations

@Generated
(from the java.xml.ws.annotation module) and
@Nonnull
(from a JSR 305 implementation). We discovered three things:
  • both annotations are in the
    javax.annotation
    package, thus creating a split
  • we need to add the module manually because it’s a Java EE module
  • doing so makes the JSR 305 portion of the split package invisible

We can use

--patch-module
to mend the split:

java
    --add-modules java.xml.ws.annotation
    --patch-module java.xml.ws.annotation=jsr305-3.0.2.jar
    --class-path $dependencies
    -jar $appjar

This way all classes in

jsr305-3.0.2.jar
becomes part of the module java.xml.ws.annotation and can hence be loaded for a successful execution. Yay!

There are a few things to look out for, though. First, patching a module does not automatically add it to the module graph. If it is not required explicitly, it might still need to be added with

--add-modules
. Then, classes added to a module with
--patch-module
are subject to normal accessibility rules:
  • code that depends on them needs to read the patched module, which must export the necessary packages
  • likewise these classes’ dependencies need to be in an exported package in a module read by the patched one

This might require manipulating the module graph with command line options like

--add-reads
and
--add-exports
. Since named modules can not access code from the class path, it might also be necessary to create some automatic modules.

Extending The Module Graph With
--add-modules

The option

--add-modules $modules
, which is available on
javac
and
java
, allows explicitly defining a comma-separated list of root modules beyond the initial module. (Root modules form the initial set of modules from which the module graph is built by resolving their dependencies.) This allows you to add modules (and their dependencies) to the module graph that would otherwise not show up because the initial module does not depend on them (directly or indirectly).

An important use case for

--add-modules
are Java EE modules

A particularly important use case for

--add-modules
are Java EE modules, which are not resolved by default when running an application from the class path. As an example, let’s pick a class that uses
JAXBException
from the Java EE module java.xml.bind. Here’s how to make that module available for compilation with
--add-modules
:

javac
    --class-path $dependencies
    --add-modules java.xml.bind
    -d ${output_dir}
    ${source_files}

When the code is compiled and packaged, you need to add the module again for execution:

java
    --class-path $dependencies
    --add-modules java.xml.bind
    -jar $appjar

Other use cases for

--add-modules
are optional dependencies.

The

--add-modules
option has three special values:
ALL-DEFAULT
,
ALL-SYSTEM
, and
ALL-MODULE-PATH
. The first two only work at run time and are used for very specific cases that this post does not discuss. The last one can be quite useful, though: With it, all modules on the module path become root modules and hence all of them make it into the module graph.

When adding modules it might be necessary to let other modules read them, so let’s do that next.

Extending The Module Graph With
--add-reads

The compiler and runtime option

--add-reads $module=$targets
adds readability edges from $module to all modules in the comma-separated list $targets. This allows $module to access all public types in packages exported by those modules even though $module has no
requires
clauses mentioning them.

As an example let’s turn to the ServiceMonitor application, which has a monitor.statistics module that could sometimes make use of a monitor.statistics.fancy module. Without resorting to optional dependencies (which would likely be the proper solution for this specific case), we can use

--add-modules
to add the fancy module and then
add-reads
to allow monitor.statistics to read it:

java
    --module-path mods
    --add-modules monitor.statistics.fancy
    --add-reads monitor.statistics=monitor.statistics.fancy
    --module monitor

Thoughts On Command Line Options

With Java 9, you might end up applying more command line options than ever before – it sure has been like that for me. While doing so I had a few insights that might make your life easier.

Argument Files

Command line options do not actually have to be applied to the command. An alternative are so-called argument files (or @-files), which are plain text files that can be referenced on the command line with

@<file-name>
. Compiler and runtime will then act as if the file content had been added to the command.

The example on

--patch-module
showed how to run code that uses annotations from Java EE and JSR 305:

java
    --add-modules java.xml.ws.annotation
    --patch-module java.xml.ws.annotation=jsr305-3.0.2.jar
    --class-path $dependencies
    -jar $appjar

Here,

--add-modules
and
--patch-module
are added to make the compilation work on Java 9. We could put these two lines in a file called
java-9-args
and then launch as follows:

java @java-9-args
    --class-path $dependencies
    -jar $appjar

What’s new in Java 9 is that the JVM also recognizes argument files, so they can be shared between compilation and execution.

Unfortunately, argument files don’t work with Maven because the compiler plugin already creates a file for all of its own options and Java does not supported nested argument files. Sad.

Relying On Weak Encapsulation

Don’t rely on weak encapsulation

The Java 9 runtime allows illegal access by default to code on the class path with nothing more than a warning. That’s great to run unprepared applications on Java 9, but I advise against relying on that during a proper build because it allows new illegal accesses to slip by unnoticed. Instead, I collect all the

--add-exports
and
--add-opens
I need and then activate strong encapsulation at run time with
--illegal-access=deny
.

The Pitfalls Of Command Line Options

Using command line options has a few pitfalls:

  • these options are infectious in the sense that if a JAR needs them, all of its dependencies need them as well
  • developers of libraries and frameworks that require specific options will hopefully document that their clients need to apply them, but of course nobody reads the documentation until it’s too late
  • application developers will have to maintain a list of options that merge the requirements of several libraries and frameworks they use
  • it is not easy to maintain the options in a way that allow sharing them between different build phases and execution
  • it is not easy to determine which options can be removed due to a dependency update to a Java 9 compatible version
  • it can be tricky to apply the options to the right Java processes, for example for a build tool plugin that does not run in the same process as the build tool

Command line options are a fix, not a proper solution

All of these pitfalls make one thing very clear: Command line options are a fix, not a proper solution, and they have their own long-term costs. This is no accident – they were designed to make the undesired possible. Not easy, though, or there would be no incentive to solve the underlying problem.

So do your best to only rely on public and supported APIs, not to split packages, and to generally avoid picking fights with the module system. And, very importantly, reward libraries and frameworks that do the same! But the road to hell is paved with good intentions, so if everything else fails, use every command line option at your disposal.

Reflection

These five options should get you through most thickets:

  • --add-exports
    to export a package, which makes its public types and members accessible (
    javac
    and
    java
    )
  • --add-exports
    to open a package, which makes all its types and members accessible (
    java
    )
  • --patch-module
    adds classes to a specific module
  • --add-modules
    adds the listed modules and their transitive dependencies to the module grap
  • --add-reads
    makes one module read another

As discussed, command line options come with a set of pitfalls, so make sure to only use them where absolutely necessary and work to reduce those cases.

The post Five Command Line Options To Hack The Java 9 Module System appeared first on blog@CodeFX.

Unified Logging In Java 9 With The -Xlog Option

$
0
0

Java 9 comes with a unified logging architecture (JEP 158) that pipes a lot of messages that the JVM generates through the same mechanism, which can be configured with the

-Xlog
option. This gives uniform access to log messages from different subsystems such as class loading, threading, the garbage collector, the module system, or the interaction with the underlying operating system.

The

-Xlog
option can be a bit intimidating, so in this post we will master it step by step, learning how to use it to select which messages and information to show.

What Is Unified Logging?

The unified logging infrastructure is very similar to known logging frameworks

The JVM-internal, unified logging infrastructure is very similar to known logging frameworks like Log4j or Logback that you might have used for your application. It generates textual messages, attaches some meta information like tags (describing the originating subsystem), a log level (describing the importance of the message), and time stamps before printing them somewhere. The logging output can be configured according to your needs.

Logging can be activated with the

java
option
-Xlog
. This is the only flag regarding this mechanism – any further configuration is immediately appended to that option. Configurable aspects of logging are:
  • which messages to log (by tag and/or by log level)
  • which information to include (for example time stamps and process IDs)
  • which output to use (for example a file)

We’ll look at each of them in turn, but before doing that, we can have a look at the kind of messages

-Xlog
produces. Simply execute
java -Xlog
(maybe append
-version
to get rid of the helpful but long display of command line options) and have a look at the output – of which there is a lot. One of the first messages tells us that the HotSpot virtual machine begins its work:

$ java -Xlog -version

# truncated a few messages
> [0.002s][info][os       ] HotSpot is running with glibc 2.23, NPTL 2.23
# truncated a lot of messages

It shows how long the JVM has been running (2 ms), the message’s log level (

info
), its tags (only
os
), and the actual message.
Let’s see how to influence these details.

Defining Which Messages Should Be Shown

The log level and tags can be used to define what exactly the logs should show by defining pairs

<tag-set>=<level>
, which are called selectors. All tags can be selected with
all
and the level is optional and defaults to
info
. Here’s how to use it:

$ java -Xlog:all=warning -version

# no log messages; great, warning free!

Let’s try something else:

$ java -Xlog:logging=debug -version

> [0.034s][info][logging] Log configuration fully initialized.
> [0.034s][debug][logging] Available log levels:
    off, trace, debug, info, warning, error
> [0.034s][debug][logging] Available log decorators: [...]
> [0.034s][debug][logging] Available log tags: [...]
> [0.034s][debug][logging] Described tag combinations:
> [0.034s][debug][logging]  logging: Logging for the log framework itself
> [0.034s][debug][logging] Log output configuration:
> [0.034s][debug][logging] #0: stdout [...]
> [0.034s][debug][logging] #1: stderr [...]]

Lucky shot! I had to truncate the output but trust me, there’s a lot of helpful information in those messages. You don’t have to take that route, though,

-Xlog:help
shows the same information but more beautifully formatted (see below).

A little surprising (at least at first) is the detail that messages only match a selector if their tags exactly match the given ones. Given ones? Plural? Yes, a selector can name several tags by concatenating them with

+
. Still, a message has to contain exactly those to be selected.

Hence, using

gc
(for garbage collection) versus
gc+heap
, for example, should select different messages. This is indeed the case:

$ java -Xlog:gc -version

> [0.009s][info][gc] Using G1

$ java -Xlog:gc+heap -version

> [0.006s][info][gc,heap] Heap region size: 1M

Several selectors can be defined – they just have to be separated with commas:

$ java -Xlog:gc,gc+heap -version

> [0.007s][info][gc,heap] Heap region size: 1M
> [0.009s][info][gc     ] Using G1

Using this strategy it is very cumbersome to get all messages that contain a certain flag. Luckily, there is an easier way to do that, namely the wildcard

*
, which can be used with a single tag to define a selector:

$ java -Xlog:gc*=debug -version

> [0.006s][info][gc,heap] Heap region size: 1M
> [0.006s][debug][gc,heap] Minimum heap 8388608  Initial heap 262144000
    Maximum heap 4192206848
# truncated about two dozen message
> [0.072s][info ][gc,heap,exit         ] Heap
# truncated a few messages showing final GC statistics

Using selectors, there are three easy steps to get to know a subsystem of the JVM:

  • Find interesting tags in the output of
    java -Xlog:help
    .
  • Use them with
    -Xlog:tag_1*,tag_2*,tag_n*
    to display all
    info
    messages that were tagged with any of them.
  • Selectively switch to lower log levels with
    -Xlog:tag_1*=debug
    .

Defining Where Messages Should Go

Compared to the non-trivial selectors, the output configuration is really simple. It comes after the selectors (separated by a colon) and has three possible locations:

  • stdout
    : The default output. On the console that is the terminal window unless redirected, in IDEs it is often shown in a separate tab or view.
  • stderr
    : The default error output. On the console that is the terminal window unless redirected, in IDEs it is usually shown in the same tab/view as
    stdout
    but printed red.
  • file=<filename>
    : Defines a file to pipe all messages into. Putting in
    file=
    is optional.

Unlike with common logging frameworks, it is unfortunately not possible to use two output options simultaneously.

Here’s how to put all

debug
messages in the file
application.log
:

$ java -Xlog:all=debug:file=application.log -version

More output options are available that allow log file rotation based on file size and number of files to rotate.

Defining What Messages Should Say

As I said in the introduction, each message consist of text and meta-information. Which of these additional pieces of information will be printed, is configurable by selecting decorators, which are listed in the following table. This happens after the output location and another colon.

Option Description
time
Current time and date in ISO-8601 format.
uptime
Time since the start of the JVM in seconds and milliseconds (e.g., 6.567s).
timemillis
The same value as generated by
System.currentTimeMillis()
.
uptimemillis
Milliseconds since the JVM started.
timenanos
The same value as generated by
System.nanoTime()
.
uptimenanos
Nanoseconds since the JVM started.
pid
The process identifier.
tid
The thread identifier.
level
The level associated with the log message.
tags
The tag-set associated with the log message.

Let’s say we want to print the time stamp, the uptime in milliseconds, and the thread ID for all garbage collection debug messages to the console. Here’s how to do that:

$ java -Xlog:gc*=debug:stdout:time,uptimemillis,tid -version

# truncated messages
> [2017-02-01T13:10:59.689+0100][7ms][18607] Heap region size: 1M

Putting The Pieces Together

Formally, the

-Xlog
option has this syntax:

-Xlog:<selectors>:<output>:<decorators>:<output-options>

Each of the parameters following

-Xlog
is optional but if one is used, so have to be all the others that come before it.
  • Selectors are pairs of tag sets and log levels. This part is also called the what-expression, a phrase you will likely encounter when the configuration is not syntactically correct.
  • With
    output
    the target location for the log messages (in short, the terminal window or a log file) can be defined.
  • Decorators can be used to define what information the messages should include.
  • Yes, annoyingly the output mechanism and further output options are split, with decorators in between.

For more details, have a look at the online documentation or the output of

java -Xlog:help
:

-Xlog Usage: -Xlog[:[what][:[output][:[decorators][:output-options]]]]
        where 'what' is a combination of tags and levels on the form
            tag1[+tag2...][*][=level][,...]
        Unless wildcard (*) is specified, only log messages tagged with
            exactly the tags specified will be matched.

Available log levels:
    off, trace, debug, info, warning, error

Available log decorators:
    time (t), utctime (utc), uptime (u), timemillis (tm), uptimemillis (um),
    timenanos (tn), uptimenanos (un), hostname (hn), pid (p), tid (ti),
    level (l), tags (tg)
    Decorators can also be specified as 'none' for no decoration.

Available log tags:
    [... many, many tags ... ]
    Specifying 'all' instead of a tag combination matches all tag combinations.

Available log outputs:
    stdout, stderr, file=<filename>
    Specifying %p and/or %t in the filename will expand to the JVM's PID and
    startup timestamp, respectively.

Some examples:
    [... a few helpful examples to get you going ... ]

Reflection

In Java 9, most JVM subsystems use a unified logging mechanism. This makes it easier than before to configure logging, so you get the exact messages you need. The

-Xlog
option allows the definition of
<selectors>
,
<output>
,
<decorators>
, and
<output-options>
, which can be used to define precisely what gets logged, where the output shows up, and what it looks like.

The post Unified Logging In Java 9 With The -Xlog Option appeared first on blog@CodeFX.

Code First Java 9 Module System Tutorial

$
0
0

The Java Platform Module System (JPMS) brings modularization to Java and the JVM and it changes how we program in the large. To get the most out of it, we need to know it well, and the first step is to learn the basics. In this tutorial we’ll take an existing demo application and modularize it with Java 9. We’ll create module declarations (

module-info.java
) and use the module path to compile, package, and run the application – code first, explanations second, so you can cut to the chase.

The project is the same one I use in my book on the module system – it’s online at GitHub, so head over there if you want to follow along. All commands like

javac
,
jar
, and
java
refer to the Java 9 variants.

The Application Before The Module System

I promised you code first, but introducing an entire application, even just a small one, that way would be a waste of your time – particularly since we won’t need much of that code later on. So I’ll do this the proper way and bore you with words.

Let’s imagine a network of services that cooperate to delight our users; maybe a social network or a video platform. We want to monitor those services to determine how healthy the system is and spot problems when they occur (instead of when customers report them). This is where the example application, the ServiceMonitor comes in: It (surprise!) monitors these services.

As luck would have it, the services already collect the data we want, so all the ServiceMonitor needs to do is query them periodically. Unfortunately not all services expose the same REST API – two generations are in use, Alpha and Beta. That’s why

ServiceObserver
is an interface with two implementations.

Once we have the diagnostic data, in the form of a

DiagnosticDataPoint
, they can be fed to a
Statistician
, which aggregates them to
Statistics
. These, in turn, are stored in a
StatisticsRepository
as well as made available via REST by
MonitorServer
. The
Monitor
class ties everything together.

All in all, we end up with these types:

  • DiagnosticDataPoint
    : service data for a time interval
  • ServiceObserver
    : interface for service observation that returns
    DiagnosticDataPoint
  • AlphaServiceObserver
    and
    BetaServiceObserver
    : each observes a variant of services
  • Statistician
    : computes
    Statistics
    from
    DiagnosticDataPoint
  • Statistics
    : holds the computed statistics
  • StatisticsRepository
    : stores and retrieve
    Statistics
  • MonitorServer
    : answers REST calls for the statistics
  • Monitor
    : ties everything together
ServiceMonitor's classes

The application depends on the Spark micro web framework, which can be found in the

libs
folder together with its transitive dependencies.

Compiling and running the application is straight forward:

# compile
$ javac
    --class-path 'libs/*'
    -d classes/monitor ${source-files}
# package
$ jar --create
    --file jars/monitor.jar
    --manifest=${manifest}
    -C classes/monitor .
# run
$ java
    --class-path 'libs/*'
    -jar jars/advent.jar

Entering Module Land

The next step is small but important. It changes nothing about the code or its organization, but moves all of it into a single module by creating a file

module-info.java
in the project’s root with the following content:

module monitor {
    requires spark.core
}

We can then compile, package, and run it as follows:

$ javac
    --module-path libs
    -d classes/monitor ${source-files}
$ jar --create
    --file mods/monitor.jar
    --main-class monitor.Main
    -C classes/monitor .
$ java
    --module-path mods
    --module monitor

Very similar to what we did with the non-modularized version, except we’re now using something called a “module path” and need no manifest to define the project’s main class. Let’s see how that works.

Modules

Modules are like JARs with additional characteristics

The basic building block of the JPMS are modules (another big surprise). Like JARs, they are a container for types and resources; but unlike JARs, they have additional characteristics – these are the most fundamental ones:

  • a name, preferably one that is globally unique
  • declarations of dependencies on other modules
  • a clearly defined API that consists of exported packages

The JDK was split into about a hundred modules. You can list them with

java --list-modules
and look at an individual module with
java --describe-module ${module}
. Go ahead, give it a try with java.sql or java.logging:

$ java --describe-module java.sql

> java.sql@9
> exports java.sql
> exports javax.sql
> exports javax.transaction.xa
> requires java.logging transitive
> requires java.base mandated
> requires java.xml transitive
> uses java.sql.Driver

A module’s properties are defined in a module declaration, a file

module-info.java
in the project’s root, which looks as follows:

module ${module-name} {
    requires ${module-name};
    exports ${package-name};
}

It gets compiled into a

module-info.class
, called module descriptor, and ends up in the JAR’s root. This descriptor is the only difference between a plain JAR and a modular JAR.

Let’s go through the three module properties one by one: name, dependencies, exports.

Name

The most basic property that JARs are missing is a name that compiler and JVM can use to identify it. It is hence the most prominent characteristic of a module. We will have the possibility and even the obligation to give every module we create a name.

The best name a module is the reverse-domain naming scheme that is already commonly used for packages

Naming a module will often be pretty natural as most tools we use on a daily basis, be it IDEs, build tools, or even issue trackers and version control systems, already have us name our projects. But while it makes sense to take that name as a springboard on the search for a module name, it is important to choose wisely!

The module system leans heavily on a module’s name. Conflicting or evolving names in particular cause trouble, so it is important that the name is:

  • globally unique
  • stable

The best way to achieve that is the reverse-domain naming scheme that is already commonly used for packages.

In the example of the ServiceMonitor application, we would choose a module name like org.codefx.monitor, but that would crowd the examples, so I’ll stick to the shorter monitor:

module monitor {

}

Dependencies

All dependencies have to be made explicit with

requires
clauses

Another thing we missed in JARs was the ability to declare dependencies, but with the module system, these times are over: Dependencies have to be made explicit – all of them, on JDK modules as well as on third-party libraries or frameworks.

Dependencies are declared with

requires
clauses, which consist of the keyword itself followed by a module name. When scanning modules, the JPMS builds a module graph, where modules are nodes and
requires
clauses get turned into so-called readability edges – if module monitor requires module spark.core, then at runtime monitor reads spark.core.

The module system will throw an error if it cannot find a required module with the right name, which means compiling as well as launching an application will fail if modules are missing. This achieves reliable configuration one of the goals of the module system, but can be prohibitively strict – check my post on optional dependencies to see a more lenient alternative.

ServiceMonitor needs code from the JDK and its dependency Spark. All JDK code it needs can be found in the module java.base, the so-called base module. Because it contains essential types like

Object
, all Java code needs it and so it doesn’t need to be required explicitly. Spark, on the other hand, can be referred to as spark.core, which yields:

module monitor {
    requires spark.core
}

Exports

A module’s API is defined by its

exports
clauses

A module lists the packages it exports. For code in one module (say monitor) to access types in another (say

Spark
in spark.core), the following accessibility rules must be fulfilled:
  • the accessed type (
    Spark
    ) must be public
  • the package containing the type (also called
    spark
    ) must be exported by its module (spark.core)
  • the accessing module (monitor) must read the accessed one (spark.core), which is typically achieved by requiring it (
    monitor requires spark.core
    )

Reflection lost its superpowers

If any of these rules are violated at compile or run time, the module systems throws an error. This means that

public
is no longer really public. A public type in a non-exported package is as inaccessible to the outside world as a non-public type in an exported package. Also note that reflection lost its superpowers. It is bound by the exact same accessibility rules unless command line flags are used.

Since ServiceMonitor bundles the entire application in a single module (for now), no outside code needs to access any code, so we don’t actually have to export anything.

Module Path

We now know how we can define modules and their essential properties. What’s still a little unclear is how exactly we tell the compiler and runtime about them. The answer is a new concept that parallels the class path:

The module path is a list whose elements are artifacts or directories that contain artifacts. Depending on the operating system, module path elements are either separated by

:
(Unix-based) or
;
(Windows). It is used by the module system to locate required modules that are not found among the platform modules. Both
javac
and
java
as well as other module-related commands can process it–the command line options are
--module-path
and
-p
.

All artifacts on the module path are turned into modules. This is even true for plain JARs, which get turned into automatic modules.

Compiling, Packaging, Running

Compiling works much like before:

$ javac
    --module-path libs
    -d classes/monitor ${source-files}

The module system kicks in as soon as a

module-info.java
is among the source files. All non-JDK dependencies the module under compilation requires need to be on the module path, in this case in the
libs
folder.

Packaging with JAR is unchanged as well. The only difference is that you no longer need a manifest to declare an application’s entry point – you can use

--main-class
for that:

$ jar --create
    --file mods/monitor.jar
    --main-class monitor.Main
    -C classes/monitor .

Finally, launching looks a little different. We use the module path instead of the class path to tell the JPMS where to find modules. All we need to do beyond that is to name the main module with

--module
:

$ java
    --module-path mods
    --module monitor

Done!

Splitting Into Modules

Now that we got one module going, it’s time to really start using the module system and split the ServiceMonitor up. For an application of this size it is of course ludicrous to turn it into several modules, but it’s a demo, so here we go.

The most common way to modularize applications is a separation by concerns. ServiceMonitor has the following, with the related types in parenthesis:

  • collecting data from services (
    ServiceObserver
    ,
    DiagnosticDataPoint
    )
  • aggregating data into statistics (
    Statistician
    ,
    Statistics
    )
  • persisting statistics (
    StatisticsRepository
    )
  • exposing statistics via a REST API (
    MonitorServer
    )

But not only the domain logic generates requirements. There are also technical ones:

  • data collection must be hidden behind an API
  • Alpha and Beta services each require a separate implementation of that API (
    AlphaServiceObserver
    and
    BetaServiceObserver
    )
  • orchestration of all concerns (
    Monitor
    )

This results in the following modules with the mentioned publicly visible types:

  • monitor.observer (
    ServiceObserver
    ,
    DiagnosticDataPoint
    )
  • monitor.observer.alpha (
    AlphaServiceObserver
    )
  • monitor.observer.beta (
    BetaServiceObserver
    )
  • monitor.statistics (
    Statistician
    ,
    Statistics
    )
  • monitor.persistence (
    StatisticsRepository
    )
  • monitor.rest (
    MonitorServer
    )
  • monitor (
    Monitor
    )

Superimposing these modules over the class diagram, it is easy to see the module dependencies emerge:

Monitor's modules

Reorganizing Source Code

A real-life project consists of myriad files of many different types. Obviously, source files are the most important ones but nonetheless only one kind of many – others are test sources, resources, build scripts or project descriptions, documentation, source control information, and many others. Any project has to choose a directory structure to organize those files and it is important to make sure it does not clash with the module system’s characteristics.

If you have been following the module system’s development under Project Jigsaw and studied the official quick start guide or some early tutorials, you might have noticed that they use a particular directory structure, where there’s a

src
directory with a subdirectory for each project. That way ServiceMonitor would look as follows:

ServiceMonitor
 + classes
 + mods
 - src
    + monitor
    - monitor.observer
       - monitor
          - observer
             DiagnosticDataPoint.java
             ServiceObserver.java
       module-info.java
    + monitor.observer.alpha
    + monitor.observer.beta
    + monitor.persistence
    + monitor.rest
    + monitor.statistics
 - test-src
    + monitor
    + monitor.observer
    + monitor.observer.alpha
    + monitor.observer.beta
    + monitor.persistence
    + monitor.rest
    + monitor.statistics

This results in a hierarchy

concern/module
and I don’t like it. Most projects that consist of several sub-projects (what we now call modules) prefer separate root directories, where each contains a single module’s sources, tests, resources, and everything else mentioned earlier. They use a hierarchy
module/concern
and this is what established project structures provide.

The default directory structure, implicitly understood by tools like Maven and Gradle, implement that hierarchy. First and foremost, they give each module its own directory tree. In that tree the

src
directory contains production code and resources (in
main/java
and
main/resources
, respectively) as well as test code and resources (in
test/java
and
test/resources
, respectively):

ServiceMonitor
 + monitor
 - monitor.observer
    - src
       - main
          - java
             - monitor
                - observer
                   DiagnosticDataPoint.java
                   ServiceObserver.java
             module-info.java
          + resources
       + test
          + java
          + resources
    + target
 + monitor.observer.alpha
 + monitor.observer.beta
 + monitor.persistence
 + monitor.rest
 + monitor.statistics

I will organize the ServiceMonitor almost like that, with the only difference that I will create the bytecode in a directory

classes
and JARS in a directory
mods
, which are both right below
ServiceMonitor
, because that makes the scripts shorter and more readable.

Let’s now see what those declarations infos have to contain and how we can compile and run the application.

Declaring Modules

We’ve already covered how modules are declared using

module-info.java
, so there’s no need to go into details. Once you’ve figured out how modules need to depend on one another (your build tool should know that; otherwise ask JDeps), you can put in
requires
clauses and the necessary
exports
emerge naturally from imports across module boundaries.

module monitor.observer {
    exports monitor.observer;
}

module monitor.observer.alpha {
    requires monitor.observer;
    exports monitor.observer.alpha;
}

module monitor.observer.beta {
    requires monitor.observer;
    exports monitor.observer.beta;
}

module monitor.statistics {
    requires monitor.observer;
    exports monitor.statistics;
}

module monitor.persistence {
    requires monitor.statistics;
    exports monitor.persistence;
}

module monitor.rest {
    requires spark.core;
    requires monitor.statistics;
    exports monitor.rest;
}

module monitor {
    requires monitor.observer;
    requires monitor.observer.alpha;
    requires monitor.observer.beta;
    requires monitor.statistics;
    requires monitor.persistence;
    requires monitor.rest;
}

By the way, you can use JDeps to create an initial set of module declarations. Whether created automatically or manually, in a real-life project you should verify whether your dependencies and APIs are as you want them to be. It is likely that over time, some quick fixes introduced relationships that you’d rather get rid of. Do that now or create some backlog issues.

Compiling, Packaging, And Running

Very similar to before when it was only a single module, but more often:

$ javac
    -d classes/monitor.observer
    ${source-files}
$ jar --create
    --file mods/monitor.observer.jar
    -C classes/monitor.observer .

# monitor.observer.alpha depends on monitor.observer,
# so we place 'mods', which contains monitor.observer.jar,
# on the module path
$ javac
    --module-path mods
    -d classes/monitor.observer.alpha
    ${source-files}
$ jar --create
    --file mods/monitor.observer.alpha.jar
    -C classes/monitor.observer.alpha .

# more of the same ... until we come to monitor,
# which once again defines a main class
$ javac
    --module-path mods
    -d classes/monitor
    ${source-files}
$ jar --create
    --file mods/monitor.jar
    --main-class monitor.Main
    -C classes/monitor .

Congratulations, you’ve got the basics covered! You now know how to organize, declare, compile, package, and launch modules and understand what role the module path, the module graph, and modular JARs play.

On The Horizon

If you weren’t so damn curious this post could be over now, but instead I’m going to show you a few of the more advanced features, so you know what to read about next.

Implied Readability

The ServiceMonitor module monitor.observer.alpha describes itself as follows:

module monitor.observer.alpha {
    requires monitor.observer;
    exports monitor.observer.alpha;
}

Instead it should actually do this:

module monitor.observer.alpha {
    requires transitive monitor.observer;
    exports monitor.observer.alpha;
}

Spot the

transitive
in there? It makes sure that any module reading monitor.observer.alpha also reads monitor.observer. Why would you do that? Here’s a method from alpha‘s public API:

public static Optional<ServiceObserver> createIfAlphaService(String service) {
    // ...
}

It returns an

Optional<ServiceObserver>
, but
ServiceObserver
comes from the monitor.observer module – that means every module that wants to call alpha‘s
createIfAlphaService
needs to read monitor.observer as well or such code won’t compile. That’s pretty inconvenient, so modules like alpha that use another module’s type in their own public API should generally require that module with the
transitive
modifier.

There are more uses for implied readability.

Optional Dependencies

This is quite straight-forward: If you want to compile against a module’s types, but don’t want to force its presence at runtime you can mark your dependency as being optional with the

static
modifier:

module monitor {
    requires monitor.observer;
    requires static monitor.observer.alpha;
    requires static monitor.observer.beta;
    requires monitor.statistics;
    requires monitor.persistence;
    requires static monitor.rest;
}

In this case monitor seems to be ok with the alpha and beta observer implementations possibly being absent and it looks like the REST endpoint is optional, too.

There are a few things to consider when coding against optional dependencies.

Qualified Exports

Regular exports have you make the decision whether a package’s public types are accessible only within the same module or to all modules. Sometimes you need something in between, though. If you’re shipping a bunch of modules, you might end up in the situation, where you’d like to share code between those modules but not outside of it. Qualified exports to the rescue!

module monitor.util {
    exports monitor.util to monitor, monitor.statistics;
}

This way only monitor and monitor.statistics can access the

monitor.util
package.

Open Packages And Modules

I said earlier that reflection’s superpowers were revoked – it now has to play by the same rules as regular access. Reflection still has a special place in Java’s ecosystem, though, as it enables frameworks like Hibernate, Spring and so many others.

The bridge between those two poles are open packages and modules:

module monitor.persistence {
    opens monitor.persistence.dtos;
}

// or even

open module monitor.persistence.dtos { }

An open package is inaccessible at compile time (so you can’t write code against its types), but accessible at run time (so reflection works). More than just being accessible, it allows reflective access to non-public types and members (this is called deem reflection). Open packages can be qualified just like exports and open modules simply open all their packages.

Services

Instead of having the main module monitor depend on monitor.observer.alpha and monitor.observer.beta, so it can create instances of

AlphaServiceObserver
and
BetaServiceObserver
, it could let the module system make that connection:

module monitor {
    requires monitor.observer;
    // monitor wants to use a service
    uses monitor.observer.ServiceObserverFactory;
    requires monitor.statistics;
    requires monitor.persistence;
    requires monitor.rest;
}

module monitor.observer.alpha {
    requires monitor.observer;
    // alpha provides a service implementation
    provides monitor.observer.ServiceObserverFactory
        with monitor.observer.alpha.AlphaServiceObserverFactory;
}

module monitor.observer.beta {
    requires monitor.observer;
    // beta provides a service implementation
    provides monitor.observer.ServiceObserverFactory
        with monitor.observer.beta.BetaServiceObserverFactory;
}

This way, monitor can do the following to get an instance of each provided observer factory:

List<ServiceObserverFactory> observerFactories = ServiceLoader
    .load(ServiceObserverFactory.class).stream()
    .map(Provider::get)
    .collect(toList());

It uses the

ServiceLoader
API, which exists since Java 6, to inform the module system that it needs all implementations of

ServiceObserverFactory
. The JPMS will then track down all modules in the module graph that provide that service, create an instance of each and return them.

There are two particularly interesting consequences:

  • the module consuming the service does not have to require the modules providing it
  • the application can be configured by selecting which modules are placed on the module path

Services are a wonderful way to decouple modules and its awesome that the module system gives this mostly ignored concept a second life and puts it into a prominent place.

Reflection

Ok, we’re really done now and you’ve learned a lot. Quick recap:

  • a module is a run-time concept created from a modular JAR
  • a modular JAR is like any old plain JAR, except that it contains a module descriptor
    module-info.class
    , which is compiled from a module declaration
    module-info.java
  • the module declaration gives a module its name, defines its dependencies (with
    requires
    ,
    requires static
    , and
    requires transitive
    ) and API (with
    exports
    and
    exports to
    ), enables reflective access (with
    open
    and
    opens to
    ) and declares use or provision of services
  • modules are placed on the module path where the JPMS finds them during module resolution, which is the phase that processes descriptors and results in a module graph

If you want to learn more about the module system, read the posts I linked above, check the JPMS tag, or get my book The Java Module System (Manning).

The post Code First Java 9 Module System Tutorial appeared first on blog@CodeFX.


Making JSR 305 Work On Java 9

$
0
0

Do you use JSR 305 annotations like

@Nonnull
or
@Nullable
together with annotations from the
javax.annotation
package
, for example
@Generated
or
@PostConstruct
? If so, then migrating to Java 9 won’t be straight-forward due to split packages. Here’s how to fix that problem.

Overview

I use a little demo project that you can have a closer look at over at GitHub. We’re only concerned with the class

MixedJsr305AndJavaxAnnotation
, which uses both
@Nonnull
(from JSR 305) and
@Generated
(from the JDK).

What’s the problem?

The Java Platform Module System doesn’t like it when two modules contain types in the same package (totally random example:

javax.annotation.Nonnull
from JSR 305 and
javax.annotation.Generated
from the JDK). This is called a split package and I explain it in more detail in the Java 9 migration guide. All you need to know for now is how compiler and runtime react to it:
  • split between named modules (including platform, application, and automatic modules): the module system throws an error

module org.codefx.demo.java9_.jsr305_
    reads package javax.annotation from both jsr305 and java.annotation

  • split between named module (e.g. from the JDK) and the class path (which ends up in the unnamed module): class path portion of the package becomes invisible and compiler and runtime will complain of missing classes even though they are present on the class path
  • split between JARs on the class path: no problem, works as before

In which case you are depends on your setup. Let’s step through the possibilities one by one and see which work and which don’t. But before we come to that, there are two other things to talk about.

Alternatives To JSR 305

First I want to quote Stephen Connolly:

FTR there were never any annotations published by JSR 305

As a result, if you redistribute an Oracle JRE with the “annotations claimed to be JSR 305 but never actually officially published from the JSR spec page” then you are violating the Oracle JVM redistribution terms.

Please stop referring to these as JSR 305 annotations… at best they are “presumed JSR 305 annotations”

He’s totally right and if you have the chance you should move away from the “presumed JSR 305 annotations”. If it’s

null
-safety you’re after, I recommend using
Optional
or one of the other sets of annotations, for example those from SpotBugs, Eclipse (works outside the IDE with the Eclipse compiler), or JetBrains (code analysis can be run outside IntelliJ).

If moving away from these annotations is not an option for your project (at least not in the short term), you will unfortunately have to read on.

Platform Annotations

The other thing we have to talk about concerns the second actor in this drama: the platform module containing the

javax.annotation
package, namely java.xml.ws.annotation. This is a Java EE module and since Oracle got rid of Java EE (now Enterprise Eclipse for Java, EE4J), it does not resolve such modules by default. You’ll have to use the
--add-modules
command line option
to get it into the readability graph. We’ll see shortly how that affects your situation.

With all of that settled, let’s see where you’re at.

Non-modular Project

Assuming you are just trying to get your code to compile on Java 9 you will be in the situation that your project is non-modular, i.e. it has no module declaration

module-info.java
.

Missing Platform Module

If you’re simply running your Java 8 build on Java 9, you will likely see errors like this one:

$ javac
    --class-path deps/jsr305-3.0.2.jar
    -d build
    src/MixedJsr305AndJavaxAnnotation.java

> error: cannot find symbol
> import javax.annotation.Generated;
>                        ^
>     symbol:   class Generated
>     location: package javax.annotation

This has nothing to do yet with split packages. It just means that your code depends on the

javax.annotation
package that shipped with the JDK, but that the module containing it was not resolved. It needs to be added with
--add-modules
, which brings you to the next problem.

Invisible JSR-305 Annotations

Once you use

--add-modules java.xml.ws.annotation
to make sure java.xml.ws.annotation is part of the readability graph, the
javax.annotation
package is split between that module and the class path, which, as explained above, leads to the class path portion becoming invisible:

$ javac
    --class-path deps/jsr305-3.0.2.jar
    --add-modules java.xml.ws.annotation
    -d build
    src/MixedJsr305AndJavaxAnnotation.java


> error: cannot find symbol
> import javax.annotation.Nonnull;
>                        ^
>     symbol:   class Nonnull
>     location: package javax.annotation

(That error looks very similar to the last one, but now it’s

@Nonnull
that can’t be found.)

Patching The Platform Module

The hammer that hits all the split-package-nails is the

--patch-module
option. With it you can instruct the module system to pick some classes and stuff them into a module. Because it messes with module contents, it should be seen as a last resort.

$ javac
    --add-modules java.xml.ws.annotation
    --patch-module java.xml.ws.annotation=deps/jsr305-3.0.2.jar
    -d build
    src/org/codefx/demo/java9/jsr305/MixedJsr305AndJavaxAnnotation.java

This approach only works until java.xml.ws.annotation is removed in 2018

Due to

--patch-module
the module system treats all classes in
jsr305-3.0.2.jar
as if they were in java.xml.ws.annotation. This mends the split and lets the code compile. During execution you need to repeat the
--add-modules
and
--patch-module
options.

Note that this way

jsr305-3.0.2.jar
no longer needs to be on the class path. In this particular example it leads to
--class-path
becoming superfluous because no other dependency was needed.

This approach works, but only until java.xml.ws.annotation is removed from the JDK (likely in March or September 2018). Read on for a long-term solution.

Keeping The Split On The Class Path

As mentioned earlier, a split between JARs on the class path causes no problems. You can use this to your advantage by fulfilling your dependency on the JDK API with an external JAR, namely

javax.annotation:javax.annotation-api
. With it, the code can be compiled as follows:

$ javac
    --class-path deps/jsr305-3.0.2.jar:deps/javax.annotation-api-1.3.1.jar
    -d build
    src/org/codefx/demo/java9/jsr305/MixedJsr305AndJavaxAnnotation.java

This is the most elegant solution as it requires no hacking of the module system and no special command line options. Just adding a small dependency solves everything. At least until you try to modularize…

Modular Project

If you’re modularizing your project, there’s no good solution available. As far as I know there is no artifact that contains both parts of the split, i.e. the content of both

javax.annotation-api-1.3.1.jar
and
jsr305-3.0.2.jar
. Taken together:
  • you need two artifacts to fulfill these dependencies
  • your module declaration needs to require them in order for your code to be able to use them
  • requiring them means that they must be named modules (explicit or automatic ones)
  • you have two named modules that split a package

This leaves you with picking one of these two artifacts (requiring it as an automatic module until it’s modularized) and patching the other one into it.

Alternatively you could create an artifact that combines the other two and make it available in your organization’s repository (e.g. Sonatype’s Nexus). It would be nice to upload that to Maven Central, but I’m afraid there could be licensing issues with publishing code in the

javax.*
packages (I don’t actually know, though).

Reflection

If you use JSR-305 annotations (e.g.

@Nonnull
or
@Nullable
) together with annotations from the
javax.annotation
package
(e.g.
@Generated
or
@PostConstruct
), then running on Java 9 will likely create a split package that you need to work around.

As long as your code is non-modular, the best solution is to explicitly require both

jsr305
and
javax.annotation-api
, so they both end up on the class path, where package splits don’t matter. If you modularize your code, there is no way around
--patch-module
unless you want to create your own artifact combining the other two.

The post Making JSR 305 Work On Java 9 appeared first on blog@CodeFX.

First Contact With ‘var’ In Java 10

$
0
0

Java 10 will be released on March 20th 2018 and all features that want to be shipped with it must be merged into the main development line by December 14th. Of the few features that target Java 10 as of yet, local-variable type inference (JEP 286) is surely the most interesting one. It brings the

var
keyword to Java and gives you the option to cut variable declarations short:

var users = new ArrayList<User>();

And that’s it, thanks for reading!

Nah, I’m sure you’re interested to learn more. In this post I’ll discuss where

var
applies and where it doesn’t, how it impacts readability, and what happened to
val
.

Overview

This post is the first in a series about Java 10. I created a demo project for all features on GitHub, which also explains how to set up JDK 10. Check it out if you want to play with the new features yourself.

Replacing Type Declarations With var

As a Java developer we’re used to typing types twice, once for the variable declaration and once again for the following constructor:

URL codefx = new URL("http://codefx.org")

We also often declare types for variables that are used just once and on the next line:

URL codefx = new URL("http://codefx.org")
URLConnection connection = codefx.openConnection();
Reader reader = new BufferedReader(
    new InputStreamReader(connection.getInputStream()));

From Java 10 on developers can choose to let the compiler infer types by using

var

This is not particularly terrible, but it is somewhat redundant. And while IDEs can help a lot with writing such code, readability suffers when variable names jump around a lot because their types have very different character counts or when developers avoid declaring intermediate variables because type declarations would eat up a lot of attention without adding much value.

From Java 10 on, developers have an alternative and can choose to let the compiler infer the type by using

var
:

var codefx = new URL("http://codefx.org");
var connection = codefx.openConnection();
var reader = new BufferedReader(
    new InputStreamReader(connection.getInputStream()));

The compiler uses the initializer’s type

When processing

var
, the compiler looks at the right hand side of the declaration, the so-called initializer, and uses its type for the variable. And not just for internal bookkeeping, it writes that type into the resulting bytecode.

As you can see, this saves a few characters when typing, but more importantly it deduplicates redundant information and neatly aligns the variable’s names, which eases reading them. The cost is obvious: Some variables’ types, of

connection
for example, are not immediately obvious. IDEs can of course show them on demand, but that doesn’t help in any other environment (think code reviews).

var
is a reserved type name

By the way, in case you’re worried about clashes with methods and variables named

var
: don’t be. Technically,
var
is not a keyword, but a reserved type name, meaning it can only be used in places where the compiler expects a type name, but everywhere else it’s a valid identifier. That means that only classes called
var
will no longer work, but that shouldn’t happen particularly often.

Local-variable type inference looks like a straight-forward feature, but that’s deceptive. You might already have some questions:

  • bleh, is this Java or JavaScript?
  • where can I use this?
  • won’t
    var
    hurt readability?
  • why is there no
    val
    or
    let
    ?

Let’s go through them one by one.

No, This Is Not JavaScript

This does not change Java’s commitment to static typing

I want to start by stressing that

var
does not change Java’s commitment to static typing by one iota. The compiler infers all involved types and puts them into the class files as if you typed them yourself.

Case in point, here’s the result of IntelliJ’s (actually Fernflower’s) decompilation of the class file with the URL example:

URL codefx = new URL("http://codefx.org");
URLConnection connection = codefx.openConnection();
BufferedReader reader = new BufferedReader(
    new InputStreamReader(connection.getInputStream()));

This is byte by byte the same result as if I had declared the types yourself. In fact, this feature only exists in the compiler and has no runtime component whatsoever, which also means there is no performance impact. So relax, this is not Javascript and nobody’s going to turn 0 into god.

If you’re still worried that lacking explicit types makes all code worse, I have a question for you: Have you ever written a lambda without defining its argument types?

rhetoricalQuestion.answer(yes -> "see my point?");

Where To Use var (And Where Not To)

JEP 286’s title, “local-variable type inference”, kinda gives away where

var
can be used: for local variables. More precisely, for “local variable declarations with initializers”, so even the following won’t work:

// nope
var foo;
foo = "Foo";

It really has to be

var foo = "Foo"
. Even then it doesn’t cover all cases as
var
won’t work with so called “poly expressions”, like lambdas and method references, whose type the compiler determines in relation to an expected type:

// none of this works
var ints = {0, 1, 2};
var appendSpace = a -> a + " ";
var compareString = String::compareTo

The only other eligible spot besides local variables are

for
loops:

var numbers = List.of("a", "b", "c");
for (var nr : numbers)
    System.out.print(nr + " ");
for (var i = 0; i < numbers.size(); i++)
    System.out.print(numbers.get(i) + " ");

That means fields, method signatures, and

catch
clauses still require manual type declaration.

// nope
private var getFoo() {
    return "foo";
}

Avoiding “Action At A Distance” Errors

That

var
can only be used locally is not a technical limitation, but a design decision. Sure, it would be nice to have it work like this:

// cross fingers that compiler infers List<User>
var users = new ArrayList<User>();
// but it doesn't, so this is a compile error:
users = new LinkedList<>();

The compiler could easily look at all assignments and infer the most concrete type that fits all of them, but it doesn’t. The JDK team wanted to avoid “action at a distance” errors, meaning changing code in some place should not lead to a seemingly unrelated error far away.

As an example look at the following:

// inferred as `int`
var id = 123;
if (id < 100) {
    // very long branch; unfortunately
    // not its own method call
} else {
    // oh boy, much more code...
}

So far, so… I don’t want to say “good”, but you know what I mean. I’m sure you’ve seen such code. Now we append this line:

id = "124"

What would happen? This is not a rhetorical question, think about it.

The answer is that the

if
-condition throws an error because
id
will no longer be an
int
and can thus not be compared with
<
. That error is at quite a distance from the change that caused it and on top of that it’s a surely unforeseen consequence of simply assigning a value to a variable.

From that perspective the decision to limit type inference to the immediate declaration makes sense.

Why Can’t Field And Method Types Be Inferred?

Fields and methods have a far larger scope than local variables and as such the distance between changes and errors increases considerably. In the worst case, changing a method parameter’s type can lead to binary incompatibilities and thus runtime errors. That’s a rather extreme consequence of having changed some implementation detail.

So because non-private fields and methods become part of a type’s contract and because that shouldn’t be changed accidentally, these types are not inferred. Sure, an exception could have been made for private fields or methods, but that would make the feature rather weirdly scoped.

The basic idea is that local variables are implementation details and can not be referenced from “far away” code, which reduces the need to strictly, explicitly, and verbosely define their type.

Focus on var

Background On var

Let’s have a look behind the scenes and find out why

var
was introduced, how its impact on readability was envisioned and why there is no
val
(or
let
) accompanying it. If you’re interested in even more detail have a look at the JEP 286 discussions, the
var
FAQ
, or the Project Amber mailing list.

But why?!

Java’s overall tendency to be pretty verbose, particularly compared to younger languages, is one of the biggest pain points for developers and a common critique of the language by novice and experienced Java devs alike. Project Amber, under which

var
was developed, aims “to explore and incubate smaller, productivity-oriented Java language features” and has the goal to generally reduce the ceremony involved in writing and reading Java code.

Local-variable type inference is aligned with that goal. On the writing side, it obviously makes declaring variables much easier, although I’d guess that a good half of my declarations are generated by the IDE, either during a refactoring or because it’s just plain faster to write a constructor or method call and then create a variable for it.

Beyond making declarations easier it also makes them more amenable. What do I mean by that? Declarations can be quite ugly, particularly if enterprise-grade class names and generics are involved.

InternationalCustomerOrderProcessor<AnonymousCustomer, SimpleOrder<Book>> orderProcessor = createInternationalOrderProcessor(customer, order);

There’s a long-ass type name, which pushes the variable name most of the way to the end of the line and leaves you with either bumping line length to 150 chars or initializing the variable on a new line. Both options suck if you’re aiming for readability.

var orderProcessor = createInternationalOrderProcessor(customer, order);

With

var
it is much less burdensome and easier on the eye to declare intermediate variables and we might end up doing that in places where we didn’t before. Think about nested or chained expressions where you decided against breaking them apart because the reduction in complexity got eaten by the increase in ceremony. Judicious use of
var
can make intermediate results more obvious and more easily accessible.

In short,

var
is about reducing verbosity and ceremony, not about saving characters.

And What About Readability?

Now let’s turn to readability. Surely it must get worse when types are missing, right? Generally speaking, yes. When trying to figure out how a piece of code works, types are an essential ingredient and even if IDEs would develop features that allow displaying all inferred types, it would still be more indirect than having them always present in the source.

So

var
starts at a readability disadvantage and has to make up for it. One way it does that is by aligning variable names:

// with explicit types
No no = new No();
AmountIncrease<BigDecimal> more = new BigDecimalAmountIncrease();
HorizontalConnection<LinePosition, LinePosition> jumping =
    new HorizontalLinePositionConnection();
Variable variable = new Constant(5);
List<String> names = List.of("Max", "Maria");

// with inferred types
var no = new No();
var more = new BigDecimalAmountIncrease();
var jumping = new HorizontalLinePositionConnection();
var variable = new Constant(5);
var names = List.of("Max", "Maria");

Type names are important, but variable names can be better. Types describe a general concept in the context of the entire Java ecosystem (for JDK classes), a general use case (library or framework), or a business domain (application) and thus will always have general names. Variables on the other hand are defined in a specific and very small context, in which their name can be very precise.

With

var
, variable names become front and center and stand out in a way they didn’t before, particularly if code highlighters mark the keyword and thus make it easier to instinctively ignore it. For a while I spent an hour or two per day reading Kotlin and I immediately got used to this. It can considerably improve readability.

As pointed out above, the other improvement to readability can come from having more intermediate variables declared because it comes at a cheaper cost when writing and reading.

Finding A Style

It is of course easy to go overboard with

var
and have code with shitty variable names and no visible types. It is up to us, the community in the large and each team in the small, to come up with a style that suits our needs and strikes a balance between verbosity and clarity.

Brian Goetz, Java language architect at Oracle and in charge of Project Amber, gave a first heuristic:

Use the

var
construct when it makes the code clearer and more concise and you’re not loosing essential information.

In that line I hope IDEs will not generally warn if a type declaration can be replaced with

var
. This is not an all-the-way construct like lambdas.

Why Is There No val/let?

Many languages that have

var
also offer an additional keyword for immutable variables. It’s usually called
val
, sometimes
let
, but Java 10 has neither and we have to use
final var
instead. The rationale is:
  • while immutability is important, local variables are the least important place for it
  • since Java 8 we have the concept of effectively final, already pushing us into the direction of immutable local variables
  • where
    var
    was almost universally applauded (74% strongly, 12% mildly in favor) feedback on both
    var
    /
    val
    and
    var
    /
    let
    was very mixed

I agree with the first two points and have to accept the latter, but I still find the result a bit disappointing. Having

val
or
let
next to
var
would ease the tension between those developers putting
final
on everything and those appalled by the verbosity.

Well, maybe in the future… until then we have to use

final var
.

Reflection

When declaring local variables you can use

var
instead of a class or interface name to tell the compiler to infer the type. This only works if the variable is immediately initialized, for example as in
var s = ""
. Indexes in
for
loops can also be declared with
var
. The type inferred by the compiler is put into the bytecode, so nothing changes at runtime – Java is still a statically typed language.

Beyond local variables, for example in fields and method signatures,

var
can not be applied. This was done to avoid “action at a distance” errors and to keep an inferred variable’s use site close to its declaration site, thus easing readability concerns.

While

var
can be used to make code worse, this is a chance for Java developers to write more readable code by finding a new balance between the noise of declarations and the complexity of nested/chained expressions.

The post First Contact With ‘var’ In Java 10 appeared first on blog@CodeFX.

Code First Java 9 Tutorial

$
0
0

So, Java 9 came out last year… What now? Where to get started? If that’s what you’re asking yourself, then you’ve come to the right place! This Java 9 tutorial is a condensation of all you need to know to find your way around the new release, to get you ready to explore it in more depth. Most topics begin with a block of code, so you can see right away how it works.

We start with setup (including tool support and migration challenges) before coming to Java 9’s upsides: language changes (e.g. private interface methods), new and improved APIs (e.g. collection factory methods and improvements to streams and optionals), changes to the JVM (e.g. multi-release JARs), and finally the new release’s flagship feature, the module system. There will be plenty of links for you to explore these topics further.

You can find the examples shown here, and more, in this GitHub project.

Getting Started With Java 9

You can download JDK 9 from Oracle. Personally, I prefer to download ZIPs and just unpack them instead of using JDK 9 as my default JVM, but that might be a left-over from using the early-access build. Nowadays you could give it a try.

Tool Support

For the best integration into your favorite IDE you should use its most current version as Java 9 support is constantly improved. If the cutting edge isn’t for you, you should at least be on Intellij IDEA 2017.2 or Eclipse Oxygen.1a (before that version, Eclipse needed Java 9 support plugins – they are obsolete now).

Similarly, use a current version of your build tool. In the case of Maven this should at least be 3.5.0 of the application itself and 3.7.0 of the compiler plugin. For Gradle, use at least 4.2.1.

Six tips for running Maven on Java 9.

Migration Challenges

While modularization remains fully optional, migrating to Java 9, i.e. simply building and executing a project on the new release, may require a few changes. The entire JDK has been modularized and together with some other internal changes this causes migration challenges when compiling and running code on Java 9. They can usually be fixed in the short-term with the new command line options, so you can take your time to properly resolve them.

Here are the seven most common challenges you might encounter:

  • illegal access to internal APIs
  • dependencies on Java EE modules
  • split packages
  • casting to
    URLClassLoader
  • rummaging around in runtime images
  • boot class path
  • new version strings

⇝ Read my post on migration challenges to learn how to overcome them.

Language Changes

Java 8 revolutionized how we write code – Java 9 does not even get close. But it does improve a few details and people looking for clean and warning-free code will appreciate them.

Private Interface Methods

public interface InJava8 {

    default boolean evenSum(int... numbers) {
        return sum(numbers) % 2 == 0;
    }

    default boolean oddSum(int... numbers) {
        return sum(numbers) % 2 == 1;
    }

    // before Java 9, this had to be `default`
    // and hence public
    private int sum(int[] numbers) {
        return IntStream.of(numbers).sum();
    }

}

As you can see, private interface methods are just that, the possibility to add

private
methods to interfaces. They are exactly like other private methods:
  • can not be
    abstract
    , i.e. must contain a body
  • can not be overriden
  • can only be called in the same source file

Their only use case is to share code between default methods without requiring you to add another default method to the interface’s API.

Try With Effectively Final Resources

void doSomethingWith(Connection connection)
        throws Exception {
    // before Java 9, this had to be:
    // try (Connection c = connection)
    try(connection) {
        connection.doSomething();
    }
}

If

connection
is effectively final, you can write
try (connection)
instead of the laborious
try (Connection c = connection)
that you had to use before Java 9. Finally!

Diamond Operator

<T> Box<T> createBox(T content) {
    // before Java 9, we had to put `T` there
    return new Box<>(content) {
        // useless anonymous class
    };
}

The diamond operator can now be applied to anonymous classes. In some cases the compiler might derive a type that the Java type system can not express (didn’t know those existed; they are called non-denotable types), in which case you get a compile error (this was the reason why they were not allowed in the first place). Here’s an example:

Box<?> createCrazyBox(Object content) {
    List<?> innerList = Arrays.asList(content);
    // compile error
    return new Box<>(innerList) {
        // useless anonymous class
    };
}

Private Safe Varargs And Less Deprecation Warning

import java.io.LineNumberInputStream;

@Deprecated
public class DeprecatedImportsAndSafeVarargs<T> {

    LineNumberInputStream stream;

    @SafeVarargs
    private void compareToNext(T... args) {
        // [...]
    }

}

On Java 8, the

import
directive would cause a warning because
java.io.LineNumberInputStream
is deprecated and the
@SafeVarargs
annotations would cause a compile error because it was not applicable to non-final methods. From Java 9 on, imports no longer cause deprecation warnings and
@SafeVarargs
can be applied to private methods (final or not).

New And Improved APIs

The lack of cohesion of the new and improved APIs might make it seem that nothing much happened, but that’s far from the truth! Much work went into them – they just don’t have a well-marketable label like “Streams and Lambdas”.

Stream API

public Stream<LogMessage> fromWarningToError() {
    return messages.stream()
        .dropWhile(message -> message.lessThan(WARNING))
        // this actually excludes the error
        .takeWhile(message -> message.atLeast(ERROR));
}

The stream API saw good improvements, of which a single example can only show a little. The changes are:

  • Stream::ofNullable
    creates a stream of either zero or one element, depending on whether the parameter passed to the method was
    null
    or not.
  • Stream::iterate
    create a stream much like a
    for
    loop.
  • Stream::dropWhile
    takes a predicate and removes elements from the stream’s beginning until the predicate fails for the first time – from then on, the stream remains the same and no more elements are tested against the predicate (unlike
    filter
    would do).
  • Stream::takeWhile
    takes a predicate and returns elements from the stream’s beginning until the predicate fails for the first time – there the stream ends and no more elements are tested against the predicate (unlike
    filter
    would do).

More on stream improvements.

Optional API

public interface Search {

    Optional<Customer> inMemory(String id);
    Optional<Customer> onDisk(String id);
    Optional<Customer> remotely(String id);

    default void logLogin(String id, Logger logger) {
        inMemory(id)
            .or(() -> onDisk(id))
            .or(() -> remotely(id));
            .ifPresentOrElse(
                logger::customerLogin,
                () -> logger.unknownLogin(id));
    }

}

The

Optional
API was improved as well – too much for a single example. The changes are:

  • Optional::stream
    creates a stream of either zero or one element, depending on the optional is empty or not – great to replace
    .filter(Optional::isPresent).map(Optional::get)
    stream pipelines with
    .flatMap(Optional::stream)
    .
  • Optional::or
    takes a supplier of another
    Optional
    and when empty, returns the instance supplied by it; otherwise returns itself.
  • Optional::ifPresentOrElse
    extends
    Optional::isPresent
    to take an additional parameter, a
    Runnable
    , that is called if the
    Optional
    is empty.

More on

Optional
improvements.

Collection Factories

List<String> list = List.of("a", "b", "c");
Set<String> set = Set.of("a", "b", "c");
Map<String, Integer> mapImmediate = Map.of(
    "one", 1,
    "two", 2,
    "three", 3);
Map<String, Integer> mapEntries = Map.ofEntries(
    entry("one", 1),
    entry("two", 2),
    entry("three", 3));

The new collection factory methods

List::of
,
Set::of
,
Map::of
return collections that:
  • are immutable (unlike e.g.
    Array::asList
    , where elements can be replaced) but do not express that in the type system – calling e.g.
    List::add
    causes an
    UnsupportedOperationException
  • roundly reject
    null
    as elements/keys/values (unlike
    ArrayList
    ,
    HashSet
    , and
    HashMap
    , but like
    ConcurrentHashMap
    )
  • for
    Set
    and
    Map
    , randomize iteration order between JDK runs

Here’s a good introduction to collection factory methods.

Reactive Streams

I’m gonna break with the code first approach here, because for reactive streams there is too much code involved – have a look at the demo.

Reactive streams require three basic types:

  • Publisher
    produces items to consume and can be subscribed to.
  • Subscriber
    subscribes to publisher and offers methods
    onNext
    (for new items to consume),
    onError
    (to inform if publisher encountered an error),
    onComplete
    (if publisher is done).
  • Subscription
    is the connection between publisher and subscriber and can be used to
    request
    items or
    cancel
    the subscription

The programmatic flow is as follows:

  • Creation and subscription:
    • create
      Publisher pub
      and
      Subscriber sub
    • call
      pub.subscribe(sub)
    • pub
      creates
      Subscription script
      and calls
      sub.onSubscription(script)
    • sub
      stores
      script
  • Streaming:
    • sub
      calls
      script.request(10)
    • pub
      calls
      sub.onNext(element)
      (max 10x)
  • Canceling:
    • sub
      may call
      sub.OnError(err)
      or
      sub.onComplete()
    • sub
      may call
      script.cancel()

There are no reactive APIs in JDK 9. For now, it only contains these interfaces (in

java.util.concurrent.Flow
) to offer reactive libraries like RxJava that implement those interfaces a common integration point in the JDK. In the future, JDK APIs might make use of them themselves.

Here’s a good introduction to the flow API.

Stack-Walking

private static Class<?> getCallingClass() {
    return StackWalker
        .getInstance(RETAIN_CLASS_REFERENCE)
        .walk(frames -> frames
            .map(StackFrame::getDeclaringClass)
            .filter(declaringClass -> declaringClass != Utils.class)
            .findFirst()
            .orElseThrow(IllegalStateException::new);
}

The new stack-walking API makes it easier to walk the Java call stack and considerably improves performance of partial walks (e.g. when only to determine the immediate caller like logging frameworks do) and walks that require cheaper information (i.e. no source code information like line number).

The trick is to first get a

StackWalker
instance and then hand a
Function<Stream<StackFrame>, T>
(plus wild cards) to
walk
, so when the walker hands you a stream of frames, you do your thing and compute your
T
(in the case above finding the
Class
that called into
Utils
), which
walk
will then return.

Why doesn’t

walk
simply return a
Stream<StackFrame>
? Because the stream is lazy (that’s the whole point of the new API) and you could get weird results when evaluating it at some random future time. Hence
walk
forces you to evaluate the frames within its call.

Deep dive into stack-walking API.

OS Processes

public static void main(String[] args) throws Exception {
    // tree -i /home/nipa | grep pdf
    ProcessBuilder ls = new ProcessBuilder()
        .command("tree", "-i")
        .directory(Paths.get("/home/nipa").toFile());
    ProcessBuilder grepPdf = new ProcessBuilder()
        .command("grep", "pdf")
        .redirectOutput(Redirect.INHERIT);
    List<Process> lsThenGrep = ProcessBuilder
        // new in Java 9
        .startPipeline(asList(ls, grepPdf));

    System.out.println("Started processes...");

    CompletableFuture[] lsThenGrepFutures = lsThenGrep.stream()
        // onExit returns a CompletableFuture<Process>
        .map(Process::onExit)
        .map(processFuture -> processFuture.thenAccept(
            process -> System.out.println(
                "Process " + process.getPid() + " finished.")))
        .toArray(CompletableFuture[]::new);
    // wait until all processes are finished
    CompletableFuture
            .allOf(lsThenGrepFutures)
            .join();

    System.out.println("Processes done");
}

The process API got a few new methods to create process pipelines as well as new methods on

Process

  • boolean supportsNormalTermination()
  • long pid()
  • CompletableFuture<Process> onExit()
  • Stream<ProcessHandle> children()
  • Stream<ProcessHandle> descendants()
  • ProcessHandle toHandle()

… and a new type

ProcessHandle
with some interesting static factory methods:

  • Stream<ProcessHandle> allProcesses()
  • Optional<ProcessHandle> of(long pid)
  • ProcessHandle current()

Looks like all you need to build a simple task manager with Java. 😊

Version API

Version version = Runtime.version();
System.out.println(
    version.major()
        + "." + version.minor()
        + "." + version.security());

Java 9 changed the version scheme (and Java 10 changes it again), which made all that prodding of system properties and parsing their values all the more error-prone. Java 9 finally resolves that with

Runtime.Version
, which gives you safe access to Java 9’s (and 10+’s) version information with methods like

major
and
minor
(which have been renamed for Java 10+ to
feature
and
interim
).

Further Changed APIs

JVM Changes

Not only the language and API was improved, though. The JVM got some new features as well. Naturally, its a little tougher to show them with code first, but I’ll do my best.

Multi-Release JARs

Say you have a class

Main
and another one
Version
.
Version
is special because you need it to run different code on Java 8 and 9. With multi-release JARs you can do that as follows:
  • write
    Main
    for Java 8 and compile it into the folder
    classes-8
  • create two implementations of
    Version
    with the same fully-qualified name and the same public API; one targets Java 8, the other Java 9
  • compile them into two different folders
    classes-8
    and
    classes-9

With Java 9’s

jar
you can do this:

jar
    --create --file mr.jar
    -C classes-8 .
    --release 9 -C classes-9 .

Without the last line in that command, it’s the typical way to package a bunch of classes into a JAR that would look like this:

└ org
    └ codefx ... (moar folders)
        ├ Main.class
        └ Version.class

With the last line the JAR looks like this, though:

└ org
    └ codefx ... (moar folders)
        ├ Main.class
        └ Version.class
└ META-INF
    └ versions
        └ 9
            └ org
                └ codefx ... (moar folders)
                    └ Version.class

JVMs before 8 ignore the

META-INF/versions
folder, but Java 9 will first look there when loading classes. That means running the JAR on Java 9 will execute a different
Version
class than when running on Java 8.

With multi-release JARs you can create artifacts that execute different code, depending on the JVM version they run on. This allows your library to use the best API on each JVM version, for example the throwable-creating (for stack information) and property-parsing (for version information) on Java 8 and earlier and the

StackWalking
and
Runtime.Version
APIs on Java 9.

⇝ Read my detailed guide to multi-release JARs.

Redirected Platform Logging

No code this time, because you’re unlikely to write any. This is the job of your favorite logging framework’s maintainers, so they can get their project ready to be the backend for all JDK log messages (not JVM logging). Because from Java 9 on, the JDK will send its log messages through a set of interfaces (

System.LoggerFinder
,
System.Logger
) for which logging frameworks can provide implementations.

This feature works well with multi-release JARs, which allows the framework to work fine on older Java versions, while benefiting from the additional functionality if run on Java 9.

Unified Logging

$ java -Xlog:gc*=debug -version

> [0.006s][info][gc,heap] Heap region size: 1M
> [0.006s][debug][gc,heap] Minimum heap 8388608  Initial heap 262144000
    Maximum heap 4192206848
# truncated about two dozen message
> [0.072s][info ][gc,heap,exit         ] Heap
# truncated a few messages showing final GC statistics

This time it’s about JVM logging. Thanks to a unified infrastructure (JEP 158, JEP 271), log messages from most (in the future, all) JVM subsystems can be configured with the same command line flag.

Internally, it works similarly to common logging frameworks, with messages getting a a level, a message, a time stamp, tags, etc. What’s a little unusual is the configuration with

-Xlog
.

In-depth guide to unified logging.

JVM Performance Improvements

As usual, the JVM got once again faster in Java 9. Here’s the list of the performance-related changes:

  • compact strings reduce average heap size by 10% to 15% (JEP 254)
  • improved (“indified”) string concatenation significantly reduces overhead when putting strings together (JEP 280)
  • Java 9 is aware of cgroup memory limits, which makes it play nicer with Docker et al (this was backported to Java 8)
  • something with interned strings and class data sharing (JEP 250)
  • contended locks reduce the performance overhead caused by internal bookkeeping (JEP 143)
  • security manager performance hit was reduced (JEP 232)
  • Java 2D rendering got better with the Marlin renderer (JEP 265)

⇝ There’s a great talk by Aleksey Shipilëv about the challenges and impact of implementing compact strings and indified string concatenation.

Further JVM changes

There are many more changes I can’t go into detail on. For something approaching completeness, I will list them instead.

  • new version strings (JEP 223)
  • GNU-style command line options (JEP 293)
  • command line flag validation (JEP 245)
  • reserved stack areas (JEP 270)

Module System

The JPMS posits that artifacts should be represented at runtime as modules

The Java Platform Module System (JPMS) is undoubtedly Java 9’s major feature. It posits that artifacts should no longer be plain JARs but JARs that describe a module, modular JARs, so to speak, and that they should be represented at runtime as modules.

A JAR is made modular by adding a module descriptor,

module-info.class
, which gets compiled from a module declaration,
module-info.java
:

module com.example.project {
    requires org.library;
    requires io.framework;
    exports com.example.project.pack;
}

As you can see a modules has a name, expresses dependencies, and defines some exports. The module system has many features, but its two cornerstones are:

  • making sure all required modules are presented when an application gets compiled or launched (called reliable configuration)
  • preventing access to all classes except the public ones in those exported packages (string encapsulation)

This allows compiler and runtime to fail faster when dependencies are missing or code does things it’s not supposed to and will make Java applications, particularly large ones, more stable. Other interesting features are more refined imports and exports (e.g. optional dependencies), services, or the possibility to create runtime images with

jlink
with exactly the modules your application needs.

By aligning the JVM’s conception (which sees all code on the class path as a big ball of mud) with ours (which usually sees trees of dependencies with artifacts that have names, dependencies, and APIs) an jarring conceptual dissonance is mended.

To process modules the module system introduces a concept paralleling the class path: the module path. It expects modular JARs and represents artifacts it finds as modules.

The class path won’t go anywhere, though, and remains a completely appropriate way to build and run projects. This and a few specific mechanisms (mostly unnamed module and automatic modules) allow the Java ecosystem to modularize almost independently from one another without forcing any project to either go modular or stay plain against its maintainers will.

For a thorough introduction to the module system:

Reflection

And that’s it. Phew… share:

twittergoogle_plusredditlinkedin

Subscribe:

twittergoogle_plusrssmail

The post Code First Java 9 Tutorial appeared first on blog@CodeFX.

Improve Launch Times On Java 10 With Application Class-Data Sharing

$
0
0

Surely the most prominent Java 10 feature is type inference with

var
, but there’s another gem hidden in 10 that’s worth exploring: application class-data sharing (AppCDS). It allows you to reduce launch times, response time outliers, and, if you run several JVMs on the same machine, memory footprint. Here’s how!

Overview

The Java 10 demo project has a script showing off application class-data sharing. AppCDS is an extension of the commercial class-data sharing (CDS) feature that Oracle’s JVM contains since Java 8. If you’re interested in a little more background, check out JEP 310, which generalized CDS to AppCDS.

Note that you need OpenJDK 10 to experiment with AppCDS because Oracle JDK only contains CDS and the option

-XX:+UseAppCDS
won’t work. Unfortunately, Oracle’s OpenJDK 10 builds do not contain JavaFX, so you will have a hard time applying AppCDS to a JavaFX application.

Application Class-Data Sharing In A Nutshell

To execute a class’ bytecode, the JVM needs to perform a couple of preparatory steps. Given a class name, it looks the class up in a JAR, loads it, verifies the bytecode, and pulls it into an internal data structure. That takes some time of course, which is most noticeable when the JVM launches and needs to load at least a few hundred, more likely a couple of thousand classes.

The thing is, as long as the application’s JARs do not change, this class-data is always the same. The JVM executes the same steps and comes to the same result every time it runs the app.

The idea behind AppCDS is to create a class-data archive once and then share it, so that the JVM need not recreate it

Enter application class-data sharing! The idea behind it is to create this data once, dump it into an archive, and then reuse that in future launches and even share it across simultaneously running JVM instances:

  1. create a list of classes to include in the archive (possibly with
    -XX:DumpLoadedClassList
    )
  2. create an archive with the option
    -Xshare:dump
  3. use the archive with the option
    -Xshare:on

When launching with

-Xshare:on
, the JVM maps the archive file into its own memory and thus has most classes it needs readily available and does not have to muck around with JARs. The memory region can even be shared between concurrently running JVM instances, which frees up memory that would otherwise be wasted on replicating the same information in each instance.

AppCDS significantly reduces the time the JVM has to spend on class loading

AppCDS significantly reduces the time the JVM has to spend on class loading, which is most noticeable during launch. It also prevents long response times in the case where a user is the first to access a feature that requires a lot of classes that weren’t loaded yet.

Working With A JDK Class-Data Archive

The simplest way to get started with class-data sharing is to limit it to JDK classes, so we’ll do that first. We will then observe that a simple “Hello, World” JAR can be launched in almost half the time.

Creating A JDK Class-Data Archive

The JRE already comes with a list of classes, which

-Xshare:dump
uses by default, so we can go straight to step 2 and generate the archive:

$ java -Xshare:dump

You might have to execute this command with admin rights because by default the JVM creates the archive file

classes.jsa
in
${JAVA_HOME}/lib/server
. On Linux, the resulting file is about 18 MB.

Using A JDK Class-Data Archive

Just as

-Xshare:dump
creates the archive in a default location,
-Xshare:on
reads from that same default, so using the archive is pretty simple:

$ java -Xshare:on -jar app.jar

We can use unified logging with

-Xlog
to observe CDS in action by analyzing the class loading log messages:

$ java
    -Xshare:on
    -Xlog:class+load:file=cds.log
    -jar app.jar

The file

cds.log
then contains messages like the following:

> [0.008s][info][class,load] java.lang.Object source: shared objects file
# [...]
> [0.042s][info][class,load] org.codefx.demo.java10.jvm.acds.HelloAppCDS source:
    file:/.../app.jar

As you can see,

Object
was loaded from the “shared objects file”, which is another term for the archive, whereas
HelloAppCDS
isn’t. A simple count of lines shows that for an app with a single class 585 classes are loaded from the archive and 3 aren’t.

There are always some classes whose data can not be archived

That implies that some JDK classes could not be loaded from the archive and that’s exactly right. There are some specific conditions under which it is not possible to archive a class’ data, so the JVM won’t embed it and will instead end up loading it at run time as usual.

Launch Time Measurements

A crude way to measure the performance improvement is to simply time the execution of the entire demo app while toggling

-Xshare
between
off
and
on
:

$ time java -Xshare:off -jar app.jar
> Hello, application class-data sharing!
>
> real    0m0.078s
> user    0m0.094s
> sys     0m0.012s

$ time java -Xshare:on -jar app.jar
> Hello, application class-data sharing!
>
> real    0m0.043s
> user    0m0.053s
> sys     0m0.014s

The interesting bit is

real
, which gives the wall-clock time it took to run the app. Without class-data sharing those numbers vary between 70 ms and 85 ms (need I add “on my machine”?), with sharing they land between 40 ms and 50 ms. The exact numbers don’t really matter, but you can see that there’s quite some potential.

Note, though, that this is the maximum performance gain you are going to get for a JDK archive. The more classes the application comes with, the lower becomes the share of JDK classes and hence the relative effect of loading them faster. To scale this effect to large applications, you need to include their classes in the archive, so let’s do that next.

Working With An Application Class-Data Archive

To actually share data for application classes as opposed to just JDK classes, we need to activate AppCDS by adding the command line switch

-XX:+UseAppCDS
(remember, public downloads of Oracle JDK 10 don’t include this feature). Creating and using an archive that includes application class-data follows the same logic as for JDK classes.

Creating A List Of Application Classes

This time we can not skip the step to come up with the list of classes to include in the archive. There are at least two ways to create that list: You can either do it manually or ask the JVM to do it for you. I’ll tell you about the latter and once we have a file in hand, you will see how you might have generated it by hand.

To have the JVM create the list, run the application with the

-XX:DumpLoadedClassList
option:

$ java
    -XX:+UseAppCDS
    -XX:DumpLoadedClassList=classes.lst
    -jar app.jar

The JVM will then dutifully record all loaded classes. If you want to include just the classes you need to launch, exit the app right after that. If, on the other hand, you want to include classes for specific features, you should make sure they are used at least once.

In the end you get a file

classes.lst
that contains the slash separated name of each class that was loaded. Here are the first few lines:

java/lang/Object
java/lang/String
java/io/Serializable
java/lang/Comparable
java/lang/CharSequence

As you can see, there’s nothing magical about this file and you could easily generate it by other means.

Creating An Application Class-Data Archive

The second step is to actually create the archive. We do that much like before but have to mention the list of classes with

-XX:SharedClassListFile
. And because this archive is application-specific, we don’t want it in
${JAVA_HOME}
, so we define the location with
-XX:SharedArchiveFile
:

$ java
    -XX:+UseAppCDS
    -Xshare:dump
    -XX:SharedClassListFile=classes.lst
    -XX:SharedArchiveFile=app-cds.jsa
    --class-path app.jar

Note that we do not launch the application with

-jar
. Instead we just define the class path with all the JARs the application needs. We will see in a minute why the path’s details are of the utmost importance and also why you can’t use wildcards like
lib/*
or exploded JARs like
target/classes
.

Depending on the size of the class list, this step might take a while. When it’s done, your archive, in this case

app-cds.jsa
, is ready to be used.

Using An Application Class-Data Archive

Using the is pretty straightforward. We need to unlock AppCDS, activate sharing and point to the archive:

$ java
    -XX:+UseAppCDS
    -Xshare:on
    -XX:SharedArchiveFile=app-cds.jsa
    -jar app.jar

If we analyze the class loading log messages as before, we see that with the application class

HelloAppCDS
could be loaded from the archive:

> [0.049s][info][class,load] org.codefx.demo.java10.jvm.acds.HelloAppCDS source:
    shared objects file

Heed The Class Path Content

What are the two biggest challenges in software development?

  1. naming
  2. cache invalidation
  3. off-by-one errors

The class-data archive is a cache

Cheap joke, I know, but relevant because the class-data archive is essentially a cache and so we need to ask ourselves under which circumstances it becomes stale and how that can be detected.

It is obviously a problem if a JAR was replaced and now contains classes that differ from the archive’s (say, someone drops an updated dependency into the app’s

lib
folder). And because the class path is always scanned linearly, even just reordering the same artifacts can change the app’s run-time behavior. In summary, the archive is no longer a correct representation of the class path, if the latter doesn’t contain the same artifacts in the same order as when the archive was created.

The class path used to launch the app must have the archive’s path as a prefix

To have a decent chance at detecting a class path mismatch, the JVM embeds the string that lists the class path content in the archive when creating it. When you launch the app with the archive, the JVM compares the actual class path with the embedded one and only launches if the latter is a prefix of the former.

That means the class path used to launch the app must first list all elements of the archive’s path in the same order, but can then append more JARs as it sees fit. This is of course by no means fool proof, but should detect a decent chunk of problematic situations.

The more specific the class path that is used to create the archive, the more reliably can it be used to “invalidate” the cache / archive. And that’s why you can use neither wild cards nor exploded JARs when creating the archive.

(If you look closely, you will notice that in my examples, I create the archive with

--class-path app.jar
but launch without class path because I use
-jar app.jar
. Wait, a non-empty class path can hardly be the prefix on an empty class path. It works nonetheless because the JAR specified with
-jar
is implicitly added to the class path.)

We’ve been talking an awful lot about the class path – what’s with the module path? Can you put code from JPMS modules into the archive? Unfortunately, not – from the JEP:

In this release, CDS cannot archive classes from user-defined modules (such as those specified in

--module-path
). We plan to add that support in a future release.

Launch Time Measurements

For a simple “Hello, World” application there is of course no performance boost of AppCDS over CDS because loading one class more or less from the archive has no measurable impact. So I took application class-data sharing for a ride with a large desktop application.

I focused on the launch, which loads about 25’100 classes and takes about 15 seconds. During that time, there’s a lot more going on that just fetching classes, but at least there are no network operations to skew the results.

The archive contained 24’212 classes (so about 900 could not be included) and has roughly 250 MB. Using it to run the application brought the launch time down by about 3 seconds (20 %), which I felt quite good about. Your mileage will vary, of course, so you have to do this yourself to know whether it’s worth the effort for your app.

Reflection

There are three steps to creating and using an archive with application class-data:

  1. Creating a list of classes to include in the archive:

$ java
    -XX:+UseAppCDS
    -XX:DumpLoadedClassList=classes.lst
    -jar app.jar

  1. Creating an archive:

$ java
    -XX:+UseAppCDS
    -Xshare:dump
    -XX:SharedClassListFile=classes.lst
    -XX:SharedArchiveFile=app-cds.jsa
    --class-path app.jar

  1. Using the archive:

$ java
    -XX:+UseAppCDS
    -Xshare:on
    -XX:SharedArchiveFile=app-cds.jsa
    -jar app.jar

Keep in mind that the class path used to launch the application must have the one used to create the archive as a prefix and that you can’t use wildcards or exploded JARs for the latter. Your launch time improvements depend a lot on your application, but anything between a couple and almost 50 % is possible. Finally, if you have any problems, use

-Xlog:class+load
to get more information.

The post Improve Launch Times On Java 10 With Application Class-Data Sharing appeared first on blog@CodeFX.

Unlocking Intersection Types With ‘var’ In Java 10

$
0
0

Whether you’re already using Java 10 or not, I’m sure you’ve heard all about

var
, Java’s new keyword reserved type name that allows you to declare local variables without having to specify their types. While

var
looks simple at first glance, it opens the door to a richer type system and can be turned into a surprisingly powerful tool with which you can do some really cool things. Today I want to look at how to use it to work with intersection types:

// `elements` should implement both
// `Closeable` and `Iterator<String>`
// (DOES NOT COMPILE!)
Closeable & Iterator<String> elements = // ...
// `try` only works because `elements` is `Closeable`
try (elements) {
    // the method `stream` expects an `Iterator<E>`,
    // (and returns a `Stream<E>`), so it can only be
    // called if `elements` implements it
    Optional<String> first = stream(elements)
        .filter(s -> s.startsWith("$"))
        .findAny();
}

The declaration of

elements
looks pretty weird: It wants the variable to be of two types,
Closeable
and
Iterator<String>
. That’s called an intersection type, but Java’s syntax doesn’t allow them and so the code doesn’t even compile – our goal is to find a variant that does. After some background on the why, what, and how of intersection types in Java, I will show you how to use them with
var
.

Spoiler: The declaration will end up as

var elements = // ...
.

Overview

To see the code snippets in action, check my Java 10 demo project.

What Are Intersection Types?

We’ve all been in this situation: You’re writing a method and at some point realize that some parameter’s type isn’t powerful enough – you need more! Maybe a

List
instead of a
Collection
or a
Megacorp
instead of a mere regular
Corporation
. Often that’s pretty straight forward: There’s a type which has what you need, so you can simply use it.

Other times it’s not so easy, though. As an example, say you need an

Iterator
to also be
Closeable
, so you can do this:

static <E> Optional<E> firstMatch(
        /* Closeable & Iterable<E> */ elements,
        Predicate<? super E> condition) {
    try (elements) {
        // `stream` turns `Iterator<E>` into `Stream<E>`
        return stream(elements)
            .filter(condition)
            .findAny();
    }
}

You want to iterate over

elements
, so it needs to be
Iterable
, but you also want the safety of a
try
-with-resources block, so it needs to be
Closeable
as well. If you think of the type
Iterable
as a circle that contains all variables of that type and likewise for
Closeable
,
elements
needs to be in the intersection of these circles – it needs to be of an intersection type of
Iterable
and
Closeable
.
Intersection types - here of Closeable and Iterable

Just Create A New Interface

The obvious way to limit

elements
to the desired types is to create a new interface:

public interface CloseableIterator<E> extends Closeable, Iterator<E> { }

static <E> Optional<E> firstMatch(
        CloseableIterator<E> elements,
        Predicate<? super E> condition)
        throws IOException {
    // ...
}

That’s great if you have control over all implementations of

Closeable
and
Iterator
and can change them to also implement
CloseableIterator
. But what if you don’t? Neither the JDK nor third-party libraries and frameworks know anything about your new interface and although they might return instances that implement both
Closeable
and
Iterator
, you still couldn’t pass them to
firstMatch
:

// `Scanner` implements `Iterator<String>` and `Closeable`
Scanner scanner = new new Scanner(System.in);
Optional<String> dollarWord =
    // compile error because `scanner` is no `CloseableIterator` :(
    firstMatch(iterator, s -> s.startsWith("$"));

No, what you really need is a way to express

Closeable & Iterable<E>
without creating a new interface.

Declaring Intersection Types With Generics

Generics allow intersection types in bounded type parameters

While you can’t declare a variable as

Closeable & Iterable<E>
, that expression can be syntactically valid in Java. Just not as a type in a variable declaration but as a bounded type parameter, for example in a generic method:

private static <E, T extends Closeable & Iterator<E>>
        Optional<E> firstMatch(T elements, Predicate<? super E> condition)
        throws IOException {
    // ...
    }

That’s not straight forward, but it works: First you declare a new type parameter

T
which has to extend
Closeable
and
Iterator
– here the ampersand is syntactically correct and limits
T
to the intersection of these two types – and then you declare
elements
as being of that type.

The next step is to get an actual instance of

Closeable & Iterator
. That’s pretty easy, every class that implements both interfaces, for example
Scanner
, works fine:

Scanner scanner = new Scanner(System.in);
Optional<String> dollarWord = firstMatch(
    scanner,
    s -> s.startsWith("$"));

But what if the instance is not so easy to create? What if you need to involve another method and its return type can not be specific, like

Scanner
but needs to be the general intersection? We can once again use generics for that:

@SuppressWarnings("unchecked")
private static <T extends Closeable & Iterator<String>>
        T createCloseableIterator(boolean empty) {
    if (empty)
        return (T) new Empty();
    else
        return (T) new Scanner(System.in);
}

You can combine

createCloseableIterator
and
firstMatch
as follows:

Optional<String> dollarWord = firstMatch(
    createCloseableIterator(empty),
    s -> s.startsWith("$"));

That’s actually pretty neat and all of that works without

var
, so where does local-variable type inference enter the picture? That’s next!

Declaring Variables Of Intersection Types

The construction we have so far has the weak spot that it breaks down under refactoring. Want to extract a variable for

createCloseableIterator(empty)
? That’s too bad because, without
var
, you can’t:

// does not compile
Closeable & Iterator<String> elements = createCloseableIterator(empty);
// compiles, but can not be passed to `firstMatch`
Closeable elements = createCloseableIterator(empty);
Iterator<String> elements = createCloseableIterator(empty);
// compiles and can be passed, but can fail at run time
// (depending on `empty`)
Scanner elements = createCloseableIterator(empty);
Empty elements = createCloseableIterator(empty);

What you really need is the first line, but Java’s syntax won’t let you compile it. That doesn’t mean the JVM can’t handle it, though. In fact, this would work:

static <T extends Closeable & Iterator<String>>
        void readAndPrint(boolean empty)
        throws IOException {
    T elements = createCloseableIterator(empty);
    Optional<String> dollarWord =
        firstMatch(elements, s -> s.startsWith("$"));
    System.out.println(dollarWord);
}

This goes too far, though. We’re now exposing a type

T
to callers of this method that they see neither as parameter nor return type and would clearly be confused by. And imagine what the method would look like if we needed another such variable!

No,

createCloseableIterator
and
firstMatch
have a right to a complicated signature because they actually need it, but
readAndPrint
is ridiculous! Is there no better way to declare
elements
?

After all this build-up, and given your knowledge of how

var
works, the solution is rather straight forward:

static void readAndPrint(boolean empty) throws IOException {
    var elements = createCloseableIterator(empty);
    Optional<String> dollarWord =
        firstMatch(elements, s -> s.startsWith("$"));
    System.out.println(dollarWord);
}

Use var to capture a generic method’s returned intersection type

There you go, with

var
you can declare
elements
in a way that lets the compiler deduce that its type is the intersection of
Closeable
and
Iterator
, just like you wanted. That means you can now freely declare such variables and use them across methods without having to expose your callers to crazy generic signatures.

Reflection

While the concept of intersection types is quite simple and a common theme in other (JVM) languages like Scala, Java’s lack of first-class support makes it complicated to work with. You have to resort to non-trivial constructions using generics and that always reduces the code’s readability and accessibility. Still, there are situations where they are the best available solution and their added complexity is acceptable.

With the introduction of

var
the situation improved considerably. Even if it’s still a long shot from proper support, being able to easily declare variables of an intersection type makes them much more usable. If you haven’t already, it is high time to add intersection types to your tool belt.

The post Unlocking Intersection Types With ‘var’ In Java 10 appeared first on blog@CodeFX.

Tricks with ‘var’ and anonymous classes (that you should never use at work)

$
0
0

Using

var
to infer types of local variables is a great tool for writing readable code. More than that, it makes working with intersection types much more pleasant. It has a dark underbelly, though, and while experimenting with it I found a few

var
-tricks related to anonymous classes (ad-hoc fields and ad-hoc methods) that were fun to explore but I think are ultimately not fit for duty.

These tricks hinge on the fact that compiler and JVM know a richer type system than can be expressed with Java’s syntax. With

var
, though, a type does not need to be written out and can instead be determined by the more powerful compiler. This is particularly helpful when working with anonymous classes, which can not be expressed in source code.

Overview

To see the code snippets in action, check my Java 10 demo project.

Ad-hoc Fields

Every time you call a constructor, you have the chance to add some fields and methods right then and there:

Megacorp megacorp = // ...
Map<Megacorp, Address> headquarters = // ...
Object corpWithHq = new Object() {
    Megacorp _corp = megacorp;
    Optional<Address> _hq =
        Optional.ofNullable(headquarters.get(megacorp));
};
// does not compile
System.out.println(corpWithHq._corp);

The compiler will create an anonymous subclass (in this case of

Object
) and then instantiate it. But since the subclass is only created during compilation there is no way to reference it in the source code and so
corpWithHq
is declared as an
Object
. The unfortunate consequence is that the fields
_corp
and
_hq
can’t be referenced because they aren’t part of
Object
’s API’.

If you use

var
, on the other hand, things work just fine:

Megacorp megacorp = // ...
Map<Megacorp, Address> headquarters = // ...
var corpWithHq = new Object() {
    Megacorp _corp = megacorp;
    Optional<Address> _hq =
        Optional.ofNullable(headquarters.get(megacorp));
};
// compiles
System.out.println(corpWithHq._corp);

With var, a variable can be of an anonymous type

Now,

corpWithHq
’s type is the anonymous subclass and you can happily toil away with the fields you added.

Enriching Streams

That specific example may not have been overly exciting but there are cases where this approach starts to look very appealing. I’m sure you’ve occasionally written a stream pipeline where you enriched the stream’s elements with some other piece of information, but needed to keep both kinds of elements around. They form a pair, but Java has no tuples, so you start looking for another solution. Maybe this?

List<Megacorp> megacorps = // ...
Map<Megacorp, Address> headquarters = // ...
Optional<Megacorp> firstWithValidHq = megacorps.stream()
    // we stream megacorps, but need to add addresses ...
    .map(megacorp -> new Object() {
        Megacorp _corp = megacorp;
        Optional<Address> _hq =
            Optional.ofNullable(headquarters.get(megacorp));
    })
    // ... only for evaluation, though ...
    .filter(o -> o._hq.isPresent())
    .filter(o -> isValid(o._hq.get()))
    // ... in the end we can get rid of them again
    .map(o -> o._copr)
    .findAny();

Interestingly enough, this example works without

var
and already compiles on Java 8 because the streams intermediate’s type,
Stream<$Anonymous>
, never needs to be expressed in source ode. With
var
, you’re able to declare intermediate variables, though, which wouldn’t work without it:

List<Megacorp> megacorps = // ...
Map<Megacorp, Address> headquarters = // ...
// Optional<$Anonymous>
var firstWithValidHq = megacorps.stream()
    .map(megacorp -> new Object() {
        Megacorp _corp = megacorp;
        Optional<Address> _hq =
            Optional.ofNullable(headquarters.get(megacorp));
    })
    .filter(o -> o._hq.isPresent())
    .filter(o -> isValid(o._hq.get()))
    // note that the map is gone!
    .findAny();

Without the second

map
,
firstWithValidHq
is an
Optional
containing the anonymous class with the two fields
_corp
and
_hq_
.

Evaluation

As I mentioned in the intro, I don’t think this is a trick you should use frequently, if at all. First, creating the anonymous class is pretty verbose and will often span several lines. Then it mixes two non-trivial features, anonymous classes and type inference, which makes the code harder to read and understand.

What bugs me the most, though, is that it falls apart under simple refactoring. Assume the example we’ve seen grows a bit and somebody wants to extract two methods, one that determines the megacorp with its headquarter and another that processes it:

List<Megacorp> megacorps = // ...
Map<Megacorp, Address> headquarters = // ...
// `determineCorp` must be declared to return a concrete
// type, so no amount of `var` magic is gonna help here
var corp = determineCorp(megacorps, headquarters);
processCorp(corp);

Using var with anonymous classes makes code harder to read, understand, and refactor

So while refactoring is the right idea, thanks to the ad-hoc fields it’s an order of magnitude more work because a type with the right fields needs to be created and used. That may very well deter developers from actually executing the refactoring and resistance to continuous improvement is not exactly a hallmark of maintainable code.

If you’re looking for alternatives, I sometimes use

Map.Entry
for pairs, which Java 9 made much more usable with the static method
Map::entry
. Beyond that you could be looking for a library that comes with tuples, something you usually find in functional libraries like Vavr. If you’re patient, you can wait for Project Amber’s data classes, which will make local classes a one-liner.

Ad-hoc Methods

Just like fields, you can add methods:

Megacorp corp = new Megacorp(/* ... */) {
    final BigDecimal SUCCESS_BOUNDARY = new BigDecimal("500000000");

    boolean isSuccessful() {
        return earnings().compareTo(SUCCESS_BOUNDARY) > 0;
    }

    boolean isEvil() {
        return true;
    }
};

And just like with fields, if you declare

corp
as
Megacorp
, the compiler will not let you use the new methods
isSuccessful
and
isEvil
. With
var
it does:

var corp = // like before
System.out.printf(
        "Corporation %s is %s and %s.\n",
        corp.name(),
        corp.isSuccessful() ? "successful" : "a failure",
        corp.isEvil() ? "evil" : "a failure"
);

It’s the same principle as with ad-hoc fields and I have the same criticism.

Evaluation

Like with ad-hoc fields, the code’s readability and refactorability (what a word) suffer without appreciable benefits.

Alternatively, methods like

isSuccessful
and
isEvil
could either be members of
Megacorp
or a subclass or, if that’s not possible or desirable for whatever reason, they can always be implemented as utility methods. While that makes calling them a little less natural (
Megacorps.isEvil(corp)
instead of
corp.isEvil()
), it has the added benefit to enable reuse.

Reflection

Using local-variable type inference with

var
, it is easy to add fields or methods to objects in an ad-hoc manner, simply by creating an instance of an anonymous class and assigning it to a local variable whose type is inferred. While that is a neat trick, it does not carry its own weight:

  • code becomes less readable (with anonymous classes and type inference it relies on non-trivial Java features)
  • code becomes harder to refactor (those types can not readily be used in method signatures)

So instead of relying on a little magic with a lot of downsides, I recommend to stick to proven alternatives:

  • instead of ad-hoc fields, use
    Map::entry
    , your favorite FP library’s tuple types, or wait for data classes
  • instead of ad-hoc methods, extend the class directly or use utility functions

But take my advice with a grain of salt, I don’t like anonymous classes anyways.

The post Tricks with ‘var’ and anonymous classes (that you should never use at work) appeared first on blog@CodeFX.

Unlocking Traits With ‘var’ In Java 10

$
0
0

Local-variable type inference, or more succinctly

var
, turns out to be surprisingly versatile. It unlocks intersection types, allows using ad-hoc fields and methods of anonymous types, and, as we will see now, enables working with traits. In the end, I don’t think it’s worth the hassle, but your mileage may vary – maybe it helps you out of a bind one day.

Overview

To see the code snippets in action, check my Java 10 demo project.

What’s A Trait?

A trait ad-hoc combines features of several types into one

Some programming languages have a concept of traits, which allows creating a new type in the middle of a variable declaration that combines features of several types into one. Conceptionally, this could look as follows:

type Megacorp {
    String name();
    BigDecimal earnings();
}

trait IsSuccessful {
    boolean isSuccessful() {
        return earnings().compareTo(SUCCESS_BOUNDARY) > 0;
    }
}

trait IsEvil {
    boolean isEvil() { return true; }
}

Megacorp & IsSuccessful & IsEvil corp =
    new (Megacorp & IsSuccessful & IsEvil)(/*...*/);
System.out.printf(
    "Corporation %s is %s and %s.\n",
    // relying on `corp` as `Megacorp`
    corp.name(),
    // relying on `corp` as `IsSuccessful`
    corp.isSuccessful() ? "successful" : "a failure",
    // relying on `corp` as `IsEvil`
    corp.isEvil() ? "evil" : "a failure"
);

Looks nice, doesn’t it? If only the syntax wouldn’t be totally made up… But, as we will see in a minute, we can take a few steps into that direction with some trickery involving lambdas, default methods, and

var
. Before we come to that, though, I want to address a question you might have at this point.

Wait, I Thought These Were Intersection Types!?

If this reminds you of intersection types, you’re on to something. The ampersand in the declaration has the same semantics: The resulting intersection inherits from the joined types and it follows that variables of that intersection type have the methods of all joined types – in this case of

Megacorp
,
IsSuccessful
, and
IsEvil
.

The difference between intersection types and traits is that the former represent a requirement: A variable, parameter, or return value has to be of the intersection of the joined types. Traits, on the other hand, are about composing types: The variable created by the construction will be of the intersection of the joined types.

Naturally, intersection types and traits go well together:

public void createWithTraits() {
    // create `corp` with two traits
    Megacorp & IsSuccessful & IsEvil corp =
        new (Megacorp & IsSuccessful & IsEvil)(/*...*/);
    // call method that requires intersection
    // of `Megacorp` and `IsEvil`
    requireIntersection(corp)
}

public void requireIntersection(Megacorp & IsEvil corp) {
    // relying on `corp` as `Megacorp` and `IsEvil`
    System.out.printf(
        "Corporation %s is %s.\n",
        corp.name(),
        corp.isEvil() ? "evil" : "a failure"
    );
}

Now, if only any of that were valid Java syntax… Let’s see how close we can get.

Creating The Skeleton For Traits

What the hell is going on here?

The basic recipe to creating traits has three ingredients:

  • prepare an interface that can be created with a lambda expression
  • cast that lambda to the intersection of the required types
  • assign the entire kerfuffle to a
    var
    -ed variable, so the type information don’t get trimmed to a single type

In the end, it will look like this:

// compiler infers desired intersection type for `corp`
var corp = (MegacorpDelegate & IsSuccessful & IsEvil) () -> megacorp;

I guess this leads us to the question: What the hell is going on here?

Let’s start with the initial type,

Megacorp
in this example: It needs to be an interface. To intersect it with other types, an extending interface like
MegacorpDelegate
needs to be created that default implements all methods by forwarding to a delegate:

interface Megacorp {
    String name();
    BigDecimal earnings();
}

@FunctionalInterface
interface MegacorpDelegate extends Megacorp {
    Megacorp delegate();
    default String name() { return delegate().name(); }
    default BigDecimal earnings() { return delegate().earnings(); }
}

The delegating interface must have a single abstract method, which returns the delegate

It is critical that the delegating interface’s only abstract method is one that returns the delegate. This enables you to create an instance of it from a given

Megacorp
with a lambda expression like
() -> megacorp
. Lambdas are crucial here because they are so-called poly expressions: They don’t have a defined type – instead it is inferred from the rest of the statement, in this case from the cast you place before them.

The delegating interface doesn’t add any domain functionality, it is only a technical requirement to make traits work in Java. As such, the delegating interface is conceptually close to the original one and I recommend placing it in the same package.

With that, some parts of the earlier declaration start to work:

Megacorp megacorp = // ...
// compiler infers `MegacorpDelegate` for `corp`
var corp = (MegacorpDelegate) () -> megacorp;

Creating such a delegating interface gives you the skeleton that you can now flesh out with the actual traits by writing interfaces that extend the original interface and add new methods.

var traits

Creating Traits

After creating a delegating interface like

MegacorpDelegate
everything is set up for coding the actual traits. They add specific features to the original interface, here
Megacorp
, that are apparently not general enough to become part it. That’s why traits are more likely to end up close to where they are needed instead of close to the original interface.

Trait interfaces must not have abstract methods

The second critical requirement is that the trait interfaces must not have abstract methods! If they do, their intersection with the delegating interface could not be created with a lambda.

Here are some example for our megacorporations:

interface IsSuccessful extends Megacorp {
    BigDecimal SUCCESS_BOUNDARY = new BigDecimal("500000000");

    default boolean isSuccessful() {
        return earnings().compareTo(SUCCESS_BOUNDARY) > 0;
    }

}

interface IsEvil extends Megacorp {

    default boolean isEvil() {
        return true;
    }

}

Because

IsSuccessful
and
IsEvil
have no abstract methods, their intersection with
MegacorpDelegate
can still be created with a lambda:

// compiles and runs
Megacorp megacorp = // ...
var corp = (MegacorpDelegate & IsSuccessful & IsEvil) () -> megacorp;
System.out.printf(
    "Corporation %s is %s and %s.\n",
    // relying on `Megacorp`
    corp.name(),
    // relying on `IsSuccessful`
    corp.isSuccessful() ? "successful" : "a failure",
    // relying on `IsEvil`
    corp.isEvil() ? "evil" : "a failure"
);

Et voilà, given a

Megacorp
instance you can decide on the spot which other functionality you want to compose with it. You can quickly and easily code that functionality up for your very specific context and start using it very naturally with plain method calls on the instance.

But the best thing is, neither the creators of

Megacorp
,
MegacorpDelegate
, or even
IsSuccessful
and
IsEvil
need to know what exactly you’re doing and neither do you need to create new types for any specific combination of traits. The latter is particularly important if you have many possible traits because the combinatorial explosion of creating a class for every combination would quickly overwhelm you.

The Dark Side

Ok, we’ve finally settled how to simulate traits in Java by using lambda expressions, default methods, intersection types, and local-variable type inference. This is a nice thought experiment and might even serve you as a stopgap one day, but it has severe disadvantages, which is why I don’t recommend to use it except in critical emergencies.

The first thing you might have noticed throughout this post is that simulating traits requires the combination of a decent number of non-trivial Java features, which always makes code less readable and thus less maintainable. Speaking of maintainability, variables of an intersection type only flourish within a method – as soon as they need to be passed to or from a method they require some generic trickery to be represented. This makes refactoring such code considerably more complicated.

There’s also some setup involved in creating the delegating interface and, even though that is unlikely to play a relevant role, the memory indirection it incurs reduces performance. Beyond that, traits need to be interfaces without abstract methods, which limits the functionality you can implement.

The final death knell is something else, though: Default methods can not implement

Object
methods, which has the consequence that the combined instances can never forward a call to

equals
or
hashCode
to the underlying delegate. This is bound to break code that puts such instances in collections and will result in strange and hard-to-debug misbehavior.

So as much fun as playing with traits is, I strongly recommend never to simulate them this way. Alternatives are to simply gather the required methods in utility classes or to create subclasses after all.

Reflection

These are the ingredients to ad-hoc compose traits with a given interface:

  • a delegating interface extending the given interface:
    • uses default methods to forward all calls to delegate
    • has single abstract method
      delegate()
      , so it can be created with
      () -> delegate
  • trait interfaces that have no abstract methods
  • declare variables as follows:

var variable = (Interface & Trait1 & Trait2) () -> delegate;

This looks like a great way to avoid deep inheritance trees and instead compose the exact behavior you need exactly where you need. Alas, it has several downsides:

  • cumbersome creation of the delegating interface
  • uses several non-trivial language features
  • use of intersection types makes refactoring harder
  • can not correctly implement
    equals
    /
    hashCode

The post Unlocking Traits With ‘var’ In Java 10 appeared first on blog@CodeFX.


All You Need To Know For Migrating To Java 11

$
0
0

Java 11 is released today! Formally it marks the end of a monumental shift in the Java ecosystem. With the challenges of migrating from Java 8 onto a modular and flexible JDK, with the six-month release cycle, the new licensing and long-term support models, we’ve entered a new era! Now the code bases have to follow and many projects will move from Java 8 directly to Java 11. If that describes yours, you’ve come to the right place – this migration guide tells you everything you need to know when moving from Java 8 to Java 11.

We’ll start with a super-quick tour through the new release cadence, licensing, and support before discussing how to prepare a migration (TL;DR: update all the things!) and finally, how to overcome the four most common hurdles (if you already migrated to Java 9, you can skip most of that). Note that we’re talking migration, not modularization (that’s not required and should be a separate step), so we won’t be creating any modules.

On Releases, JDKs, And Licenses

This may seem like boring stuff, but the six-month release cycle, the commercialization of Oracle’s JDK, and the open question of long-term support for OpenJDK probably has more impact on your project than the technical challenges of moving to Java 11. So let’s discuss this, but be quick about it – more details in the links.

Java Next (talk at JavaZone 2018)

New Release Cadence

This is the most well-known change, so I’ll keep it short:

  • new major release every six months (March and September)
  • two minor updates for each (one and four months later)

Radical new plans for Java – CodeFX Weekly #34

OpenJDK Is The New Default

In the dark times (before September 2018), Oracle’s JDK (and before that Sun’s JDK) was richer in features, more performant, and (perceived to be) more stable – it was hence the default choice for most of the ecosystem. OpenJDK was like the ugly duckling that most developers tried to avoid. But with Java 11 it has blossomed into a swan!

Oracle JDK is fully commercial

Oracle worked hard to make Oracle JDK 11 and OpenJDK 11 almost identical from a technical point of view – so much so that the most important difference is the license file they ship with. Oracle further pushes developers towards OpenJDK by making their branded JDK commercial, meaning you can’t use it in production without paying Oracle from day one after its release (you can use it for development and testing).

As a consequence, OpenJDK will become the new default – with full feature set, prime performance, and a free license (GPL+CE) it’s a great choice. On the next rung come, side by side, Oracle and other vendors with their OpenJDK variants for which they sell long-term support.

Time to look beyond Oracle’s JDK
Oracle JDK Releases for Java 11 and Later

Long-Term Support

Oracle ships OpenJDK builds at jdk.java.net and, as mentioned, publishes two updates for each major version. So what happens after six months if you want to stay on a specific major version while still receiving updates with security and bug fixes? Two choices:

  • pay someone for commercial support
  • hope for free support of OpenJDK

As to commercial support, there are various vendors that have you covered for the specific versions they mark as long-term support (the focus seems to be on 11/17/23/…):

There will likely be free LTS for OpenJDK

Regarding OpenJDK, there are very promising discussions on the mailing list that suggest that there will be at least four years of public updates to the same versions. Most likely, each LTS version will get a steward that manages the updates and it looks like it may be Red Hat for Java 11. That covers the sources, but where can we get the final binaries from? AdoptOpenJDK is gearing up to continuously build the various OpenJDK versions for all kinds of platforms.

Put together, we’d get free OpenJDK LTS, organized by companies that are well-known in the Java community and continuously built by AdoptOpenJDK. That would be awesome!

No Free Java LTS Version? – CodeFX Weekly #56
Java is still available at zero-cost
Java is still free

Preparing Your Migration

Here you are: With your favorite vendor’s JDK installed (and maybe some LTS in the back pocket), you want your Java 8 project to work on Java 11. I know you’re ready to roll but before we go in, we need to discuss how to best approach and prepare the migration.

Short Or Long-Lived?

When starting to migrate from Java 8 to Java 11 (or later), the first thing you have to answer is whether you can and want to do this in one fell swoop or over a longer period of time. If the project causes little trouble and you are raising your minimum requirements, then go for a quick migration where you use the new version for the entire build process, including the target for compilation. All that’s left is to fix any problems that may pop up.

Don’t create a long-lived migration branch

If you don’t want to raise the minimum version requirement or the project is too large to migrate in a single sitting or a short-lived branch, I recommend the following approach:

  • Don’t create a long-lived branch for the migration – instead do it on the regular development branch. This way, you’re not facing merge conflicts and can be sure that your colleagues’ changes are always also built on Java 11.
  • Configure your continuous integration server to run the build once in its common configuration and once on Java 11.
  • Learn about your build tool’s support for configuration specific to individual Java versions. (In Maven, that would be profiles.) This way you can keep the old build running while adding required configuration for the new version.
  • Try to keep the version-specific configuration to a minimum. For example, prefer updating dependencies over adding command line flags (more on that below).

This way you can take all the time you need to guarantee that your project works on Java 8 as well as on Java 11 (or later). Whether you want to keep building on both (or more) versions or flip the switch once you’re done depends on the project’s minimum Java requirement. When you’re eventually leaving Java 8 behind for good, don’t forget to merge the version-specific bits into the default configuration to reduce complexity.

Planning Your Java 9 Update (fully applies to Java 11)
Maven on Java 9 – Six Things You Need To Know (fully applies to Java 11)

Update All The Things

The first rule of moving to Java 11 is you do not talk … to update all the things. Your IDE, your build tool, its plugins, and, most importantly, your dependencies. You don’t have to do all of these updates in advance, but if you can, you absolutely should – it will very likely get you past some hurdles you can then stay blissfully unaware of.

Here are the recommended minimum versions for a few tools:

  • IntelliJ IDEA: 2018.2
  • Eclipse: Photon 4.9RC2 with Java 11 plugin
  • Maven: 3.5.0
    • compiler plugin: 3.8.0
    • surefire and failsafe: 2.22.0
  • Gradle: blocked by #5120

Some dependencies that you should keep an eye on (and versions that are known to work on Java 11):

  • Anything that operates on bytecode like ASM (7.0), Byte Buddy (1.9.0), cglib (3.2.8), or Javassist (3.23.1-GA). Since Java 9, the bytecode level is increased every six months, so you will have to update libraries like these pretty regularly.
  • Anything that uses something that operates on bytecode like Spring (5.1), Hibernate (unknown), Mockito (2.20.0), and many, many other projects.

The second bullet is not very helpful in its generality, but it’s the unfortunate truth: Many powerful projects work with bytecode under the hood. It helps to develop an eye for identifying problems related to that. Some (obvious?) tips:

  • stack traces ending in bytecode manipulation libraries
  • errors or warnings complaining about the bytecode level
  • errors or warnings mumbling about "unknown (constant pool) entries"

If you don’t update in advance, it should still be the first action you take when encountering a problem with any specific tool or dependency. Either way, you may occasionally encounter problems even though your dependencies are up to date. In that case, have a look at the precise artifact causing the problem – chances are it’s a transitive dependency, in which case you should look into updating it separately.

With an older version of Hibernate, for example, it was necessary to update Javassist to work on Java 11:

<dependency>
    <groupId>org.hibernate</groupId>
    <artifactId>hibernate-core</artifactId>
    <!-- LOOK OUT: YOU SHOULD USE A NEWER VERSION! -->
    <version>5.2.12.Final</version>
</dependency>
<dependency>
    <!-- update Hibernate dependency on
            Javassist from 3.20.0 to 3.23.1
            for Java 11 compatibility -->
    <groupId>org.javassist</groupId>
    <artifactId>javassist</artifactId>
    <version>3.23.1-GA</version>
</dependency>

Likewise, with the outdated version 3.7.0 of the Maven compiler plugin, its ASM dependency needed updating:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-compiler-plugin</artifactId>
    <!-- LOOK OUT: YOU SHOULD USE 3.8.0! -->
    <version>3.7.0</version>
    <configuration>
        <release>${java.version}</release>
    </configuration>
    <dependencies>
        <dependency>
            <!-- update compiler plugin dependency on
                    ASM for Java 11 compatibility -->
            <groupId>org.ow2.asm</groupId>
            <artifactId>asm</artifactId>
            <version>6.2</version>
        </dependency>
    </dependencies>
</plugin>

Only dive into the problem if updating is impossible or doesn’t help

Unfortunately, not all projects are well-maintained or were even discontinued, in which case you need to look for alternatives. Examples are FindBugs (use SpotBugs instead), Log4j 1 (use Log4J 2), and Cobertura (use JaCoCo).

Only if the problem lies in your own code or such updates/replacements don’t help or aren’t possible, does it make sense to dive into the actual problem.

A Word On The Module System

I’m sure you’ve heard about the Java Platform Module System (JPMS) that was introduced in Java 9. Since it’s causing most of the compatibility challenges you’re going to face during a migration from Java 8, it definitely helps a lot to understand its basics – for example, by reading this fine module system tutorial (cough) or my book (cough cough). But keep in mind that you are not required to create modules to have your code run on Java 9 or later!

You don’t need modules to run on Java 9+

The class path is here to stay and if your code or its dependencies don’t do anything forbidden (more on that later), you can expect it to Just Work™ on Java 9, 10, or 11 exactly as it did on 8 – modules are no requirement and so this post does not address modularization, just migration.

Migrating From Java 8 To Java 11

We’re done preparing, time to go! Let’s see which problems you may expect on Java 11 and how to fix them.

Removal Of Java EE Modules

There used to be a lot of code in Java SE that was actually related to Java EE. It ended up in six modules that were deprecated for removal in Java 9 and removed from Java 11. Here are the removed technologies and packages:

  • the JavaBeans Activation Framework (JAF) in
    javax.activation
  • CORBA in the packages
    javax.activity
    ,
    javax.rmi
    ,
    javax.rmi.CORBA
    , and
    org.omg.*
  • the Java Transaction API (JTA) in the package
    javax.transaction
  • JAXB in the packages
    javax.xml.bind.*
  • JAX-WS in the packages
    javax.jws
    ,
    javax.jws.soap
    ,
    javax.xml.soap
    , and
    javax.xml.ws.*
  • Commons Annotation in the package
    javax.annotation

Symptoms

Here’s a compile error for a class using

JAXBException
from the java.xml.bind module:

error: package javax.xml.bind does not exist
import javax.xml.bind.JAXBException;
                     ^

If you get it past the compiler but forget to massage the run time, you’ll get a

NoClassDefFoundError
:

Exception in thread "main" java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException
    at monitor.Main.main(Main.java:27)
Caused by: java.lang.ClassNotFoundException: javax.xml.bind.JAXBException
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:185)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:496)
    ... 1 more

Fixes

Add third-party dependencies that contain the classes you need. The easiest way to do that is to stick to the reference implementations (given as Maven coordinates without version – use the most current ones):

For more details, sources, and other recommendations, see this StackOverflow answer.

Illegal Access To Internal APIs

One of the module system’s biggest selling points is strong encapsulation. It makes sure non-public classes as well as classes from non-exported packages are inaccessible from outside the module. First and foremost, this of course applies to the platform modules shipped with the JDK, where only

java.*
and
javax.*
packages are fully supported. Most
com.sun.*
and
sun.*
packages, on the other hand, are internal and hence inaccessible by default.

While the Java 11 compiler behaves exactly as you would expect and prevents illegal access, the same is not true for the run time. To offer a modicum of backwards compatibility it eases migration and improves the chances of applications built on Java 8 to run on Java 11 by granting access to internal classes. If reflection is used for the access, a warning is emitted.

Symptoms

During compilation against Java 11 you may see compile errors similar to the following:

error: package com.sun.imageio.plugins.jpeg is not visible
import com.sun.imageio.plugins.jpeg.JPEG;
                              ^
  (package com.sun.imageio.plugins.jpeg is declared
  in module java.desktop, which does not export it)

Warnings emitted for reflection look as follows:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by j9ms.internal.JPEG
    (file:...) to field com.sun.imageio.plugins.jpeg.JPEG.TEM
WARNING: Please consider reporting this
    to the maintainers of j9ms.internal.JPEG
WARNING: Use --illegal-access=warn to enable warnings
    of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
# here's the reflective access to the static field com.sun.imageio.plugins.jpeg.JPEG.TEM

Fixes

The most obvious and sustainable fix for dependencies on internal APIs is to get rid of them. Replace them with maintained APIs and you paid back some high-risk technical debt.

If that can’t be done for whatever reason, the next best thing is to acknowledge the dependencies and inform the module system that you need to access it. To that end you can use two command line options:

  • The option
    --add-exports module/package=$readingmodule
    exports
    $package
    of $module to $readingmodule. Code in $readingmodule can hence access all public types in
    $package
    but other modules can not. When setting $readingmodule to
    ALL-UNNAMED
    , all code from the class path can access that package. During a migration to Java 11, you will always use that placeholder (you will have to change it when you modularize). The option is available for the
    java
    and
    javac
    commands.
  • This covers access to public members of public types but reflection can do more than that: With the generous use of
    setAccessible(true)
    it allows interaction with non-public classes, fields, constructors, and methods (sometimes called deep reflection), which even in exported packages are still encapsulated. The
    java
    option
    --add-opens
    uses the same syntax as
    --add-exports
    and opens the package to deep reflection, meaning all of its types and their members are accessible regardless of their visibility modifiers.

You obviously need

--add-exports
to appease the compiler but using
--add-exports
and
--add-opens
for the run time has advantages as well:
  1. the run time’s permissive behavior will change in future Java releases, so you have to do that work at some point anyway
  2. --add-opens
    makes the warnings for illegal reflective access go away
  3. as I will show in a minute, you can make sure no new dependencies crop up by making the run time actually enforce strong encapsulation

Five Command Line Options To Hack The Java Module System

Going Further

Compiling against Java 11 helps hunting down dependencies on internal APIs in the project’s code base. But the libraries and frameworks your project uses are just as likely to make trouble.

JDeps is the perfect tool to find compile dependencies on JDK-internal APIs in your project and your dependencies. If you’re not familiar with it, I’ve written a tutorial that gets you started. Here’s how to use it for the task at hand:

jdeps --jdk-internals -R --class-path 'libs/*' $project

Here,

libs
is a folder containing all of your dependencies and
$project
your project’s JAR. Analyzing the output is beyond this article’s scope but it’s not that hard – you’ll manage.

Finding reflective access is a little tougher. The run time’s default behavior is to warn you once for the first illegal access to a package, which is insufficient. Fortunately, there’s the

--illegal-access=$value
option, where
$value
can be:
  • permit
    : Access to all JDK-internal APIs is permitted to code on the class path. For reflective access, a single warning is issued for the first access to each package. (Default in Java 9, but will be removed in a future release.)
  • warn
    : Behaves like
    permit
    but a warning is issued for each reflective access.
  • debug
    : Behaves like
    warn
    but a stack trace is included in each warning.
  • deny
    : The option for those who believe in strong encapsulation: All illegal access is forbidden by default.

Particularly

deny
is very helpful to hunt down reflective access. It is also a great default value to set once you’ve collected all required
--add-exports
and
--add-opens
options. This way, no new dependencies can crop up without you noticing it.

Removal Of Deprecated APIs and JavaFX

Since Java 9, the

@Deprecated
annotation got a Boolean attribute:
forRemoval
. If
true
, the deprecated element is going to be removed as soon as the next major release. That’s mildly shocking – in the past
@Deprecated
just meant yellow squiggly lines.

Removed Classes and Methods

Here are some of the more common classes and methods that were removed between Java 8 and 11:

  • sun.misc.Base64
    (use
    java.util.Base64
    )
  • com.sun.java.swing.plaf.nimbus.NimbusLookAndFeel

    (use
    javax.swing.plaf.nimbus.NimbusLookAndFeel
    )
  • on
    java.util.LogManager
    ,
    java.util.jar.Pack200.Packer
    /
    Unpacker
    :
    methods
    addPropertyChangeListener
    and
    removePropertyChangeListener
  • on
    java.lang.Runtime
    : methods
    getLocalizedInputStream
    and
    getLocalizedOutputStream
  • various methods on
    SecurityManager

JDK 9 Release Notes – Removed APIs, Features, and Options
JDK 10 Release Notes – Removed Features and Options

Heeding Deprecation Warnings

To make it easier to keep up with deprecation warnings, I recommend using the command line tools

jdeps
and
jdeprscan
. They work on class files and JARs and you can find them in your JDK’s
bin
folder. The former is a multi-purpose dependency analysis tool while the latter focuses on reporting the use of deprecated APIs, highlighting those that will be removed.

A JDeps Tutorial – Analyze Your Project’s Dependencies
Java Platform, Standard Edition Tools Reference – jdeprscan

JavaFX

Then there’s JavaFX. It was never part of Java SE (i.e. The Standard™) and few OpenJDK variants shipped with it. For a while, Oracle seemed to push JavaFX and so they included it in their JDK, but that dwindled out and with Oracle aligning its JDK with OpenJDK, they no longer ship JavaFX. In fact, from Java 11 on, you will have a hard time finding any JDK that ships with JavaFX.

Don’t worry, though, the future is bright. OpenJFX, the project behind JavaFX, pulled the entire UI framework into their own artifacts that you simply add as a regular dependency. You can download them from Gluon or even Maven Central.

Casting To URL Class Loader

Java 9 and the module system improved the platform’s class loading strategy, which is implemented in a new type and in Java 11 the application class loader is of that type. That means it is not a

URLClassLoader
, anymore, so the occasional
(URLClassLoader) getClass().getClassLoader()
or
(URLClassLoader) ClassLoader.getSystemClassLoader()
sequences will no longer execute. This is another typical example where Java 11 is backwards compatible in the strict sense (because that it’s a
URLCassLoader
was never specified) but which can nonetheless cause migration challenges.

Symptoms

This one is very obvious. You’ll get a

ClassCastException
complaining that the new
AppClassLoader
is no
URLClassLoader
:

Exception in thread "main" java.lang.ClassCastException:
    java.base/jdk.internal.loader.ClassLoaders$AppClassLoader
    cannot be cast to java.base/java.net.URLClassLoader
        at monitor.Main.logClassPathContent(Main.java:46)
        at monitor.Main.main(Main.java:28)

Fixes

The class loader was probably cast to access methods specific to

URLClassLoader
. If so, you might have to face some serious changes.

If you want to access the class path content, check the system property

java.class.path
and parse it:

String pathSeparator = System
    .getProperty("path.separator");
String[] classPathEntries = System
    .getProperty("java.class.path")
    .split(pathSeparator);

If you’ve used the

URLClassLoader
to dynamically load user provided code (for example as part of a plugin infrastructure) by appending to the class path, then you have to find a new way to do that as it can not be done with Java 11. You should instead consider creating a new class loader for that. This has the added advantage that you’ll be able to get rid of the new classes as they are not loaded into the application class loader. You should also read up on layers – they give you a clean abstraction for loading an entirely new module graph.

Beyond that, your chances to do a migration with only small changes are slim. The only supported (and hence accessible) super types of the new

AppClassLoader
are
SecureClassLoader
and
ClassLoader
and only few methods were added here in 9. Still, have a look, they might do what you’re looking for.

Summary

As executive summary:

  • background:
    • new release every six months
    • pick OpenJDK by default
    • assume that there will be free LTS, otherwise pay a commercial vendor
  • preparations:
    • avoid long-lived branches
    • update all the things
    • keep the module system in mind
  • challenges
    • replace Java EE modules with third-party implementations
    • if absolutely necessary, use
      --add-exports
      or
      --add-opens
      to gain access to internal APIs
    • heed deprecation warnings as classes and methods will be removed
    • add JavaFX as a regular dependency
    • don’t cast the application class loader to
      URLClassLoader

Further reading:

The post All You Need To Know For Migrating To Java 11 appeared first on blog@CodeFX.

Java 11 HTTP/2 API Tutorial

$
0
0

Since Java 11, the JDK contains a new HTTP API in

java.net.http
with
HttpClient
,
HttpRequest
, and
HttpResponse
as its principal types. It’s a fluent, easy-to-use API that fully supports HTTP/2, allows you to handle responses asynchronously, and can even send and receive bodies in a reactive manner. In this post, I introduce you to the new API and show you how to send synchronous and asynchronous requests. Reactive requests and responses are reserved for the next post and a third one may cover web sockets.

Overview

The code snippets shown here are a simplified version of what you can find in my Java X demo repository. Go check it out, the code’s fun to play with!

The Building Blocks

In a nutshell, sending a request and receiving a response follows these steps:

  1. use a builder to create an immutable, reusable
    HttpClient
  2. use a builder to create an immutable, reusable
    HttpRequest
  3. pass the request to the client to receive an
    HttpResponse

I love the focus on immutability!

Right off the bat, I love the focus on immutability! You can configure clients and requests wherever you want, keep them around, and reuse them without worrying about negative interactions between different requests or threads. And even though I recently went on record badmouthing the builder pattern, I think this is a great use case for it.

Let’s quickly go through the steps one by one.

Configuring An HTTP Client

To create an

HttpClient
, simply call

HttpClient.newBuilder()
, configure ahead, and finish with
build()
:

HttpClient client = HttpClient.newBuilder()
    // just to show off; HTTP/2 is the default
    .version(HTTP_2)
    .connectTimeout(Duration.ofSeconds(5))
    .followRedirects(SECURE)
    .build();

Besides the HTTP version, connection timeout, and redirect policy, you can also configure the proxy, SSL context and parameters, the authenticator, and cookie handler. There’s also an

executor
-method, but I’ll deal with that later.

As I mentioned, the client is immutable and thus automatically thread-safe, so feel free to configure once, use everywhere.

Configuring An HTTP Request

To create an

HttpRequest
, apply the same pattern as with the client – call

HttpRequest.newBuilder()
, configure,
build()
:

HttpRequest request = HttpRequest.newBuilder()
    .GET()
    .uri(URI.create("http://codefx.org"))
    .header("Accept-Language", "en-US,en;q=0.5")
    .build();

You don’t have to set the URL in

uri(URI)
and can instead pass it straight to
newBuilder(URI)
. I think I prefer it this way, though, because you can so nicely read it as "GET codefx.org".

With

header(String, String)
, you add a name/value pair to the request’s header. If you want to override existing values for a header name, use
setHeader
. If you have many header entries and don’t want to repeat
header
a whole lot, give
headers(String...)
a try, where you alternative between names and values:

HttpRequest request = HttpRequest.newBuilder()
    .GET()
    .uri(URI.create("http://codefx.org"))
    .headers(
        "Accept-Language", "en-US,en;q=0.5",
        "Accept-Encoding", "gzip, deflate, br")
    .build();

Besides headers and more HTTP methods (

PUT
,
POST
, and the generic
method
), you can request a
"100 CONTINUE"
before sending the request body (if there is one), as well as override the client’s preferred HTTP version and timeout.

If you send anything else but a

GET
, you need to include a
BodyPublisher
when configuring the HTTP method:

BodyPublisher requestBody = BodyPublishers
    .ofString("{ request body }");
HttpRequest request = HttpRequest.newBuilder()
    .POST(requestBody)
    .uri(URI.create("http://codefx.org"))
    .build();

What’s up with

BodyPublisher
, you ask? (He rhetorically asked while being the only person around.) It has to do with handling the request body reactively, which, remember, I’ll cover in the next post. For now it suffices to say that you can get instances of it from
BodyPublishers
– depending in what form your body comes, you can call these (and a few more) static methods on it:
  • ofByteArray(byte[])
  • ofFile(Path)
  • ofString(String)
  • ofInputStream(Supplier<InputStream>)

Pass the returned

BodyPublisher
to the request builder’s
PUT
,
POST
, or
method
and you’re golden.

Receiving An HTTP Response

Receiving an

HttpResponse
is as easy as calling

HttpClient.send(...)
. Well, almost. You also have to provide a so-called
BodyHandler<T>
, which is in charge ob handling the response’s bytes as they are being received and transform them into something more usable. Like with
BodyPublisher
, I’ll go into this later.

For now I’ll just use

BodyHandlers.ofString()
, which means the incoming bytes will be interpreted as a single string. This defines the response’s generic type as
String
:

HttpResponse<String> response = client.send(
    request,
    BodyHandlers.ofString());
// `HttpResponse<T>.body()` returns a `T`
String respnseBody = response.body();

Besides the body, the response also contains the status code, headers, SSL session, a reference to the request, as well as intermediate responses that handled redirection or authentication.

Java HTTP/2 API

Synchronous HTTP Request Handling

Let’s put things together and search the ten longest Wikipedia articles for a given term. Since the upcoming experiments all use the same URLs and search term and can also reuse the same client, we can declare them all in static fields:

private static final HttpClient CLIENT = HttpClient.newBuilder().build();

private static final List<URI> URLS = Stream.of(
    "https://en.wikipedia.org/wiki/List_of_compositions_by_Franz_Schubert",
    "https://en.wikipedia.org/wiki/2018_in_American_television",
    "https://en.wikipedia.org/wiki/List_of_compositions_by_Johann_Sebastian_Bach",
    "https://en.wikipedia.org/wiki/List_of_Australian_treaties",
    "https://en.wikipedia.org/wiki/2016%E2%80%9317_Coupe_de_France_Preliminary_Rounds",
    "https://en.wikipedia.org/wiki/Timeline_of_the_war_in_Donbass_(April%E2%80%93June_2018)",
    "https://en.wikipedia.org/wiki/List_of_giant_squid_specimens_and_sightings",
    "https://en.wikipedia.org/wiki/List_of_members_of_the_Lok_Sabha_(1952%E2%80%93present)",
    "https://en.wikipedia.org/wiki/1919_New_Year_Honours",
    "https://en.wikipedia.org/wiki/List_of_International_Organization_for_Standardization_standards"
).map(URI::create).collect(toList());

private static final String SEARCH_TERM = "Foo";

With the HTTP client, URLs, and search term ready, we can build our requests (one per URL), send them out, wait for the response to return, and then check the body for the search term:

static void blockingSearch() {
    URLS.forEach(url -> {
        boolean found = blockingSearch(CLIENT, url, SEARCH_TERM);
        System.out.println(
            "Completed " + url + " / found: " + found);
    });
}

static boolean blockingSearch(
        HttpClient client, URI url, String term) {
    try {
        HttpRequest request = HttpRequest
            .newBuilder(url).GET().build();
        HttpResponse<String> response = client.send(
            request, BodyHandlers.ofString());
        return response.body().contains(term);
    } catch (IOException | InterruptedException ex) {
        // to my colleagues: I copy-pasted this code
        // snippet from a blog post and didn't fix the
        // horrible exception handling - punch me!
        return false;
    }
}

Depending on my internet connection, running that program takes between 2 and 4 seconds.

Each call to HttpClient::send blocks, wasting precious time and resources

That’s all fine and dandy, but where’s the reactive part?! The naive implementation above blocks on each of the ten requests, wasting precious time and resources! There are three places where the code can be changed to become non-blocking:

  • send request asynchronously
  • provide request body as reactive stream
  • process response body as reactive stream

I’m gonna explain the first one here, and leave the other two for later.

Asynchronous HTTP Request Handling

HttpClient::sendAsync immediately returns a CompletableFuture for the response

The most straightforward way to make the calls non-blocking is to send them asynchronously and

HttpClient
has a method just for that:
sendAsync
sends the request and immediately returns a
CompletableFuture<HttpResponse<T>>
.

By default, the request is handled by an executor service deep in the JVM’s bowels, but if you call

HttpClient.Builder::executor
while building the client, you can define a custom
Executor
for these calls. Whichever executor takes care of the request/response, you can use your thread to continue with more important stuff. For example, requesting the next nine Wikipedia pages. 😉

Not so fast, though, first we need to append some computations to the

CompletableFuture
, so when the request returns, we see the expected output:

static CompletableFuture<Void> asyncSearch(
        HttpClient client, URI url, String term) {
    HttpRequest request = HttpRequest
        .newBuilder(url).GET().build();
    return client
        .sendAsync(request, BodyHandlers.ofString())
        .thenApply(HttpResponse::body)
        .thenApply(body -> body.contains(term))
        .exceptionally(__ -> false)
        .thenAccept(found ->
            System.out.println(
                "Completed " + url + " / found: " + found));
}

As mentioned,

HttpClient::sendAsync
returns a
CompletableFuture<HttpResponse<T>>
that eventually completes with the response. (If you don’t know the
CompletableFuture
API well, think of
thenApply
as
Optional::map
and
thenAccept
as
Optional::ifPresent
. For explanations and more processing options, check the JavaDoc for
CompletableFuture
.) We then extract the request body (a
String
), check whether it contains the search term (thus transforming to a
Boolean
) and finally print that to standard out. We use
exceptionally
to map any errors that may occur while handling the request or response to a "not found" result.

Note that

thenAccept
returns a
CompletableFuture<Void>
:
  • it’s
    Void
    because we are expected to have finished processing the content in the specified
    Consumer
  • it’s still a
    CompletableFuture
    , so we can wait for it to finish

Because, in this demo, that’s what we need to do eventually. The threads running our requests are daemon threads, which means they don’t keep our program alive. If

main
sends ten asynchronous requests without waiting for them to complete, the program ends immediately after the ten sends and we never see any results. Hence:

static void asyncSearch() {
    CompletableFuture[] futures = URLS.stream()
        .map(url -> asyncSearch(CLIENT, url, SEARCH_TERM))
        .toArray(CompletableFuture[]::new);
    CompletableFuture.allOf(futures).join();
}

This usually takes about 75% of the time of the blocking approach, which, I have to admit, I find surprisingly slow. This is not a benchmark, though, so never mind. The principal fact is that our thread is free to do other things while requests are send and responses received in the background.

Handling the request/response lifecycle asynchronously is pretty neat, but it still suffers from a (potential) downside: Both the request’s and the response’s body have to be processed in one piece. I’ll tackle that in my next post.

Summary

You need two ingredients to send a request:

  • With
    HttpClient.newBuilder().$configure().build()
    you get an immutable and reusable
    HttpClient
    . You can $configure preferred HTTP version, timeout, proxy, cookie handler, executor for asynchronous requests, and more.
  • With
    HttpRequest.newBuilder().$configure().build()
    you get an immutable and reusable
    HttpRequest
    . You can override the client’s HTTP version, timeout, and so forth. If the request has a body, provide it as a
    BodyPublisher
    ; you will mostly use the factory methods on
    BodyPublishers
    for that.

With an

HttpClient
and
HttpResponse
in hand, you can call either
send
or
sendAsync
on the former. You also have to provide a
BodyHandler
, which you can get from
BodyHandlers
– it is in charge of transforming response bytes to something more amenable.

If you use

send
, the method call blocks until the response is complete and then returns an
HttpResponse<T>
. If you call
sendAsync
, the call immediately returns with a
CompletableFuture<HttpResponse<T>>
that you can then chain further processing steps to.

And that’s it! Next week: How to process request and response bodies without having to keep them in memory in their entirety.

The post Java 11 HTTP/2 API Tutorial appeared first on blog@CodeFX.

Reactive HTTP/2 Requests And Responses In Java 11

$
0
0

With Java 11’s new HTTP API you can do more than just HTTP/2 and asynchronous requests – you can also handle request and response bodies in a reactive manner, which gives you full control over the bytes going over the wire: You can throttle, you can stream (to conserve memory), and you can expose a result as soon as you found it (instead of waiting for the entire body to arrive).

In this post we’re going to look at streaming request and response bodies and because that requires a working understanding of reactive streams (introduced in Java 9 as Flow API), we’re going to quickly discuss them as well – if you already know how they work skip ahead to Streaming The Request Body. A future post may look into the reactive web sockets API.

Overview

The section Streaming The Response Body builds a solution in several steps, where individual steps may contain bugs that you should not put into your code! For a complete picture, please use the sources on GitHub.

Reactive Stream Crash Course

The HTTP/2 API uses reactive streams to handle request and response bodies. In full force, reactive streams can be used to build pipelines that are similar to Java 8 streams: Starting from a source, a bunch of operations are defined that process each item the source contains/emits.

In reactive streams, the source generates items and would like to push them through the pipeline

There are some important differences, though, most notably how items are moved through the pipeline. With Java 8 streams, the source contains the items and the terminal operation pulls them through the pipeline (think of a collection of tweets that you want to process). In reactive streams, the source generates items and would like to push them through the pipeline (think of a Twitter API that emits tweets live and you want to process them) – "would like to" because subscribers can push back to make sure they’re not overwhelmed.

While reactive streams can be used to build powerful pipelines, the HTTP/2 API uses them in a much simpler manner. Which is good because the JDK only contains the building blocks that you need to connect two steps of a larger pipeline – libraries like RxJava or Project Reactor offer advanced functionality that builds on them. Here are the three involved types:

  • Publisher
    produces items to consume and can be subscribed to. (E.g. the HTTP response can publish bytes as they arrive.)
  • Subscriber
    subscribes to a publisher and offers methods
    onNext
    for new items to consume,
    onError
    for errors the publisher encounters, and
    onComplete
    for the publisher to call when it’s done. (E.g. a JSON parser can subscribe to the response.)
  • Subscription
    is the connection between publisher and subscriber and can be used to request items or cancel the subscription

The programmatic flow is as follows:

  • creation and subscription:
    • you need a
      Publisher pub
      and a
      Subscriber sub
    • call
      pub.subscribe(sub)
    • pub
      creates
      Subscription script
      and calls
      sub.onSubscription(script)
    • sub
      stores
      script
  • streaming:
    • sub
      calls
      script.request(n)
      (where
      n
      is a positive
      int
      )
    • pub
      calls
      sub.onNext(item)
      to push items (max
      n
      times)
    • this continues for as long as publisher and subscriber want and there is no error
  • cancellation:
    • pub
      may call
      sub.OnError(err)
      or
      sub.onComplete()
    • sub
      may call
      script.cancel()

Got it? Lets see it in action!

Streaming The Request Body

If your POST/PUT/… request has a large body, you may not want to load it into memory in its entirety. And with Java’s new reactive HTTP API you don’t have to!

When creating a POST request, for example, you need to provide the body, but you don’t have to do that in the form of a

String
or
byte[]
. Formally, you have to hand over a
BodyPublisher
, which is essentially a
Publisher<ByteBuffer>
, i.e. it publishes blocks of bytes. The HTTP request will then subscribe to that publisher and request bytes to send over the wire.

We can observe that behavior by creating decorators for the interfaces

BodyPublisher
,
Subscriber
, and
Subscription
that log to standard out and then inject them into the HTTP request builder:

HttpClient client = HttpClient.newBuilder().build();
HttpRequest post = HttpRequest
    .newBuilder(POSTMAN_POST)
    // this is where the magic happens!
    .POST(new LoggingBodyPublisher(BodyPublishers.ofFile(LARGE_JSON)))
    .header("Content-Type", "application/json")
    .build();
HttpResponse<String> response = client
    .send(post, BodyHandlers.ofString());

The

POST
call is where the magic happens.
LARGE_JSON
is a
Path
and
BodyPublishers::ofFile
creates a
BodyPublisher
for it that walks the file as needed. We wrap it into the logging decorator and pass the request to the client. Once the streaming starts, you can see the following output:

# the HTTP client created a subscriber and now registers it with the
# file publisher by calling `Publisher::subscribe`
[subscribe   ] Subscriber registered:
    jdk.internal.net.http.Http1Request$FixedContentSubscriber@70ede696
# the file publisher created the subscription and passes it to the
# HTTP client by calling `Subscriber::onSubscribe`
[onSubscribe ] Subscription registered:
    jdk.internal.net.http.PullPublisher$Subscription@4adbc393
# the "handshake" is complete and the HTTP client starts requesting
# items by calling `Subscription::request`
[request     ] Items requested: 1 ↺ 1
# the file publisher received the request for the first item and
# fulfills it by calling `Subscriber::onNext` on the HTTP client
# with a single `ByteBuffer` instance (of 16kb length)
[onNext      ] Bytes passed: 16384 ↺ 16384
# the `request/onNext` cycle continues
[request     ] Items requested: 1 ↺ 2
[onNext      ] Bytes passed: 16384 ↺ 32768
[request     ] Items requested: 1 ↺ 3
[onNext      ] Bytes passed: 16384 ↺ 49152
[... snip ...]
[request     ] Items requested: 1 ↺ 85
[onNext      ] Bytes passed: 16384 ↺ 1392640
# the file publisher realizes that there are no more bytes to
# publish and calls `Subscriber::onComplete` on the HTTP client
[onComplete  ] Publishing completed

The interesting part is that the

BodyPublisher
returned by
BodyPublishers::ofFile
is lazy (it never reads more than it has to fulfill the next request) and that the HTTP client will only request new bytes once the last ones were send over the wire. That means no matter how large the file, you never need to store more than 16kb of it in memory.

It’s easy to integrate with that logic

It’s easy to integrate with that logic and, as a more elaborate example, create a publisher that connects to a database and uses pagination to, at all times, only hold a little window of the entire result in memory while transforming it to a sensible representation and streaming it as part of a request body.

Another great use case for reactive streams is live-processing of the response body. And that’s up next!

Reactive streaming of request and response body with Java 11 HTTP/2 API

Streaming The Response Body

Where a

BodyPublisher
is in charge of publishing bytes that are sent over the wire, a
BodySubscriber<T>
subscribes to the bytes received as part of the response and collects them into an instance of type
T
. The bytes come in the form of lists of byte buffers, meaning
BodySubscriber
extends
Subscriber<List<ByteBuffer>>
. Implementing that means extracting bytes from buffers, being aware of charsets, deciding where to split the resulting string, and so forth… in short, it’s a pain. So we’re not going to do it.

Of BodyHandlers and BodySubscribers

Instead we can implement a

Subscriber<String>
and pass it to
BodyHandlers::fromLineSubscriber
:

static CompletableFuture<Void> reactiveSearch(
        HttpClient client, URI url, String term) {
    HttpRequest request = HttpRequest.newBuilder(url).GET().build();
    // TODO: we need a subscriber
    Subscriber<String> stringFinder = // ...
    // TODO: we need to do something with the `CompletableFuture`
    client.sendAsync(request, BodyHandlers.fromLineSubscriber(finder));
    // TODO: we need to get the result out of the `stringFinder`
    //       and return it here as a `CompletableFuture`
    return // ...
}

What’s a

BodyHandler
, though? (He once again asked no one in particular, although the other passengers now regard him with a funny look.)

A

BodyHandler<T>
is in charge of evaluating the response’s status code, HTTP version, and header lines and to create a
BodySubscriber<T>
for the response’s bytes. The generic type
T
indicates what these bytes will eventually be transformed to and determines the
T
in
HttpResponse<T>
and thus the return type of
HttpResponse::body
. In the HTTP/2 tutorial as well as in the earlier example for streaming the request body, we called
BodyHandlers::ofString
to get a
BodyHandler<String>
, which represents the entire response body as a single string, and passed it to the client’s send methods.

This time around, we’re going to call

BodyHandlers.fromLineSubscriber(Subscriber<String>)
, though, which gives us more to do but also more freedom: It wraps our subscriber into a
BodySubscriber<Void>
(Why
Void
? Later!) that aggregates the lists of byte buffers to strings, takes them apart on newlines, and then expects our subscriber to handle these individual response lines. In return we don’t need to wait for the entire body to arrive before we can process it.

By now you know the protocol a subscriber has to follow, so lets quickly implement a bare-bones variant:

private static class StringFinder implements Subscriber<String> {

    private final String term;
    private Subscription subscription;

    private StringFinder(String term) {
        this.term = term;
    }

    @Override
    public void onSubscribe(Subscription subscription) {
        this.subscription = subscription;
        this.subscription.request(1);
    }

    @Override
    public void onNext(String line) {
        // TODO: scan the line and, if found, expose positive result
        subscription.request(1);
    }

    @Override
    public void onError(Throwable ex) {
        // TODO: expose the error
    }

    @Override
    public void onComplete() {
        // entire body was processed, but term was not found;
        // TODO: expose negative result
    }

}

As you can see,

StringFinder
implements the reactive subscriber contract by storing the subscription, requesting items (in this case lines; one by one), and processing them.

Exposing The Result With CompletableFuture

As you can also see, there are a few TODOs left – they all revolve around how to expose the result, which can either be:

  • positive: term found in body
  • negative: term not found in body
  • error: HTTP client reported an exception

We can cover these three use cases with a

CompletableFuture<Boolean>
:

private static class StringFinder implements Subscriber<String> {

    private final CompletableFuture<Boolean> found =
        new CompletableFuture<>()

    // [... other fields, constructor, `onSubscribe` as above...]

    @Override
    public void onNext(String line) {
        if (!found.isDone() && line.contains(term))
            found.complete(true);
        subscription.request(1);
    }

    @Override
    public void onError(Throwable ex) {
        found.completeExceptionally(ex);
    }

    @Override
    public void onComplete() {
        found.complete(false);
    }

    public CompletableFuture<Boolean> found() {
        return found;
    }

}

(Remember, check the demo project for the complete example.)

Let’s plug

StringFinder
into
reactiveSearch
and see what we’ve got:

static CompletableFuture<Void> reactiveSearch(
        HttpClient client, URI url, String term) {
    HttpRequest request = HttpRequest.newBuilder(url).GET().build();
    StringFinder finder = new StringFinder(term);
    // DANGER ZONE!
    client.sendAsync(request, BodyHandlers.fromLineSubscriber(finder));
    return finder
        .found()
        .exceptionally(__ -> false)
        .thenAccept(found -> System.out.println(
            "Completed " + url + " / found: " + found));
}

We pass the

StringFinder
to
fromLineSubscriber
, which wraps it into a
BodyHandler
, and then return the
CompletableFuture
our finder exposes. Something’s off, though: What’s with the future returned by
sendAsync
? Don’t we need that as well? Kinda… but we have to take a small detour to get there.

There’s an overload of

fromLineSubscriber
with the semantics that, once our subscriber processed the entire body, its result is made available as the request’s body. Given a
Function<Subscriber<String>, T>
(called a
finisher
) that extracts the result, it creates a
BodyHandler<T>
, which leads to an
HttpResponse<T>
.

The

fromLineSubscriber
we called has different semantics, tough: The provided subscriber processes the body however it pleases and without having to make it available afterwards. It hence returns a
BodyHandler<Void>
, leading to an
HttpResponse<Void>
, meaning
sendAsync
returns a
CompletableFuture
that completes when the response is fully processed but never exposes the body.

Never ignore sendAsync’s return value!

If that sounds like just another reason to ignore

sendAsync
’s return value, I’ve successfully led you down the same erroneous line of thought that I followed. And I may even have published the post this way, would I not have worked on it 35’000 feet above ground without internet connection.

Handling Errors

While

StringFinder
properly handles errors by exposing them via its
CompletaleFuture
, these aren’t all errors that can occur. On the contrary, these are just the small subset of errors that may happen while the body is streamed from server to client (e.g. loss of connection).

But there are plenty of reasons why we don’t even get to streaming the body! Not being able to establish the connection, for example. Because you’re in the air, for example. In which case

StringFinder
is never subscribed to anything, its
CompletableFuture
never completes, and waiting for it blocks forever. Where did we go wrong? Where do those kinds of errors surface?

Here’s where the

CompletableFuture
that
sendAsync
returns comes back in. It’s the thing that exposes such errors! And so we need to hook into its exception handling and make our finder’s future complete with the same exception:

StringFinder finder = new StringFinder(term);
client
    .sendAsync(request, BodyHandlers.fromLineSubscriber(finder))
    // this is a `CompletableFuture<Void>` ...
    .exceptionally(ex -> {
        finder.onError(ex);
        // ... which is why we need to return `null`
        return null;
    });

This way, the

CompletableFuture<Boolean>
returned by
StringFinder
surfaces all possible outcomes that can occur while fielding the HTTP request:
  • it will complete with
    true
    as soon as the term is found
  • it will complete with
    false
    if the entire body was scanned
  • it will complete with an exception if there is any problem (including those that occur before the body is streamed)

Neat!

Remember from the last post, that when searching the ten longest Wikipedia articles for "Foo", async calls took about 80% of the time that blocking calls took. Streaming the body instead of putting it together in its entirety reduces memory footprint, which, as is common, results in longer run time. Just, barely, though, it’s still about 85% of the blocking calls.

Canceling The Stream

Only one nit remains: We’re always streaming the entire body, even after we found the search term. Can’t we abort the stream once we’re done? Technically, yes, and it isn’t even complicated – all we need to do is add one line to

StringFinder::onNext
:

@Override
public void onNext(String line) {
    if (line.contains(search.term())) {
        found.complete(true);
        subscription.cancel();
    }
    requestLine();
}

By canceling the subscription, we won’t receive any more lines and this really shows when measuring run time: This takes about 45% of the time the blocking calls take, i.e. about 55% of the asynchronous approach. But keep in mind that this speedup highly depends on how soon the search term is found! If you search for "Foobar" instead of "Foo", none of the ten sites contains it (what a shame), and performance is back to the runtime without cancellation.

Regarding cancellation, there are two more details that I should mention. The first one is that canceling the subscription leads to the client calling

onError
with an exception like the following:

java.util.concurrent.CompletionException:
    java.io.IOException: Stream 47 cancelled

Since

onError
calls
found.completeExceptionally
, the future must already have been completed by then (or the result is always an error instead of
true
). That’s why
found.complete(true)
must come before
subscription.cancel()
!

Finally, here’s what the JavaDoc for

BodySubscriber
has to say about canceling the subscription:

Calling

cancel
before exhausting the response body data may cause the underlying HTTP connection to be closed and prevent it from being reused for subsequent operations.

I’m no expert on HTTP and refrain from making any recommendations based on that quote.

Reflection

With that it’s time to wrap it up. In short:

  • reactive streams have two active players: a publisher and a subscriber, where the former publishes items that the latter consumes
  • to stream a request body, you need a
    Publisher<ByteBuffer>
    , to which the the HTTP client will subscribe and then send requested bytes over the wire
  • to stream a response body, you will typically implement a
    Subscriber<String>
    that the client subscribes to the incoming response bytes, which it translates to a string and breaks a part line by line for the subscriber to consume
  • be careful to properly handle errors

You can be proud of yourself – learning about reactive streams and how Java’s new HTTP/2 API uses it, is no easy feat. 👍 It becomes clearer if you play around with it yourself, so I want to point you to the demo one last time: clone it, play around with it, break it, fix it. Best way to learn.

The post Reactive HTTP/2 Requests And Responses In Java 11 appeared first on blog@CodeFX.

Scripting Java 11, Shebang And All

$
0
0

There are several reasons why writing scripts in Java seems to be a bad idea, chief among them that it’s always a two step process to run a source file: It first has to be compiled (with

javac
) before it can be executed (with
java
). Enter Java 11, which, for a single source file, blends these two steps into one:

java HelloJavaScripts.java

Yes, you saw that right: The JVM accepts a source file and executes it. More than that, you can even define proper scripts, with shebang and everything, that you can execute like this:

./hello-java-scripts

Let’s see how!

Overview

As usual, find the sources in my Java X demo project.

Single-Source-File Execution

Straightforward

Executing a single source file is straightforward. Simply write a self-contained class with a

main
method …

public class HelloJavaScripts {

    public static void main(String[] args) {
        System.out.println("Hello, Java scripts!");
    }

}

… and pass that file to

java
:

$ java HelloJavaScripts.java
> Hello, Java scripts!

There you go, you’ve just executed your first Java script! (Pun fully intended.) And with that you can already start experimenting, but, as usual, there are a few details that you should know. Let’s discuss them next before coming to what may be Java 11’s hidden killer feature.

Prerequisites

First of all, the JVM can only compile source files if the jdk.compiler module is present. That’s the case for all full JDKs, but if you’re building your own images with

jlink
, you may well end up without it. In that case, you’ll see this error message:

Error: A JNI error has occurred,
    please check your installation and try again
Exception in thread "main" java.lang.InternalError:
    Module jdk.compiler not in boot Layer

There’s no way to fix this – you simply have to use a runtime image with that module.

On To The Details!

Once you start using single-source-file programs, you’ll quickly end up in situations where it helps or is even required to explicitly inform the JVM that it’s supposed to execute a source file and which Java version to compile it against. You do that with the

--source
option:

$ java --source 11 HelloJavaScripts.java

This is particularly interesting in conjunction with preview features (where specifying the source is a requirement anyways):

# running my switch expression demo with Java 12;
# more on that: https://www.youtube.com/watch?v=1znHEf3oSNI
$ java --source 12 --enable-preview Switch.java

You can name the file any way you want

Another situation where

--source
needs to be added is if the source file name does not end in
.java
. Yes, you got that right, the
.java
suffix is not mandatory! In fact, you can name the file any way you want:

$ java --source 11 hello-java-scripts

Besides

--source
and
--enable-preview
, the JVM processes many other command line options like
--class-path
,
--module-path
, and those to hack the JPMS. If you end up using a lot of flags, you can put them into a so-called @-file and reference that:

# content of file `args`:
--source 12
--enable-preview
--class-path 'deps/*'
# use that file
$ java @args HelloJavaScripts.java

Whatever command line flags you add, as you would expect, all arguments after the file name are passed to the program’s

main
method:

$ java --source 11 Greetings.java hello java scripts
# main receives [ "hello", "java", "scripts" ]

Last and maybe even least, the compiled source will be executed in the unnamed module of a class loader specifically created for it alone. That class loader’s parent is the application class loader (which loads the class path content), which means the main class has access to all dependencies on the class path.

Since the application class loader can’t also access the "main class" loader (no circular class loader dependencies allowed), the inverse is not true and classes from the class path won’t have access to the main class. I know, I know, sadly we can’t execute our Spring/Hibernate applications from a single source file. 😭

But Why?!!

You may wonder what this feature is good for. I mean, if you can’t even write a web-backend with it…

Its primary use case is similar to jshell, Java’s REPL: You can use it to run quick experiments, particularly in environments without an IDE. But unlike with jshell, you’ll be able to enjoy syntax highlighting (assuming you have access to something more advanced than Notepad), the lack of which renders the REPL unusable to me.

Although, if you’re like me, you have at least three IDE instances running at all times anyways and starting an experiment requires nothing more than Alt-Tabbing to one of them, letting it spew out a

main
or test method, and off you go. So for me, experimentation is not an important use case for single-source-file execution.

But if you occasionally write a demo or two, turning each into a single self-contained and executable file will make it easier for your audience. Sharing a single file is simpler than distributing a few of them and your audience can decide whether they want to fire up an IDE or simply throw the file at the JVM. That may make it easier for them to get started – particularly if they’re Java beginners.

One detail that may end up driving that use case are incubator modules and preview features. These are mechanisms that the JDK team can use to let us experiment with not-yet-finalized APIs and syntax. Unlocking these features requires special compiler and JVM commands and putting them into the right places in an IDE can be tedious and fragile. As we have seen above, adding them to

java
is much simpler:

$ java --source 12 --enable-preview Switch.java

There’s one other way to use single-source-files, though, and it will knock your socks off! (If you’re on Linux or macOS.)

Java Scripts With Shebang

You can add a shebang to source files

We’ve already seen most of the ingredients for scripts above but one is still missing: the shebang. The big news is, you can add it to Java source files and the JVM will ignore it when compiling the source!

Here’s the file

hello-java-scripts
:

#!/opt/jdk-11/bin/java --source 11
public class HelloJavaScripts {

    public static void main(String[] args) {
        System.out.println("Hello, Java scripts!");
    }

}

If the file is executable (with

chmod +x hello-java-scripts
), you can run it with
./hello-java-scripts
or, if it’s on your
PATH
, even with
hello-java-scripts
. Arguments following the script’s name are naturally passed on to the
main
method.

If you need to add further compiler or JVM flags, you can either put them into the source file after the

--source
option or fall back to explicitly launching the JVM as usual:

java -Xlog --source 11 hello-java-scripts

(What’s

-Xlog
you ask? Here you go.)

FYI: Before passing the file to the compiler, the JVM will replace the shebang line with an empty one. This keeps the compiler from barfing while preserving line numbers, which is handy for fixing compile errors.

"Are You Serious?!"

Fair question. As I see it, there are three criticisms of writing scripts with Java:

  • compilation and execution is a two-step process
  • Java’s programming model is not conducive to quick results
  • the JVM is slow to boot

As we’ve discussed at length, the first bullet is no longer true. The third is definitely true, although it would be interesting to see whether we could add Graal native images to the mix. The second bullet feels true, but I’m not sure whether that’s actually still the case.

Admittedly, things like Java’s file system interaction and HTTP requests are powerful but not exactly elegant. Other things are, though. Lambdas and streams, local-variable type inference with

var
, the upcoming switch expressions – all of these make Java quite expressive and easy to achieve results with. Let’s see an example.

Echo – The Java Way

The Linux command line has a very simple tool called

echo
that simply prints to the terminal whatever you pass to it:

$ echo "Hello, world"
> Hello, world

That’s too boring, even for Java, so let’s try something a little more advanced. In Linux you can "pipe" the result of one command into the next. This doesn’t actually work with

echo
(it doesn’t read from
stdin
), but let’s do it in our Java variant nonetheless:

#!/opt/jdk-11/bin/java --source 11
//  [... imports ...]

public class Echo {

    public static void main(String[] args) throws IOException {
        var lines = readInput();
        lines.forEach(System.out::println);
    }

    private static Stream<String> readInput() throws IOException {
        var reader = new BufferedReader(
            new InputStreamReader(System.in));
        if (!reader.ready())
            return Stream.empty();
        else
            return reader.lines();
    }

}

As you can see,

readInput()
tries to read from
System.in
, which is connected to
stdin
. The
if
checks whether there is any input at all because if not,
reader.lines()
will block until that changes – we want to avoid that. (I know, I know, awkward Java in action…)

Putting this into a file

echo
and making it executable allows piping text into it – the content of
haiku.txt
, for example:

$ cat haiku.txt | ./echo
# this is the unaltered content of haiku.txt
> worker bees can leave
> even drones can fly away
> the queen is their slave

Now, thanks to streams, it’s easy to throw a few more complex operations at the input. Adding command line options for sorting and making lines unique, for example, are one-liners:

public static void main(String[] args) throws IOException {
    var lines = readInput();
    // modify the stream according to command line options
    for (var arg : args)
        lines = modifyStream(arg, lines);
    lines.forEach(System.out::println);
}

private static Stream<String> modifyStream(String arg, Stream<String> input) {
    switch (arg){
        case "--sort": return input.sorted();
        case "--unique": return input.distinct();
        default: {
            System.out.println("Unknown argument '" + arg + "'.");
            return input;
        }
    }
}

In action:

$ cat haiku.txt | ./echo --sort
> even drones can fly away
> the queen is their slave
> worker bees can leave

That’s not too bad, is it?

Reflection

Running source files in two easy steps:

  • write a self-contained class with a
    main
    method
  • throw it at the JVM with
    java Script.java

Because it’s often needed, you should by default add

--source
, though. For example when experimenting with preview features:
  • write an experimental class with a
    main
    method
  • run it with
    java --source 12 --enable-preview Switch.java

Finally, consider scripting with Java:

  • write a self-contained source file with a
    main
    method
  • add a shebang with the path to your Java install as first line, for example
    #!/opt/jdk-11/bin/java --source 11
  • name the file any way you want, for example just
    script
  • make it executable with
    chmod +x script
  • run it with
    ./script

Remember that you can add all kinds of command line flags either to

java
or to the shebang line (in that case, they have to come after
--source
). Options following the source file name are passed to
main
.

The post Scripting Java 11, Shebang And All appeared first on blog@CodeFX.

Eleven Hidden Gems In Java 11

$
0
0

Java 11 introduced no ground-breaking features, but contains a number of gems that you may not have heard about yet. Sure, you likely know about the reactive HTTP/2 API and executing source files without compiling them, but did you try out the additions to

String
,
Optional
,
Collection
, and other workhorses. If not, you’ve come to the right place: Here are eleven hidden gems in Java 11!

Overview

As always, you can find all code snippets in the Java X demo project on GitHub.

Eleven Gems

Type Inference For Lambda Parameters

When writing a lambda expression, you can choose between specifying the types or omitting them:

Function<String, String> append = string -> string + " ";
Function<String, String> append = (String s) -> s + " ";

Java 10 introduced

var
, but you couldn’t use it in lambdas:

// compile error in Java 10
Function<String, String> append = (var string) -> string + " ";

In Java 11 you can. Why, though? It’s not like

var
adds anything over just omitting the type. While that is the case, allowing
var
has two minor advantages:
  • makes the mental model for
    var
    more uniform by removing a special case
  • allows type annotations on lambda parameters without having to resort to a full type name

Here’s an example for the second point:

List<EnterpriseGradeType<With, Generics>> types = /*...*/;
types.stream()
    // this is fine, but we need @Nonnull on the type
    .filter(type -> check(type))
    // in Java 10, we need to do this ~> ugh!
    .filter((@Nonnull EnterpriseGradeType<With, Generics> type) -> check(type))
    // in Java 11, we can do this ~> better
    .filter((@Nonnull var type) -> check(type))

While mixing implicit types, explicit types, and

var
in lambdas like
(var type, String option, index) -> ...
could be supported, it would (apparently) make the implementation more complicated. You hence have to choose one of the three approaches and stick with it for all parameters. Having to add
var
to all parameters just to apply an annotation to one of them, may be mildly annoying, but I think it’s bearable.

Streaming Lines With ‘String::lines’

Got a multiline string? Want to do something with every line? Then

String::lines
is the right choice:

var multiline = "This\r\nis a\r\nmultiline\r\nstring";
multiline.lines()
    // we now have a `Stream<String>`
    .map(line -> "// " + line)
    .forEach(System.out::println);

// OUTPUT:
// This
// is a
// multiline
// string

Note that the string uses Windows’

\r\n
and even though I’m on Linux,
lines()
still splits it. That’s because regardless of the operating system, the method treats
\r
,
\n
, and
\r\n
as line terminators and splits there – even if they are mixed in the same string.

The streamed lines never contain the line terminator itself. They can be empty (

"like\n\nin this\n\ncase"
, which has 5 lines), but the line at the end of the string will be ignored if its empty (
"like\nhere\n"
; 2 lines).

Unlike

split("\R")
,
lines()
is lazy and, I quote, "provides better performance […] by faster search of new line terminators". (If someone feels like firing up JMH to verify this, let me know.) It’s also much better at conveying what you want to do and returns a more convenient data structure (stream instead of array). Neat.

Stripping Whitespace With ‘String::strip’ et al

Since forever,

String
offered
trim
to remove whitespace, which it considered everything with a Unicode up to U+0020. Yep,
BACKSPACE
(U+0008) is whitespace and so is
BELL
(U+0007) but
LINE SEPARATOR
(U+2028) isn’t. 🤔

Java 11 introduces

strip
, which has a little more nuanced approach. It uses Java 5’s
Character::isWhitespace
to determine what to strip. From its Javadoc that is:
  • SPACE_SEPARATOR
    ,
    LINE_SEPARATOR
    ,
    PARAGRAPH_SEPARATOR
    , but not non-breaking space
  • HORIZONTAL TABULATION
    (U+0009),
    LINE FEED
    (U+000A),
    VERTICAL TABULATION
    (U+000B),
    FORM FEED
    (U+000C),
    CARRIAGE RETURN
    (U+000D)
  • FILE SEPARATOR
    (U+001C),
    GROUP SEPARATOR
    (U+001D),
    RECORD SEPARATOR
    (U+001E),
    UNIT SEPARATOR
    (U+001F)

Building on this logic, there are two more stripping methods,

stripLeading
and
stripTailing
, which do what you’d expect.

Finally, if you just need to know whether a string would be empty after stripping whitespace, no need to actually do it – use

isBlank
instead:

" ".isBlank()); // space ~> true
" ".isBlank()); // non-breaking space ~> false

Repeating Strings with ‘String::repeat’

Life hack:

Step 1
Obsessively observe JDK development. String::repeat in Java 11
Step 2
Scour StackOverflow for related questions. String::repeat in Java 11 😍
Step 3
Swoop in with new answer based on upcoming changes. String::repeat in Java 11
Step 4

¯\_(ツ)_/¯

Step 5

As you can see,

String
now has a method
repeat(int)
. It behaves exactly according to expectations and there aren’t any nooks and crannies to discuss.

Creating Paths With ‘Path::of’

I really like the

Path
API but going back and forth between paths as
Path
,
File
,
URL
,
URI
, and
String
, is really annoying. This has gotten a tiny bit less confusing in Java 11 by copying the two
Paths::get
methods to
Path::of
:

Path tmp = Path.of("/home/nipa", "tmp");
Path codefx = Path.of(URI.create("http://codefx.org"));

They can be deemed to be the canonical choice as both

Paths::get
methods forward to them.

Reading From And Writing To Files With ‘Files::readString’ and ‘Files::writeString’

If I need to read from a large file, I usually use

Files::lines
to get a lazy stream of its content. Likewise, for writing a lot of content that may not be present in memory all at once, I use
Files::write
by passing it an
Iterable<String>
.

But what about the easy case where I can handle the entire content as a simple string? That hasn’t been terribly convenient because

Files::readAllBytes
and the matching overload for
Files::write
operate with byte arras. 🤢

Here’s where Java 11 interjects by adding

readString
and
writeString
to
Files
:

String haiku = Files.readString(Path.of("haiku.txt"));
String modified = modify(haiku);
Files.writeString(Path.of("haiku-mod.txt"), modified);

Straightforward and simple to use. If need be, you can also pass a

CharSet
to
readString
and
OpenOption
s to
writeString
.

Null I/O With ‘Reader::nullReader’ et al

Need an

OutputStream
that discards input bytes? Need an empty
InputStream
? What about
Reader
and
Writer
that do nothing? Java 11 has got you covered:

InputStream input = InputStream.nullInputStream();
OutputStream output = OutputStream.nullOutputStream();
Reader reader = Reader.nullReader();
Writer writer = Writer.nullWriter();

I wonder, though, is

null
really the best prefix here? I don’t like how it’s used to mean "intended absence"… Maybe
noOp
would have been better?

{ } ~> [ ] With ‘Collection::toArray’

How do you turn a collection into an array?

// before Java 11
List<String> list = /*...*/;
Object[] objects = list.toArray();
String[] strings_0 = list.toArray(new String[0]);
String[] strings_size = list.toArray(new String[list.size()]);

The first option,

objects
, looses all type information, so it’s out. What about the other two? Both are cumbersome, but the first is more succinct. The latter creates an array with the required size, so it’s more performanty (i.e. "appears more performant"; cf. truthy). But does it actually perform better? No, on the contrary, it’s slower (at the moment).

But why should we care about that? Isn’t there a better way to do this? In Java 11 there is:

String[] strings_fun = list.toArray(String[]::new);

There’s a new overload of

Collection::toArray
that takes an
IntFunction<T[]>
, i.e. a function that accepts the length of the array to produce as input and returns an array of that size. That can be expressed succinctly as a constructor reference
T[]::new
(for concrete
T
).

Fun fact, the default implementation of

toArray(IntFunction<T[]>)
always passes 0 to the provided array generator. At first, I thought that decision was made based on the better performance of starting out with such a 0-length array, but now I think it may be because, for some collections, computing the size can be very expensive and so it wouldn’t be a good default implementation on
Collection
. Concrete collections like
ArrayList
could then override, but, in Java 11, they don’t. Not worth it, I guess.

This new method supersedes

toArray(T[])
unless you already have an array lying around, in which case the old method remains useful.

Not Present With ‘Optional::isEmpty’

When you use

Optional
a lot, particularly in large code bases where you interact with a lot of non-

Optional
-bearing code, you’ll frequently have to check whether the value is present.
Optional::isPresent
is there for you. But about just as often, you want to know whether the
Optional
is empty. No problem, just use
!opt.isPresent()
, right?

Sure, that works, but it’s almost always easier to understand an

if
if the condition is not negated. And sometimes, the
Optional
pops up at the end of a longer call chain and if you want to know whether it’s empty, you need to put the
!
all the way to the front:

public boolean needsToCompleteAddress(User user) {
    return !getAddressRepository()
        .findAddressFor(user)
        .map(this::canonicalize)
        .filter(Address::isComplete)
        .isPresent();
}

The

!
is easy to miss. From Java 11 on, there’s a better option:

public boolean needsToCompleteAddress(User user) {
    return getAddressRepository()
        .findAddressFor(user)
        .map(this::canonicalize)
        .filter(Address::isComplete)
        .isEmpty();
}

Inverting Predicates With ‘Predicate::not’

Talking about "not"… The

Predicate
interface has an instance method
negate
; it returns a new predicate that executes the same test but inverts the result. Unfortunately, I very rarely get to use it…

// want to print non-blank strings
Stream.of("a", "b", "", "c")
    // ugh, lambda ~> want to use method reference and negate it
    .filter(string -> !string.isBlank())
    // compiler has no target for method reference ~> error
    .filter((String::isBlank).negate())
    // ugh, cast ~> this is way worse than lambda
    .filter(((Predicate<String>) String::isBlank).negate())
    .forEach(System.out::println);

The problem is that I rarely have a

Predicate
instance at hand. Much more often, I want to create one with a method reference (and then invert it), but for that to work the compiler needs to know the target – without it, it doesn’t know what to turn the reference into. And that’s what happens when you attach a method call as in
(String::isBlank).negate()
: There’s no longer a target for
String::isBlank
and so the compiler barfs. A properly placed cast fixes that but at what cost?

There’s an easy solution, though. Don’t use the instance method

negate
, use Java 11’s new static method
Predicate.not(Predicate<T>)
instead:

Stream.of("a", "b", "", "c")
    // statically import `Predicate.not`
    .filter(not(String::isBlank))
    .forEach(System.out::println);

I like it!

Regular Expressions As Predicate With ‘Pattern::asMatchPredicate’

Have a regular expression? Want to filter based on it? What about this:

Pattern nonWordCharacter = Pattern.compile("\\W");
Stream.of("Metallica", "Motörhead")
    .filter(nonWordCharacter.asPredicate())
    .forEach(System.out::println);

I was really happy to discover this method! This Java 8 method, I should add. Oops, missed that one. 😂 Java 11 adds another such method:

Pattern::asMatchPredicate
. What’s the difference?
  • asPredicate
    checks whether the string or any substring matches this pattern (it behaves like
    s -> this.matcher(s).find()
    )
  • asMatchPredicate
    is only content if the entire string matches this pattern (it behaves like
    s -> this.matcher(s).matches()
    )

Say you have a regular expression verifying phone numbers, but it doesn’t contain

^
and
$
to mark begin and end of line. Then the following may not do what you want it to:

prospectivePhoneNumbers.stream()
    .filter(phoneNumberPatter.asPredicate())
    .forEach(this::robocall);

Did you spot the error? A string like

"y u want numberz? +1-202-456-1414"
would pass the filter because it contains a valid phone number.
Pattern::asMatchPredicate
, on the other hand, would not have let it pass because the string, in its entirety, doesn’t match the pattern.

Reflection

Here are all eleven-something gems at a glance – see if you still remember what each of them does. If you do, you passed. 😉

  • on
    String
    :
    • Stream<String> lines()
    • String strip()
    • String stripLeading()
    • String stripTrailing()
    • boolean isBlank()
    • String repeat(int)
  • on
    Path
    :
    • static Path of(String, String...)
    • static Path of(URI)
  • on
    Files
    :
    • String readString(Path) throws IOException
    • Path writeString(Path, CharSequence, OpenOption...) throws IOException
    • Path writeString(Path, CharSequence, Charset, OpenOption...) throws IOException
  • on
    InputStream
    :
    static InputStream nullInputStream()
  • on
    OutputStream
    :
    static OutputStream nullOutputStream()
  • on
    Reader
    :
    static Reader nullReader()
  • on
    Writer
    :
    static Writer nullWriter()
  • on
    Collection
    :
    T[] toArray(IntFunction<T[]>)
  • on
    Optional
    :
    boolean isEmpty()
  • on
    Predicate
    :
    static Predicate<T> not(Predicate<T>)
  • on
    Pattern
    :
    Predicate<String> asMatchPredicate()

And that’s beyond the reactive HTTP/2 API and single-source-file execution. Have fun with Java 11!

The post Eleven Hidden Gems In Java 11 appeared first on blog@CodeFX.

Viewing all 68 articles
Browse latest View live