Published slides from my “Recursive implicit proofs with Shapeless HLists” talk

Recently I was giving an introductory talk about constructing implicit proofs with Shapeless at Scala Riga Meetup.

Contents in brief

  • Analogy between Scala implicit resolution and Mathematical proofs.
  • Recursive implicit proofs (why “recursive” is important feature)
  • HList basics
  • Various implicit proofs for HLists with demos. Mathematical induction as a handy tecnique to writing HList proofs.
  • Expanding HList proof to all product-like types with shapeless.Generic
Slides (english, PDF)
Slides (english, Slideshare)

Implementing type-safe request builder: practical use of HList constraints

Hello, Scala developers!

In this post we will develop a simple type-safe request builder. Along the way, we will:

  • encode domain rules and constraints at compile time with implicits;
  • heavily use HLists;
  • use existing and implement a couple of new HList constraints.

The example in this post is a simplified version of a real request builder from my Scalist project. Implementation of intentionally omitted concepts, like response parsing and effect handling, can be found there.

The code from this article can be found on GitHub.

Prerequisites

I assume that you, Dear Reader, have a basic understanding of:

  • implicit resolution;
  • typeclasses;
  • what an HList is.

If some of those are missing, I encourage you to come back later: there’s a lot of good articles and talks on these topics out there in the web.

The problem

We will be developing a query request builder for and API that is capable of joining together multiple logical requests and returning all results in one single response. A great example is Todoist API: you are allowed to say “Give me projects, labels and reminders”, and the API will return all of them in a single JSON response.

Such an optimization is great from performance perspective. On the other hand, it imposes difficulties for a type-safe client implementation:

  • you need to somehow track the list of requested resources;
  • return value type must reflect this list and not allow to query for something else.

Those problems are not easy to solve at compile time, but, as you might have heard, almost everything is possible with Shapeless 🙂

The task

The task here is to implement a request builder with following features:

  • allows to request single resource like this:
    builder.get[Projects].execute
  • allows to request multiple resources like this:
    builder.get[Projects].and[Labels].and[Comments].execute
  • doesn’t allow requesting duplicates. This will not compile:
    builder.get[Projects].and[Projects]
  • doesn’t allow requesting types that are not resources. This will not compile:
    builder.get[Passwords]
  • execute method will return something that will contain only requested resources and will not allow to even try getting something else out of it.

Simplifications

To focus on the main topic, I will cut off all other aspects of implementing a good HTTP API client, like sending the request, parsing, abstracting over effects and so on.

To keep things simple, let’s make  execute method just ask for an implicit MockResponse typeclass instance, that will supply requested values instead of doing everything mentioned above.

Let’s code!

Model

We’ll start with some simple things, that are required though. First let’s define a model to play with. Just some case classes from task management domain:

 

What we are actually going to request are lists of domain objects. But get[Projects] looks better than get[List[Project]], so let’s define some type aliases. Also, this is a good place to put an instance of our builder:

 

Good. We’ll get to the Builder later. Now let’s define

APIResource typeclass

An instance of APIResource[R] is a marker that R can be requested with our builder.

 

A couple of comments here:

  • In a real case typeclass body will not be empty — it’s a good place to define some specific entity-related properties. A resource identifier, that is used to create a request can be a good example.
  • We mark APIResource as sealed here, because we know all the instances upfront and don’t want library users to create new ones.

Now we’re ready to implement the Builder.

Single resource request

Builder call chain starts with get method — let’s create it:

 

Quite simple for now: we allow the method to be called only for types that have an implicit APIResource instance in scope. It returns a request definition that can be executed or extended further with and method.

We will solve the execution task first. A RequestDefinition trait defines execute method for all request definitions:

 

There’s the MockResponse thing I was talking about. It allows us to avoid implementing all the real machinery that is not relevant for the topic of this post.
Almost everything about this typeclass is simple and straightforward, but there’s an interesting thing that will show up later, so I have to put the implementation here to reference it.

 

Ok, we’re able to execute requests! Let’s see how we can solve the chaining task with and method.

Chaining: 2 resources request

First, we have to decide, value of what type should be returned by an executed multiple resource request. Standard List or Map could do the job in general, but not with our requirements — precious type information will be lost.

This is where HList comes in handy — it’s designed to store several instances of not related types without loss of any information about those types.

Next. When implementing a type-safe API, we have to put all domain constraints into declarations, so that they’re available for the compiler. Let’s write out, what is required to join two resources in one request:

  • both of them must have an APIResource instance in place, and
  • their types must be different

Good, we’re ready to make our first step to a multiple resource request definition:

 

All our constraints are in place:

  • the context bound ensures that original resource type is valid.
  • AR: APIResource[RR] implicit parameter ensures that newly added resource type is valid.
  • NEQ: RR =:!= R implicit ensures that R and RR are different types.

Also, you can notice the HList in the result type parameter. It ensures the execution result to be precisely what we requested at compile time.

Now we’re all set up to dive into the most interesting and complex problem.

Chaining: multiple resources request

Here the task is to append a new resource request R to a list L of already requested ones. Again, let’s first define required constraints in words:

  1. every element of  L must have an APIResource instance in place;
  2. R must have an APIResource instance too;
  3. L must not contain duplicates. Let’s call such a list “distinct”;
  4. L must not already contain R . In another words, result list must be distinct too.

That’s a lot. Let’s see, what implementation we can come up with here.
We will start from the MultipleRequestDefinition class itself:

 

Actually, we already have everything for our first requirement. A LiftAll typeclass from shapeless ensures that every element of an HList has a specified typeclass instance.
In our case, implicit allAR: LiftAll[APIResource, L] constraints L to have an APIResource for each element.

Implicit ID: IsDistinctConstraint[L] will ensure that all elements of L are different (requirement #3). There’s no IsDistinctConstraint in shapeless 2.3.0, so we will have to implement it ourselves. We’ll come to that later.

That’s it for the class definition. Let’s move on to the and combinator:

 

Requirement #2 is trivial here. NotContainsConstraint for requirement #4 will have to be implemented by us too.

All right, so we have two HList constraints to implement. Let’s see how it’s done.

Implementing HList constraint

In general, a constraint is implemented as a typeclass, that provides instances for and only for objects, that meet the constraint.

Most of the time it can be done with a technique similar to mathematical induction. It involves two steps:

  1. Define a base case: implicit constraint instance for HNil or an HList of known length like 1 or 2.
    Base case depends on the nature of the constraint and can involve additional constraints for HList element types.
  2. Define the inductive step: implicit function, that describes how new elements can be added to an arbitrary HList, that already meets the constraint.

We will start with NotContainsConstraint. The typeclass definition is quite straightforward:

 

U  is the type that L must not contain to meet the constraint.

Let’s define the base case. Here it’s simple:

HNil doesn’t contain anything.

In general we want constraints to stay same under any circumstances, so it’s usual to define implicit rules right in the typeclass companion object:

 

Seems logical: for any type U we state that HNil doesn’t contain it. Heading over to inductive step, it can be expressed in words this way:

Given an HList that doesn’t contain U, we can add any non-U element to it and get a new HList, that still doesn’t contain U.

Let’s encode it.

 

Here we require an HList L, that doesn’t contain U (is guaranteed by implicit  ev: L NotContainsConstraint U) and a type T, that is not equal to U( ev2: U =:!= T ). Given those evidences, we can state that L :: T doesn’t contain U. We do it by supplying a new typeclass instance  new NotContainsConstraint[T :: H, U] {} .

Some tests:

 

Nice, it works!
I hope it’s transparent here how implicit resolution can or can not find a constraint instance for an HList: we start with HNil base case and go to the list head. If implicits chain is not broken along the way by a duplicate element — we get a constraint instance for the whole list.

Now we’re going to implement IsDistinctConstraint in a similar manner. And our fresh NotContainsConstraint is going to help us here!

Base case is quite simple:

HNil is a distinct list.

 

Inductive step is quite simple too:

If an HList L is distinct and it doesn’t contain type U, than U :: L is a distinct list too.

 

Tests show that everything works as expected:

Wiring everything up together

Now, when we’ve done all the preparation work, it’s time to get our builder to work.
We’ll try it out in a REPL session. Single request case goes first:

 

Everything is ok, we’re getting the mocks, defined in MockResponse. But surprise awaits us, if we try to get multiple resources:

 

There’s no implicit mock response for our HList! We will have to add some implicits into MockResponse companion to help it join our results:

 

After all those constraint tricks this simple typeclass extension should be transparent to you. We basically supply a MockResponse instance for any combination of MockResponses.

Important note: although the problem we’re solving here looks artificial, it is not — in a real world we will have to propagate requested types through all network & parsing machinery, that obtains the result. It is the only way to keep the compile-time safety.
And, similarly to our example, some tools (probably implicit) will be required for joining several results in an HList.

Finally, we get everything working! Notice the result types and how HList allows to select only requested types.

 

All safety requirements are also met — there’s no room for programmer errors:

Shapeless 2.3.1

Shapeless 2.3.1 is coming out soon, and it will contain both constraints we implemented here.

Conclusion

Creating a library or an API is a great responsibility. Providing end-user with a type-safe interface, that doesn’t allow to run and deploy malformed definitions is a high priority aspect.

HList constraints are a great tool in a Scala API developer’s toolbox. In this post we’ve seen them in action, applied to a practical example.

Thanks for reading! See you in future posts 🙂