On higher kinded and existential types,...no stereos

So I was recently bit by a type issue with existential types until someone forced me to start thinking again. Let's start with a good example of what I was trying to do:

def foo[A, M[T] <: Traversable[T]](m : M[A]) : A = m.head
def foo2[A, M[_] <: Traversable[_]](m : M[A]) : A = m.head 

foo2 will fail to compile with a type error. What gives! Aren't these the same? Well, no they are not. Let's see if we can reason through the type signature we've written. For foo, A is any type and M is a higher kinded type. M has a type parameter T. M is also some kind of subclass of Traversable. Also, the type parameter T on M is *the same* type parameter used for the Traversable parent (i.e. defined trait/class M[T] extends Traversable[T]). Now for foo2, it's a bit different. A is any type. M is some higher kinded type (takes a type parameter) but in this case we don't acre which. M is also a subclass of Traversable, and once again we don't care about the type parameter to Traversable. The big difference here is that we don't care about the type parameters to M or Traversable, meaning they could be different types. Now when we define parameter m, we're looking at M[A] <: Traversable[_] or, in other words, we still know nothing about the type in the traversable. Let's try to prove our intuition about foo2 by capturing the manifest of type parameter A and see if it has any correlation to the type in the traversable:

def foo[A : Manifest, M[_] <: Traversable[_]](m : M[A]) : A = {
  println(implicitly[Manifest[A]])
  m.head.asInstanceOf[A]
}

class Temp[X] extends Traversable[Int] {
  override def foreach[U](f : Int => U) : Unit = f(5); ()
}

foo(new Temp[String]) // Prints java.lang.String -> NOT Int

As you can see, the compiler has capture the String type parameter, not the Int as we really wanted to capture in this method. Let's see if we can correct our mistake:

def foo[A : Manifest, M[X] <: Traversable[X]](m : M[A]) : A = {
  println(implicitly[Manifest[A]])
  m.head.asInstanceOf[A]
}

class Temp[X] extends Traversable[Int] {
  override def foreach[U](f : Int => U) : Unit = f(5); ()
}
foo(new Temp[String]) //THis line errors ->
:9: error: inferred type arguments [String,Temp] do not conform to method foo's type parameter bounds [A,M[X] <: Traversable[X]]
       foo(x)

Ok, so I've seen a lot of code with this M[X] <: Traversable[X]... This should work right? Well, let's think it through: M[X] <: Traversable[X] implies that the type M is higher-kinded (takes a type parameter). It also implies that it extends Traversable, and finally that The type parameter to Traversable should be *the same* as the type parameter to M. It's this last piece we've messed up. Our Temp class does take a type parameter, but it has nothing to do with the type parameter in its traversable parent. This distinction is important, especially with collections. M[X] <: Traversable[X] can capture an awful lot of collection types, however it will miss any collection where the type is known... In other words, if I defined something like

IOByteStream extends Stream[Byte]

then

M[X] <: Stream[X]

will fail to capture this type! Let's take one more attempt, this time trying to capture a subclass of Traversable where the Traversable's type parameter is captured in a manifest.

def foo[A : Manifest, M <: Traversable[A]](m : M) : A = {
  println(implicitly[Manifest[A]])
  m.head
}

class Temp[X] extends Traversable[Int] {
  override def foreach[U](f : Int => U) : Unit = f(5); ()
}

foo[Int,Temp[String]](new Temp[String]) //We need to use our types here

Well the method compiles, but we have the issue that the typer cannot infer our types! At least we managed to express what we wanted... or did we? Our method does not care about the actual subtype of Traversable, and does not need to capture it. If you look, the return type is A, not M. Therefore we should be able to take in just a Traversable, like so:

def foo[A : Manifest](m : Traversable[A]) : A = {
  println(implicitly[Manifest[A]])
  m.head
}

class Temp[X] extends Traversable[Int] {
  override def foreach[U](f : Int => U) : Unit = f(5); ()
}

foo(new Temp[String])  //Prints Int and returns 5

I purposely made me initial example odd so I could make this point: When using the type system, try to keep things as simple as possible. Expand your definitions as needed, or when assumptions are proven invalid. Here, we had no need for the actual subtype of Traversable and yet we were focused on getting the signature right to capture a subtype. This is important for developers to learn, and even *more* important for library authors. When using type parameters, make sure to reason through the signatures to be sure they capture what is desired - no more or no less. Just for kicks, let's try a method were we *do* care about the original subtype, and wish to preserve it, yet also need to use the Traversable's type parameter. What can we do? Let's attempt to capture both type parameters:

def foo[A : Manifest, B, M[X] <: Traversable[A]](m : M[B]) : M[B] = {
  println(implicitly[Manifest[A]])
  m
}

class Temp[X] extends Traversable[Int] {
  override def foreach[U](f : Int => U) : Unit = f(5); ()
}

foo[Int,String, Temp](new Temp[String]) //We need to use our types here again

You can see we've captured both type parameters in A and B, however once again the Scala type inferencer cannot distinguish our meaning without some help. Is there anything we can do here?

Unfortunately... I don't actually know of a way to help the type inferencer here. I also realized that in my situation, that kind of a requirement was completely unnecessary.... If anyone has any ideas for the academic excercise, please let me know.

Since the new collections API starts throwing more type issues in the face of every day developers, expect a new series from me: "Enough Type magic to work with Scala Collections". As I figure out ideal ways to deal with new type signatures required in abstract methods, I'll try to post findings.