One-To-Many Relationship with additional custom field

I have a previous post you can see here: http://wp-types.com/forums/topic/one-to-many-relationship-problem/

I was able to setup the relationship I needed using the custom taxonomy to related two custom post types to each other but now I need to add something and I can't find a way to do it.

I've got Courses and Video Series. The same Video Series can belong to more than one Course. Each Course contains many Video Series. This all works.

But now within each Course I want to be able to have a sort order for the Video Series. This sort order should be specific to that Course. So for example:

Course 1
-Video Series 1
-Video Series 2
-Video Series 3

Course 2
-Videos Series 2
-Video Series 3
-Video Series 1

In this example, Video Series 1 should have a sort order value of 1 for Course 1 but a sort order value of 3 for Course 2.

Is there any way to obtain this without having to have a separate view for each Course?

Thanks.