Separate filter and pagination

I noticed when I create a view with filter-form and pagination that I need to place the pagination between the wpv-filter-start and wpv-filter-end tags otherwise they don't work properly? Is this correct?
If so, this poses a small problem since I want my filters to be at the top of the page and the pagination at the bottom.