How To Scroll And View Millions Of Records

Check the documentation for whatever other back-end you are using - there should be an equivalent methodology to allow you to implement this functionality. Now, having pointed you to this common method of extracting a limited view of data from a structured dataset, I must caution you not to expect to rely on it every time. Always question the use case, the volume of data in question, the index fields you have to work with, etc. It may be the case that in very very large sets of data a map/reduce pattern may work well, or it may be that you have data pre-loaded for particular presentations of data that you can serve up, or it may be that the offset/limit slows down with a particular volume of data, etc.... as usual, while this is a general 'how-to', the specific answer in your case may well be 'it depends'. The basic concepts and patterns remain the same, the objective is to identify how to apply the best in your situation. Here is an interesting read on how to approach things with an older structured database/RDBMS if it does not natively support the offset concept.

Dynamic pagination

Before we move off simple tables of data, it is worth pointing out how to handle dynamic pagination. Where you are dealing with a reasonable amount of data, you would expect to see a pagination control something like this. When you start adding data to it, what happens? As you can see, the middle page numbers start to pad out ... this can get very messy very quickly. A good way to handle this is to visually shift the range of where you are indicating the user is in the view... here's how Google do it for example.. note the results start now at 9 and go to 18, and they tell us its page 14 of gazillions.... (they also give themselves a pat on the back for the speed of the result, and that's fine :)) Even Google does not allow you to search everything - sometimes they say - nope, you need to be more specific in your query please... take this message for example when I clicked past the 1000th page (the things I do for research!). (btw, the red text is mine... in jest!) For further reading on pagination, have a look at the UI patterns pagination page, and if you want a good deep dive, kick your feet back and read Janko at Warp Speed on UI patterns for tables (it's good!).

Four Ways to Optimize Paginated Displays

As the last brief word on optimizing pagination, here's a summary of some ideas by Bill Karwin on the stack.

Tables and pagination are all very good, but what about this infinite scrolling business, how does that relate to the view portal concept? ... actually, it could be considered the same thing on the backend with a few tricks on the frontend, mostly concerned with pre and post caching of data the user is currently viewing. Let's look at how that works.

Core Concept 2 - caching

Infinite, or continuous scrolling, is effectively the same thing as pagination, except the user does not have to explicitly click to move to the next batch of data - we assume the intention for them. This can be triggered based on any number of triggers that suits your situation ... for example, a scroll-bar hits its limit, the user scrolls tot he bottommost part of a page, the last piece of visible data on a screen is displayed into view... the trigger point, is simply down to your design. The only real issue with infinite scrolling, however, is how to optimize the experience for the user. How do we make it so that it feels seamless and fast? ... the answer lies in caching. At a very basic level, we anticipate what the user wants to do, and do it before they ask. There are numerous solutions out there that implement this for you as part of various presentation frameworks on different platforms, but its good to know the basic concepts of what might be happening under the hood. Let's look at one example approach. Step 1 - default load

When we load up our webpage/app in the first place, we have a single default view ready for the user. This is the 'current view' shown below. In our app we know that the next thing the user will do is use their thumb/mouse to scroll the next piece of data into view. In this case, we already have our 'next view' loaded on the device/page, and are ready to show it immediately, there is no delay or distraction to the user experience. This 'next view' is cached by default, and ready to use. From a practical point of view, this means that when we set the page up, we load not only the current view, but also the 'next view' - the next view is not visible, but cached off-screen, and ready to use on-demand. Step 2 - post and pre-cache

When the user scrolls the 'next view' to become the 'current view', the current view moves away to be the 'previous view'. This new 'previous view' is cached invisible off-screen in case the user decides to scroll back up again. Immediately the 'next view' starts scrolling into place, we go straight off and fetch the next, 'next view' to replace it (it's not quite inception, but it does come to mind!). Step 3 - further management Deciding what to do next is the domain of you the system designer. If you are designing this yourself from scratch you need to decide the next steps, or, if you are using a pre-built system, you may have some configuration options you can choose to help flavor the size of cache, etc. On a simplistic level, you may decide that as soon as you have moved two steps / scrolled away from an original 'current view', that realistically the user is not going to scroll back, and you might make the decision to delete that 'previous-previous-view'. Regardless of how your strategy works, the concept of pre-loading/caching data before you need it remains solid. Data view In case it helps, here is another view of what is happening as described above, but showing how the sample record data is being traversed.... You can read more about the infinite/continuous scrolling pattern on UI patterns.

Summary

That's the very basic core concept - obviously, there is a lot more that can go on and this is the tip of the usual iceberg, however, hopefully, it gives some a guide in the right direction.