[TheRealDunkirk]

One of my biggest problems in writing data-heavy web applications is that browsers seem to be really lousy at handling the data. In fact I have 2 applications in this boat. My queries run fast (couple/few hundred ms), but displaying 10's of MB of data, over 10's of thousands of rows of data, has been brutal for a browser.

Trying various standard JS table widgets only made the problem worse, but I've discovered ag-grid (a paid JS table widget) which made this tolerable. It takes about 6 seconds to render my big page of data (which is actually on-par with a WinForms version of the system I wrote, using DevExpress).

It sounds like you have the opposite problem, and I'm wondering how you display "large data sets" from "complex database queries" in sub-second renders in a browser. Incremental loading (like a news feed) won't really work here, because my users need to be able to sort and filter and rearrange the data as a whole. I guess I could "chunk" the data to be able to see it sooner, but it's still going to hold up the browser until it finishes. I've been looking for a better way for a long time.

[jmulho]

You aren't going to like this answer at first, but bear with me. There is another way. When you see the other way done well you may conclude, as I have, that it is the best trade-off given the technology you've chosen (the web). You want to use pagination. That is, your user won't be able to see 10s of thousands of rows at once. They will have to page through them. That doesn't mean your user won't be able to sort and filter and rearrange the data as a whole; but after each action, they will only see the first page of the changed result. If you do this well, every user action will result in a near instantaneous change on screen. You'll need to make every link in the chain as fast as it can be. The key is going to be the database. The first thing you'll need to know is the fastest way to do pagination in the particular database you are using. You need to be able to do something like this: getCustomers(81,160) and get instant results. You'll need database connection pooling. You'll want to send the smallest possible payload over the wire. HTML, XML, and JSON are too verbose. Don't send the word "LastName:" with every row. This will double your payload. Include the column names only one time in the response. When the client gets the payload, slap html on it, and inject it into the DOM (replace a div: <div id="results"></div>). Let the user set their page size (40, 80, 160). If you play with this, you'll find that every device/browser/connection combination will eventually choke if you keep increasing page size (and have a large enough result set), but every device/browser/connection will enjoy beautiful responsiveness at 40 or 80 results per page. When you feel what this is like you wont want to go back to watching a spinner for 10 seconds and then watching your browser hang up every time you scroll. And your user who is using an iPhone 4 with some old version of Safari on a 3g connection really doesn't want to watch a spinner for 30 seconds, then see a blink of results, then see their browser crash. But that is exactly what will happen if you give them 10s of thousands of rows.

[Terr_]

> pagination

Even better is where you clearly separate your use-cases to avoid pagination except when you really need it. "If someone really needs to be able to visit 104th result, they need a different workflow anyway."

It's especially dangerous when you have some "Find Order" UI, and then some executive chimes in with: "My group could really use some totals about the result-set. Since this is already 90% done..."

Eventually you discover the "Find Customer" UI is abominably slow, having sacrificed the ability to find a customer for a bunch of quasi-reporting bells and whistles.

[flukus]

I'd bet the winforms version would be faster without DevExpress. We use the web version at work and it's extremely apparent that their use cases are small shops without a lot of data, they do a lot of things in incredibly inefficient ways that require/encourage encourage n+1 problems and things like server side (database) paging are like pulling teeth. To get a page working with any sort of efficiency you have to fight the framework which takes more time and effort than you save from using it.

For the JS stuff, a lot will depends on how you add the data because things like DOM redraws can be brutal. If you have a thousand items in a list for instance, adding them to the list 1 at a time could trigger the DOM to do whatever it does a thousand times, but if you can add them in one go then the DOM only changes once (with a decent framework). Whatever you're using you have to know how it interacts with the DOM.

I'm not sure how ag-grid works, but I'd be willing to bet that any table built with a DOM like structure will always be orders of magnitude slower (and eat more ram) than a real grid that simply paints.

[noir_lord]

> but displaying 10's of MB of data, over 10's of thousands of rows of data, has been brutal for a browser.

I can display >~14K rows a second with jquery datatables if I load the data from the client side up to ~100K rows then it slows down slightly (on my Thinkpad i7-7700HQ) and that's doing some funky stuff with dynamically created elements mounted with Vue - I was frankly surprised how fast browsers had gotten when you optimise in a certain dimension.

I had to do some slightly funky stuff to do it but I was surprised how little code it actually took, jq.dt is damn impressive for the sheer scope of what it does.

If I use it's ajax functionality I can display several million (but actually however many is currently on screen) in the time it takes the DB to respond.

[crooked-v]

How are you inserting the data into the page?