Aggregate Timers and SQLs also in the buffer

Migrated from NovaTec Jira. Original ticket: INSPECTIT-956

Stefan and I noticed in the load tests of CMR that the loading of all timers can be quite slow.. Of course when the buffer is something like 6GB big and is full, then most likely half of the data are timers.. This means that lets say 3GBs of timer data has to be aggregated and returned to the user when request come.. That s probably a work on almost 10 million objects (TimerData size in memory is around 350 bytes).. Thinking of what can be improved I came to several solutions:

1. With fork & join (INSPECTIT-585) it would be faster since now it s done in one thread

2. We always first find objects in the buffer, put them to list and then aggregate.. When working on so many objects, I have feeling that just creating such a huge list is slow.. Maybe we could have something like queryAndAggregate for the indexing tree where we don't need to create such list in first place

3. Aggregate timers and SQLs as they come - basically the idea of the ticket.. There is no reason we don't do this.. At this point every timer, no matter if coming stand alone or in the invocation is also put into buffer/indexed.. While the ones coming from the agent are most likely aggregated, the ones from invocations are not.. And there is no reason not to have them aggregated in the buffer.. We would not lose anything, since we already apply the similar tactics for the storage.. We don't have a single use case where we need to locate exactly that timer.. Same is for the SQLs.. The advantages on the other side are huge:

   • Smaller buffer occupation, since we dont create new buffer element for every timer

   • Smaller indexing tree cause we don't index so many objects

   • Faster or lets say 95%+ faster querying of all timer data and all SQL statements

Of course there are some problems, among which I find When to aggregate? the most important one.. Currently we are very fast in adding elements to the buffer and we want to stay like that.. If we aggregate before inserted into buffer then we will lose the ability to see timers in the UI as they arrive to the CMR.. On the other hand if we aggregate when adding to buffer it can slow down inserting considerably..

We need to discuss this..