Overview

Currently there is a problem in the indexing structure, because the heap space needed to keep the indexing tree is too high. Usually the indexing tree occupies 10%-25% of the complete space reserved for the buffer based on the structure of data received from agent. There are two main factors that influence the size of the indexing tree:

1. The average amount of children in the invocations - if the buffer is filled with smaller invocations (up to 100 children) then indexing size will tent to be closer to the 10% occupancy line. This is because then different data objects are equally distributed in the indexing tree (thus throughout several maps). When we have invocation with many children, then the leafs for SQLs and especially timers then to be much bigger, thus occupying more space due to the map expansion policies (storing 600K elements in the map requires array of 1M entries).
2. Time interval data in buffer is created - when we have a high load and agent sends lot of data in short interval we have a bigger indexing structure, because our time-stamp indexer is indexing in intervals of 15 minutes. Meaning that if all data in the buffer is created in same 15-min interval all data will fail into same branch, which would effectively mean that number of maps for storing data will be smaller.

Thus, the conclusion is that the main problem we have is the amount of space maps where we index data occupy in memory. The problems is bigger if the amount of maps is small or we have one or more maps that have many elements (500K+).

The Java maps are known to be non-storage friendly. It's assumed that often 80%+ of the map size is used for maintenance, where only 20% are real data stored in the map. Note that we use ConcurrentHashMap.

Dealing with Map sizes