Relates to the 

Problem

Some of our views like Timer Data View or SQL Show All View display aggregated data. In most cases the amount of data that is aggregated in those views is very high since the views display all instances of one data type. When data is read from storage, then a lot of time is spend to first de-serialize data from disk, which makes these queries to last longer. In addition any operation that user executes on the view, like change of the time resolution, will fire the update method which will in fact again de-serialize same data again and perform aggregation again.

Solution

Idea is to cache the results of all queries executed on the storage. This would mean that when the user queries, for example, for All Timer Data result of this query is cached and later on can be directly accessed which will be very fast. Once cached query must be accessible for all users of the same storage. 

In addition to this it would be reasonable to automatically cache all the standard view that are very likely to be access very often. This can be executed upon the storage finalization. 

Realization

We will define the new type of file, for example .cached. Each .cached file will hold results of exactly one query. To decrease the chance of too many files in a storage folder, we can additionally create [storage_root]/cached folder where those files will reside.

Each cached file can be named by the hash code of the index query used to create a result set that will be cached (to the hash code we will most likely need to include also the aggregation used). This would mean that we don't need to keep any track additionally of cached data. Plus, UI can directly search for the file on the CMR or locally and know right away if the cached data exists.

The proposal would be that CMR still does no de-serialization of data. This means that we will create a service like:

StorageService.cacheDataForStorage(StorageData storage, Collection<DefaultData> resultSet, int hash)

The CMR can then simply write the given collection to disk for the given storage and hash which should be very fast. The query execution and data de-serialization / aggregation will be again performed on the UI. The only problem with this approach is how will we cache the standard views after the finalization, but we can handle this in another way (for example with our processors).

The following diagram describes what will be the process on the UI when a query is executed:

In addition to this, it is necessary to check if there is any new cached file available on the CMR for the storage that is fully downloaded when the UI start, but that should not be a problem.