StatisticsCollector Extension
The StatisticsCollector Extension is an FNZ Studio extension that collects statistics relevant for testing or tuning an FNZ Studio installation. Data is collected in three different manners:
- A thread running every 3 seconds.
- A job running every hour, starting at minute 27 and 30 seconds.
- A listener on Garbage Collection events.
All statistics are saved in .CSV files that are stored in your data/logs/statistics folder.
Note that statistic values that are not calculated are saved as zero values in `.CSV` files, and are shown as empty strings (i.e. empty cells) in FNZ Studio Composition statistics tables.
Note that some of this data, such as I/O Statistics or CPU Statistics among others, is also accessible from FNZ Studio Composition. However, statistics in FNZ Studio Composition are updated in real time, while the StatisticsCollector Extension collects a series of files that allow you to have create a statistic history that can be used for troubleshooting.
Statistics are collected and stored per cluster node and fall into three major categories:
Network Statistics
For Network Statistics, refer also to:
- CPU statistics and filter by Work Unit Type 'Tasks'. This allows to investigate how much network activity was caused by distributed tasks.
- Node statistics and look at entry 'Hazelcast. Write Behind Queue Size'. This allows to see how much data still has to be sent to Cluster Storage in the write-behind queue.
Network Statistics comprise (see sections below):
- Distributed Maps Statistics
- Hazelcast Operations Statistics
- HTTP Request Statistics
Distributed Maps Statistics
The Distributed Maps statistics collect information about all the distributed maps and provide information on their respective states.
| Measure name | Unit | Description |
|---|---|---|
| Map Name | Human-readable name of the distributed map | |
| Hazelcast Name | Technical name of the map in Hazelcast | |
| Owned Entries (Count) | Number of objects stored in this distributed map on the local node | |
| Owned Entries (Size) | bytes | Amount of memory used by objects in this distributed map on the local node |
| Backup Entries (Count) | Number of backup objects in this distributed map on the local node | |
| Backup Entries (Size) | bytes | Amount of memory used by backup objects in this distributed map on the local node |
| Dirty Entries (Count) | Number of objects that have been modified in memory owned by the local node and that need to be persisted | |
| Locked Entries (Count) | Number of currently locked, locally-owned keys | |
| Near Cache Entries (Count) | Number of objects in the near cache on the local node of this distributed map | |
| Near Cache Hit Rate | The percentage of near-cache hits (of all near-cache access operations) for the listed map on the listed node | |
| In parentheses, the numbers of near-cache hits and near-cache misses |
Hazelcast Operations Statistics
Hazelcast Operations Statistics analyze the quantity of network traffic caused by operations on distributed maps by showing how often distributed maps have been accessed. Such access to distributed maps also affects the network.
| Measure name | Unit | Description |
|---|---|---|
| Map Name | Human-readable name of the distributed map | |
| Hazelcast Name | Technical name of the map in Hazelcast | |
| sum | The total number of Put, Get, Remove, and other operations | |
| puts | The number of Put operations | |
| gets | The number of Get operations | |
| removes | The number of Remove operations | |
| otherOperations | The total number of other operations (e.g. Map Size operation, Contains operation, etc.) | |
| events | The number of Events received (e.g. Near Cache Invalidation, Partition Lost, etc.) | |
| hits | The number of hits (reads) of the locally owned entries | |
| avgPutLatency | ms | The average latency of Put operations |
| avgGetLatency | ms | The average latency of Get operations |
| avgRemoveLatency | ms | The average latency of Remove operations |
HTTP Request Statistics
HTTP Request Statistics show four smoothing moving averages about the response time of the HTTP requests for the FNZ Studio Runtime. The four moving averages take into consideration 100, 200, 400 and 800 data points, respectively.
This feature is enabled by default and it can be disabled by setting the following property to false:
com.nm.statistics.collector.http.request.stats.enabled = false
| Measure name | Unit | Description |
|---|---|---|
| AverageServerResponseTime100 | ms | The smoothing moving average of the server response time considering the latest 100 requests |
| AverageServerResponseTime200 | ms | The smoothing moving average of the server response time considering the latest 200 requests |
| AverageServerResponseTime400 | ms | The smoothing moving average of the server response time considering the latest 400 requests |
| AverageServerResponseTime800 | ms | The smoothing moving average of the server response time considering the latest 800 requests |
JVM Statistics
JVM Statistics refer to the JVM where FNZ Studio is running.
CPU Usage Statistics
The CPU Usage Statistic collector calculates how much CPU was used by the JVM since the collector was last called. The CPU usage is collected every 3 seconds and it lists the CPU usage in percentage of available CPU.
Memory Usage Statistics
The Memory Usage Statistics collector retrieves all the information related to memory in the Java Virtual Machine (JVM).
| Measure name | Unit | Description |
|---|---|---|
| max | bytes | The maximum amount of memory that the Java virtual machine will attempt to use |
| total | bytes | The total amount of memory in the Java virtual machine |
| free | bytes | The amount of free memory in the Java Virtual Machine |
Such statistics are collected in a file named as follows: memory-usage-stats_[environment]_[date].csv (e.g memory-usage-stats_dev_2018-08-28.csv).
Memory Pool Statistics
In the context of Memory Usage Statistics, specific information is also collected for each Memory Pool (in bytes).
| Measure name | Unit | Description |
|---|---|---|
| name | Memory Pool name | |
| init | bytes | The initial amount of memory that the Java virtual machine requests from the operating system for memory management during startup |
| used | bytes | Amount of memory currently used |
| committed | bytes | Amount of memory that is guaranteed to be available for use by the Java virtual machine |
| max | bytes | Maximum amount of memory that can be used for memory management |
Java Garbage Collector Statistics
Memory Usage statistics also collect information about the Java Garbage collectors.
| Measure name | Unit | Description |
|---|---|---|
| name | Garbage Collector name | |
| collectionTime | ms | Approximate accumulated elapsed time for the collection |
| collectionCount | Total number of collections that have occurred |
Memory Allocation Statistics
Memory Allocation Statistics collect information about the memory allocated by Java Threads.
This feature is disabled by default and is enabled by setting the following property to true:
com.nm.statistics.collector.thread.memory.allocation.enabled
| Measure name | Unit | Description |
|---|---|---|
| ThreadId | The id of the thread that allocated memory | |
| ThreadName | The name of the thread that allocated memory | |
| AllocatedMemoryDiff | MB | Amount of memory that has been allocated by the current thread since the statistics were last collected. |
| TotalAllocatedMemory | MB | Total amount of memory that has been allocated by the current thread. |
Memory Holders Statistics
Memory Holders Statistics collect information about the areas in which data is stored for a long period of time.
Consider the nm.memory.holders.max.allowed.computing.time.ms configuration property (default value: 60000). If set too low, errors are logged as memory statistics cannot be computed.
| Measure name | Unit | Description |
|---|---|---|
| UpdateTime | The time at which the Memory Holder statistics were computed | |
| MemoryHolderName | The name of the Memory Holder | |
| UsedMemory | MB | The amount of memory currently held by the current Memory Holder |
| NumObjects | The number of Java objects holding memory | |
| TimeToCompute | ms | The time spent to compute the measurement for the current Memory Holder |
Garbage Collection Statistics
Every time a Garbage Collection is executed, the following details are collected:
| Measure name | Unit | Description |
|---|---|---|
| GcAction | Action performed by the garbage collector | |
| GcCause | Cause of the garbage collection | |
| GcName | Name of the garbage collector used to perform the collection | |
| Duration | ms | Elapsed time of this Garbage Collection |
| MemoryUsageBeforeGc | bytes | Memory usage of all memory pools at the beginning of this Garbage Collection |
| MemoryUsageAfterGc | bytes | Memory usage of all memory pools at the end of this Garbage Collection |
Such statistics are collected in a file named as follows: garbage-collection_[environment]_[date].csv (e.g. garbage-collection_dev_2018-08-28.csv).
Memory Pool Statistics before/after Garbage Collection
The state before and after Garbage Collection is also logged for each separate Memory Pool. The data collected is the same as in Memory Pool Statistics.
FNZ Studio Statistics
This group of statistics measures values that are specific to the FNZ Studio Platform.
CPU Statistics
CPU Statistics show the number of executions of distributed tasks in the FNZ Studio Cluster. They collect data about the Work Units running in FNZ Studio, which are of the following different types:
- HttpRequests
- ProcessJobs
- Runnables
- ScheduledJobs
- Servlets
- Tasks
- Threads
These statistics are enabled by default, as the following configuration parameter is set to true: nm.statistics.cpu.enabled. You need to set it to false to disable statistic collection.
For every Work Unit the extension collects the measurements in the table and for every measurement (apart from timestamp, contextType, context and lastMeasurementTime) count, min, max, avg and sum.
| Measure name | Unit | Description |
|---|---|---|
| contextType | The Work Unit type | |
| context | The Work Unit name | |
| lastMeasurementTime | The time at which the last measurement was taken | |
| realTime | ms | Overall measurement duration on this node |
| cpuTime | ms | CPU time for all operations on this node |
| userTime | ms | User-level CPU time for all operations on this node |
| waitedTime | ms | Waiting time (thread in states WAITING or TIMED_WAITING) for all operations on this node |
| blockedTime | ms | Blocked time (thread in state BLOCKED) for all operations on this node |
Such statistics are collected in a file named as follows: cpu-stats_[environment]_[date].csv (e.g. cpu-stats_dev_2018-08-28.csv).
Function Call Statistics
Function Calls Statistics collect data on FNZ Studio Script Functions. For each function, data is collected on values such as the total, average, minimum and maximum time to call a function (in milliseconds). Memory allocation data is also collected for single calls (and for the total number of calls).
These statistics are enabled by default, as the following configuration parameter is set to `true`: `nm.functioncallstatistics.enabled`. You need to set it to `false` to disable statistic collection.
| Measure name | Unit | Description |
|---|---|---|
| FunctionName | Function name | |
| Source | The place where the source code of the Function resides | |
| Invocations | The total number of invocations | |
| TotalTime | ms | The total time to execute calls |
| AverageTime | ms | The average time to execute a single call |
| MinTime | ms | The minimum time to execute a single call |
| MaxTime | ms | The maximum time to execute a single call |
| TotalMemoryAllocation | MB | The total memory allocated to execute calls |
| AverageMemoryAllocation | MB | The average memory allocated to execute a single call |
| MinMemoryAllocation | MB | The minimum memory allocated to execute a single call |
| MaxMemoryAllocation | MB | The maximum memory allocated to execute a single call |
Such statistics are collected in a file named as follows: function-call-stats_[environment]_[date].csv (e.g. function-call-stats_dev_2018-08-28.csv).
Serialization Statistics
Serialization Statistics show information about the serialization of the data stored in the distributed maps.
| Measure name | Unit | Description |
|---|---|---|
| MapName | The map for which statistics are displayed | |
| Persistent | Indicates whether data is stored persistently in this map | |
| CountWriteObject | The number of write operations performed for Java Serialization | |
| CountReadObject | The number of read operations performed for Java Serialization | |
| CountWriteData | The number of write operations initiated by Hazelcast | |
| CountReadData | The number of read operations initiated by Hazelcast | |
| AvgTimeWriteData | ms | The average time per operation spent on data writing |
| RollingAvgTimeWriteData | ms | The average time per operation spent for the 1000 most recent data write operations |
| AvgTimeReadData | ms | The average time per operation spent on data reading |
| RollingAvgTimeReadData | ms | The average time per operation spent for the 1000 most recent data read operations |
| MaxSizeUncompressed | bytes | The maximum uncompressed size of any data item for this map and node |
| AvgSizeUncompressed | bytes | The average uncompressed size of all data items compressed for this map and node |
| AvgSizeCompressed | bytes | The average compressed size of all data items compressed for this map and node |
| AvgCompressionRatio | The overall compression ratio (sum sizes compressed / sum sizes uncompressed) of all compressed data items | |
| AvgTimeCompressing | ms | The average time spent for compressing data |
| AvgTimeDecompressing | ms | The average time spent for decompressing data |
| CountClone | The number of clone operations for the thread-local cache | |
| AvgTimeClone | ms | The average time spent for cloning |
Such statistics are collected in a file named as follows: serialization-stats_[environment]_[date].csv (e.g. serialization-stats_dev_2018-08-28.csv).
I/O Statistics
These statistics are about the I/O operations on the persistent maps in FNZ Studio.
These statistics are enabled by default, as the following configuration parameter is set to true: nm.statistics.io.enabled. You need to set it to false to disable statistic collection.
For every measurement in the table (apart from contextType, context and lastMeasurementTime) the extension collects count, min, max, avg and sum.
| Measure name | Unit | Description |
|---|---|---|
| contextType | The type of I/O performed (e.g. Maps) | |
| context | The item for which I/O is performed (e.g. a map name) | |
| lastMeasurementTime | The time when the last measurement was taken | |
| Lock | ms | The time to acquire a lock on the resource for writing |
| Time Read | ms | The overall time for reading |
| Time Read Bytes | ms | The time needed to transfer data while reading |
| Time Write | ms | The overall time for writing |
| Time Write Bytes | ms | The time needed to transfer data while writing |
| Time Other | ms | The overall time for other operations |
| Time Other Bytes | ms | The time needed to transfer data during other operations |
| Num Bytes Read | bytes | Number of bytes transfered while reading |
| Num Bytes Write | bytes | Number of bytes transfered while writing |
| Num Bytes Other | bytes | Number of bytes transfered during other operations |
Such statistics are collected in a file named as follows: io-stats_[environment]_[date].csv (e.g. io-stats_dev_2018-08-28.csv).
Job Statistics
Job Statistics show information about the execution of scheduled jobs running in FNZ Studio. These statistics show information about the executions of the scheduled jobs running in FNZ Studio.
| Measure name | Unit | Description |
|---|---|---|
| JobName | The name of the job | |
| JobGroup | The group of the job | |
| ExecutionStrategy | Execution strategy of the job (on one node only or on all nodes of the cluster) | |
| Status | The status of the trigger associated with the job (e.g. NORMAL, PAUSED, ERROR etc.) | |
| NextExecution | The time at which the job is scheduled to run the next time | |
| LastExecution | The time at which the job was last executed | |
| Runs | The number of times the job has run | |
| Errors | The number of times the job has run with an error | |
| MinRunTime | ms | The minimum amount of time the job has taken to run |
| AvgRunTime | ms | The average amount of time for the job to run |
| MaxRunTime | ms | The total amount of time the job has been running |
| TotalMemoryAllocation | MB | The total memory allocated during job executions |
| AvgMemoryAllocation | MB | The average memory allocated during job executions |
| MinMemoryAllocation | MB | The minimum memory allocated during job executions |
| MaxMemoryAllocation | MB | The maximum memory allocated during job executions |
Such statistics are collected in a file named as follows: job-stats_[environment]_[date].csv (e.g. job-stats_dev_2018-08-28.csv).
Screen Statistics
Screen Statistics collect information about the active screens. Every screen is identified by the concatenation of the package name, screen ID and screen version UUID.
These statistics are enabled by default, as the following configuration parameter is set to true: nm.screenstatistics.enabled. You need to set it to false to disable statistic collection.
Such statistics are collected in a file named as follows: screen-stats_[environment]_[date].csv (e.g. screen-stats_dev_2018-08-28.csv).
| Measure name | Unit | Description |
|---|---|---|
| ScreenId | The id of the screen for which statistics are collected | |
| VersionUuid | The version UUID of the screen | |
| RenderCount | How many times the screen was rendered | |
| RenderTimeAvg | ms | Average time to render the screen |
| UpdateCount | How many times the update phase was executed for the current screen | |
| UpdateTimeAvg | ms | Average time for the update phase |
| ValidateCount | How many times the validation phase was executed for the current screen | |
| ValidateTimeAvg | ms | Average time for the validation phase |
| ProcessCount | How many times the process phase was executed for the current screen | |
| ProcessTimeAvg | ms | Average time for the process phase |
| TotalMemoryAllocation | MB | The total memory allocated in all phases |
| AvgMemoryAllocation | MB | The average memory allocated per phase |
| MinMemoryAllocation | MB | The minimum memory allocated in any phase |
| MaxMemoryAllocation | MB | The maximum memory allocated in any phase |
Process Instance Indexing Statistics
Process Instance Indexing statistics collect information about indexes related to Process Instances.
| Measure name | Description |
|---|---|
| Index | Index definition or 'All Indexes' text if the column represents an aggregation |
| IndexSize | Size of the current index. |
| IndexDataTimeMs | Total time spent on retrieving needed data from a Process Instance and indexing it |
| LookupIndexData | Lookup statistics about the current index |
| LookupIndexDataStatsMap | Map of lookup statistics for all existing indexes |
| LookupIndexDataCounter | Number of successful local index lookups when the requested indexed data was available |
| LookupIndexDataNoDataCounter | Number of local index lookups when the requested data was empty |
| LookupIndexDataNoDataRing | List containing the last 100 local entries when there was an index lookup but the requested data was empty |
| LookupIndexDataWithoutIndexCounter | Number of local index lookups when no index was available to be queried |
| LookupIndexDataSizeWithoutIndexCounter | Number of local index lookups when no index was available to be queried |
| Similar to the LookupIndexDataWithoutIndexCounter. However, this increases when an index is not active. | |
| RetrieveSuperSetResultTotalTimeMs | Total time spent on retrieving the result of a distributed query using indexes |
| RetrieveSuperSetResultCounter | Number of successful queries executed using indexes |
| RetrieveSuperSetResultNoDataCounter | Number of distributed queries executed when the query could not be resolved using indexes |
Process Instance Indexing Statistics are collected in a file named as follows: piindexing-stats_[environment]_[date].csv (e.g. piindexing-stats_dev_2018-08-28.csv).
Node Statistics
Node Statistics analyze the quantity of data that has to be sent to Cluster Storage in the write-behind queue. The data collected on the Node Statistics table can be grouped into four main categories:
-
Generic Node Information
-
Generic Machine Statistics
-
Generic Hazelcast Statistics
-
Hazelcast Maps Statistics
Consider that Node Statistics contain some of the statistics already contained in Distributed Maps statistics and Hazelcast Operations statistics.
See also Node Inspector in FNZ Studio.
Generic Node Information
Generic Node information is a series of static data such as Node Name, Node UUID and so on.
Generic Machine Statistics
Generic Machine Statistics are relative to the machine where FNZ Studio is running and collect data on memory and CPU, such as free and used memory and so on.
Generic Hazelcast Statistics
The Generic Hazelcast Statistics are data collected at node level (not specific to any map). Some of the data collected are:
- Write-behind queue size: shows how much data has still to be persisted in Cluster Storage (which also affects the Network).
Hazelcast Maps Statistics
The Hazelcast Maps Statistics are collected for each Hazelcast Map and, therefore, are repeated for each of them. For example, the ownedEntryCount measurement collected for the valueStoresMap is ownedEntryCountValueStoresMap.
Extension Functions
The StatisticsCollector Extension also exposes three functions to manage the collection of statistics:
- StartCollectingStatistics
- StopCollectingStatistics
- ResetStatistics
StartCollectingStatistics
The StartCollectingStatistics function can be used to enable the job and the thread in charge of collecting the all the statistics described above.
StopCollectingStatistics
The StopCollectingStatistics function can be used to disable the job and the thread in charge of collecting the all the statistics described above.
ResetStatistics
The ResetStatistics function deletes the file where the statistics are collected and resets all the cumulative statistics (e.g. Screen Statistics). In this case, the job is not stopped and the collection is not interrupted.