ProcessAuditTrail Extension

Some features described in this article may be subject to licensing and/or separate surcharge. Please talk to your FNZ contact for more information.

Introduction

The ProcessAuditTrail extension tracks a running process' information flow. It logs all actions about its state, and stores them into a relational database. Other data, such as instance or token attributes and custom messages, can be logged as well. Use this stored information for your own purposes, such as keeping an audit trail (i.e. what happened when and where), or conducting business intelligence (e.g. the time it usually takes for process XYZ to complete for customers in Germany).

Note: This extension does not track changes in the process' business data. To track data changes, use the DataAuditTrail extension.

This document is intended for people who have to install and configure this extension and/or prepare the database. It is also suitable for business users who want to see what data can be collected or what can be further analyzed.

How It Works

Consider this example:

Example_Process.png

Figure 1: Example process

Figure 1 shows an example process. A particular process instance flow is represented by the solid red line and ends following the dashed line. The process instance triggered many events in the Appway process engine. The ProcessAuditTrail extension collects these events, transforms them into log entries and stores them into the database. The level of detail of collected and stored data is configurable (see Data Collection Properties).

We illustrate how the ProcessAuditTrail extension collects and stores information based on the flow marked by the dashed line in Figure 1. The user is on the "Swim 2” Screen Task and clicks the "Repeat" button. The process instance proceeds to the "Repeat B" Script and then gets back to the Screen "Swim 2".

Figure 2 illustrates the processing of a particular user action. The user clicks on the repeat button, which triggers an HTTP request sent to Appway core, which in turn, invokes the process engine.

Example_process_details.png

Figure 2: Example process details

While the Process Engine does its work, multiple events are generated. These events are collected by the ProcessAuditTrail extension and then converted into commands by the Producer component. Commands are data structures which describe the raised events. The Producer adds the commands to a queue, and the commands are later consumed asynchronously. The consumer threads of the Consumer component translate the commands into database statements and execute them. The collected data is available in technical database tables but also as human-readable log entries (see Figure 15: Entity Relationship Diagram).

With our example dashed process flow and the extension's default configuration, the following commands are generated:

  • Token End End event of Screen "Swim 2". It updates the Screen's token in database table "token" and sets its state to completed.
  • Audit Trail Log A human-readable log message which says that the Screen is now complete. It inserts a new message record into the database table "audit_log".
  • Token Start Start event of Script "Repeat B". It inserts the Script's token into database table "token" with state= started.
  • Audit Trail Log A human-readable log message which states that the Script has started. It inserts a new message record into the database table "audit_log".
  • Token End End event of Script "Repeat B". It updates the Script's token status in database table "token" to completed.
  • Audit Trail Log A human-readable log message which says that the Script is now complete. It inserts a new message record into the database table "audit_log".
  • Token Start Start event of Screen "Swim 2". It inserts the Screen's token into database table "token" and sets its status to started.
  • Audit Trail Log A human-readable log message which says that the Screen has now started. It inserts a new message record into the database table "audit_log".
  • Token User User assigned event for Screen "Swim 2". It inserts the current user into the database table "token_user".

Collecting Events

The extension attaches to the Process Listener, which is an infrastructure that listens to events generated by the Appway process engine. These events are prepared as commands and stored into a queue. They contain a human-readable text message which is defined in various Labels. As such, the process is detached and continues working while the data is consumed from the queue and stored in the database (producer/consumer).

You can customize the Labels used with any text and translation. A list of all Labels used is presented later in this document.

Collecting_events.png

Figure 3: Collecting events

It doesn't matter on which node events for a certain process instance are generated. Since the queue is distributed, the changes are visible on all nodes.

Storing Events

A consumer thread takes a command from another thread in the queue and stores it into the database. The number of consumers is configurable ("processaudittrail.async.logger.count").

ProAuditTrail-storingCommands.png

Figure 4: Storing commands

There is always a designated node responsible for storing all commands of a certain process instance ID.

Implementation Detail

To be able to register distributed maps, this extension runs in Appway as a plug-in.

Note: If the extension startup fails, the Appway startup is aborted. If this happens, please check the log files to investigate the failure.

The Queue

The queue, as depicted earlier, is only a simplification. During implementation, it uses multiple distributed maps to achieve the queue behavior.

Distributed_maps_used.png

Figure 5: Distributed maps used

Example_map_entries.png

Figure 6: Example map entries

Three distributed maps are used:

  • logCommandIndexes Contains administration information for commands per process instance. Simply put, it maintains the proper order of the commands.
  • logCommands Contains each command. When consumed, the command results in one or more database insert, update or delete.
  • logCommandVersions Contains version information per process instance. Each database table has the process instance ID and version as primary keys. The version is maintained and incremented after each event.

Collecting

When Appway processes an HTTP request or triggers an update on a process instance, it happens in the context of a certain Java thread. While this thread is active, all Process Engine events are collected by this extension in a data structure that is visible only to this thread (so-called thread-local variables). Before the thread ends, the extension hooks in, stores the thread-local data into its distributed maps, and does a cleanup. Afterwards, the regular put to the Appway core distributed maps takes place.

Collecting_Details_Overview.png

Figure 7: Collecting details: Overview

Collecting_Details.png

Figure 8: Collecting details: Sequence Diagram (simplified)

Sequence:

  1. Get all thread-local "logCommands". Add them to a local list. Store the "logCommand" to the "logCommands" map.
  2. Lock the "logCommandIndexes" map with the current process instance ID. Get the "logCommandIndexes" entry. Create a new one if it doesn't exist. Append the local list. Put the index back to the distributed map. Unlock.
  3. Get the current max version for this process instance from the distributed map. If no longer needed, delete it; otherwise, update and rewrite.

Storing

As described earlier, each node is responsible for storing a subset of process instances into the database. The key for distributing the instances is the Process Instance ID. Each node has one or more consumer threads which takes commands out of the queue and stores them into the database. To avoid conflicts between multiple local threads, a local locking mechanism is in place (a synchronized map).

Storing_Details.png

Figure 9 : Storing details (simplified)

Sequence:

  1. Get the local set of process instances and loop through them.
  2. Lock the current process instance. If already locked, take the next free one.
  3. Get the "logCommandIndex" for this process instance.
  4. Loop through all "logCommands" and
  5. Save them to the database.
  6. After committing the batch, lock the "logCommandIndexes" map. Get the "logCommandIndex" for this process instance. Remove all previously processed entries.
  7. If there are entries left, write the new collection back to the "logCommandIndexes" map; otherwise, delete the entry.
  8. Unlock the current process instance.

Process Data Filter for Persistence

If configured, a process' data is persisted when completed ("processaudittrail.process.data.strategy") regardless of how the process completed (e.g., regular, canceled, deleted by an admin, etc.).

All data defined on the outermost process (also called "root process") qualifies for persistence. If you do not want to persist certain data, follow the filter options below. These filters also apply when you fetch data using the functions for displaying entries.

Variable Filter on Process

  • Example of Process Variables

    Example_Variables.png

    Figure 10 : Example variables

  • Filtering the variable value of "data" Open the Meta Data editor of the Process business object and add "ProcessAuditTrail-Exclude" as key and "data" as value.

    image001.png

    Figure 11: Variable Filter on Process

  • If you want to filter multiple variables, add a comma-separated list of variable names.

  • If you want to filter all variables, enter the wildcard character ("*").

Property Filter on Data Class

  • Example of Data Class definition

    Example_Data_Class_Definition.png

    Figure 12: Example Data Class definition

  • Filter the property "eyeColor" Open the Meta Data editor of the Data Class business object and add "ProcessAuditTrail-Exclude" as key and "eyeColor" as value.

    Property_Filter_on_Data_Class.png

    Figure 13: Property Filter on Data Class

  • If you want to filter multiple properties, add a comma-separated list of property names.

  • If you want to filter the whole Data Class, enter the wildcard character ("*").

Configuration and Setup

Configuring the Server

Appway needs an additional Hazelcast configuration. It changes the partitioning strategy in such a way that keys are scanned for an "@". Only the part after this character is used to evaluate the owning partition.

Note: While this configuration is not a problem for Appway default maps, it has an effect on all distributed maps! When you are using custom maps, you must take this behavior into account.

In your installation, go to the <data home>/conf directory. Add the following to the hazelcast.xml file:

Copy
<hazelcast …>
  …
  <properties>
    …
    <property name="hazelcast.partitioning.strategy.class">
    com.hazelcast.partition.strategy.StringAndPartitionAwarePartitioningStrategy</property>
    …
  </properties>
  …
</hazelcast>

Make sure you apply this configuration on all your nodes.

Configuring the Cluster Storage

Appway supports different kinds of storage mechanisms for its distributed maps. Apply one of the following configurations depending on the type of storage you have. This configuration, however, does not set the data storage for the log entries.

File System

No special configuration is needed.

Relational Database

To use this extension with the RelationalDbHazelcastStore extension, manually add the new database tables.

  • Connect to your database. You must have rights to add tables and indexes.
  • Follow the steps described in the System Installation article corresponding to your Platform version and adapt the SQL statements accordingly for the following table names:
    • "logcommands"
    • "logcommandindexes"
    • "logcommandversions"

Cassandra

To use this extension with the CassandraDataStore extension, manually add the new database tables.

  • Connect to your Cassandra installation. You must have rights to add tables.
  • Follow the steps described in the System Installation article corresponding to your Platform version and adapt the SQL statements accordingly for the following table names:
    • "logcommands"
    • "logcommandindexes"
    • "logcommandversions"

Configuring the Extension

There are two ways to configure the extension:

  • Via the Appway Studio When all nodes are up and running, go to Extensions > Permanent Extensions > ProcessAuditTrail [right click] > Edit Configuration. The configuration becomes active after the next restart. To have the same configuration active on each node, restart the whole cluster, not just the single nodes.
  • Via the "ProcessAuditTrail.jar.cfg" file Create or edit the configuration file "<data home>/extensions/ProcessAuditTrail.jar.cfg" manually. Ensure that the configuration is applied on each Appway cluster node.

Data Collection Properties

Properties define which and how each piece of data is collected. The amount of data generated depends on your solution. There is always a "strategy" and a "log" property:

  • "strategy" defines if an entry should be stored in the database table. If historized, it keeps the complete history of a value; if not, only the latest value is kept.
  • "log" defines if a human-readable log message should be generated and stored in the "audit_log" database table.

For more information on collected data stored in the database, see the chapter on Entity Relationship Diagram.

Configuration Property Description
processaudittrail.version.filter.strategyn(String) Version Filter strategynHow to log changes if a process instance's version filter changesnAffected database table: version_filternValid values: Ignore, KeepLatest, KeepHistorynDefault: KeepHistory
processaudittrail.version.filter.logn(Boolean) Version Filter lognWhether to log a human-readable log message if a process instance's version filter changesnAffected database table: audit_lognValid values: true, falsenDefault: true
processaudittrail.process.instance.status.strategyn(String) Process Instance Status strategynHow to log changes if a process instance's status changesnAffected database table: process_instance_statusnValid values: Ignore, KeepLatest, KeepHistorynDefault: Ignore
processaudittrail.process.instance.status.logn(Boolean) Process Instance Status lognWhether to log a human-readable log message if a process instance's status changesnAffected database table: audit_lognValid values: true, falsenDefault: true
processaudittrail.process.instance.attribute.strategyn(String) Process Instance Attribute strategynHow to log changes if a process instance's attribute is added, modified or removed.nAffected database table: process_instance_attributenValid values: Ignore, KeepLatest, KeepHistorynDefault: KeepLatest
processaudittrail.process.instance.attribute.logn(Boolean) Process Instance Attribute lognWhether to log a human-readable log message if a process instance's attribute is added, modified or removednAffected database table: audit_lognValid values: true, falsenDefault: true
processaudittrail.token.attribute.strategyn(String) Token Attribute strategynHow to log changes if a process instance token's attribute is added, modified or removednAffected database table: token_attributenValid values: Ignore, KeepLatest, KeepHistorynDefault: KeepLatest
processaudittrail.token.attribute.logn(Boolean) Token Attribute lognWhether to log a human-readable log message if a process instance token's attribute is added, modified or removednValid values: true, falsenDefault: true
processaudittrail.token.status.strategyn(String) Token Status strategynHow to log changes if a process instance token's status changesnAffected database table: token_statusnValid values: Ignore, KeepLatest, KeepHistorynDefault: Ignore
processaudittrail.token.status.logn(Boolean) Token Status lognWhether to log a human-readable log message if a process instance token's status changesnAffected database table: audit_lognValid values: true, falsenDefault: true
processaudittrail.token.user.strategyn(String) Token User strategynHow to log changes if a process instance token's user assignment changesnAffected database table: token_usernValid values: Ignore, KeepLatest, KeepHistorynDefault value: KeepHistory
processaudittrail.token.user.logn(Boolean) Token User lognWhether to log a human-readable log message if a process instance token's user assignment changesnAffected database table: audit_lognValid values: true, falsenDefault: true
processaudittrail.process.data.strategyn(String) Process Data strategynHow to log a process instance's root process data on process completion, termination or cancellationnValid values: Dispose, StorenDefault: Dispose

Other Properties

Configuration Property Description
processaudittrail.data.sourcen(String)

The Appway Data Source to be used to store the process audit log entriesnNote: Ensure it is available on startup.

Default: ProcessAuditTrail

processaudittrail.auto.migrate.databasen(Boolean)

Whether to auto-migrate database changes or not "Database changes" refer to the initial table and index creation and potential future updates of database objects. Set to false if your database user does not have DDL change rights.nIf set to false and there are pending changes, the extension startup is prevented and a file with all the changes is written to a temporary location. The exact location is written to the log.

Valid values: true, false

Default: true

processaudittrail.listener.priorityn(Integer)

The event listener's priority for receiving event notificationsnEnsure that other listeners that will modify data will be called after this listener. Check your system documentation about what such a listener does and its priority; otherwise, the audit messages could get messed up.nA listener implements the Java interface "com.nm.sdk.data.processes.listener.ProcessListener".nMeaning: The lower the priority, the later it will be called.

Valid range :-2147483647 - 2147483647, intended 1-100

Default: 10

processaudittrail.async.logger.countn(Integer)

Number of asynchronous loggers (consumers /DB worker threads). nProvide enough DB connections (connection pool size) since each thread will keep one all the time.

Valid values: 1 - 2147483647

Default: 5

processaudittrail.async.logger.priority

Asynchronous logger's Java thread prioritynThere are open issues with thread priorities in Java and Linux. At its current state, it only works under the following conditions:

  • Java version >= 1.6

  • Java process has to run with root privileges

  • JVM parameter -XX:UseThreadPriorities must be set

Meaning: Threads with higher priority are executed before other threads with lower priority.

Valid range: 1 - 10

Default: 7

processaudittrail.sync.versions.on.startup (Boolean)

Whether to synchronize the max version per process instance from the database on startup or not. This may be a very time-consuming task and delays the startup until all non-completed process instances versions are synchronized. This feature is disabled by default since all data is assumed to be in sync. To avoid corrupted log data, enable it when "processaudittrail.cluster.logCommandVersion.persistent" is set to false. Note: You need to run the extension with this feature enabled at least once in case of migrations from earlier versions.

Valid values: true, false.

Default value: false

processaudittrail.sync.versions.tables (String)

Configure the tables you want to use as input for the database sync (on startup ("processaudittrail.sync.versions.on.startup" set to true) or via job (schedule set via "processaudittrail.sync.versions.job.schedule")). Cover the tables needed (depends on the data collection properties used). If enabled, the tables "process_instance" and "process_instance_status" are processed by default.

Valid values: version_filter, process_instance_attribute, audit_log, token, token_attribute, token_status, token_user.

Default: [empty]

processaudittrail.sync.versions.job.schedule (String) Define an automated job to synchronize the max versions from the database. This job will be registered after a successful extension startup and, if clustered, will run only on one node. Note: While the job is running, all access to the "logcommandversionsmap" will start using distributed locks. This decreases the throughput to write data to the database. This job is not needed during regular operation. You should only enable it if Appway Support suggests you to do so. Valid values: Valid cron job pattern, e.g., daily at 2am: 0 0 2 * * ?. Default: [empty]
processaudittrail.cluster.logCommand.persistent

Whether to enable persistence for the logCommands and logCommandIndexes map or not. If you have a file system storage (local or NFS) in use and have a high load or very high I/O, set this parameter to false. The drawback is that you lose persistence for log entries which, in worst case, leads Process Audit Trail data loss (i.e., those that have not yet been written to the database).

Note: Changing this value only has an effect when restarting the whole cluster.

Valid values: true, false.

Default: true

processaudittrail.cluster.logCommandVersion.persistent (Boolean)

If you are willing to accept longer startup times, disable the persistence of the "logCommandVersions" map (same reasons as those mentioned for "processaudittrail.cluster.logCommand.persistent"). If you disable it, you must enable the sync on startup option by setting "processaudittrail.sync.versions.on.startup = true".

Valid values: true, false.

Default: true

processaudittrail.database.connection.retry.count.on.startup (Integer)

During startup, the number of times a database reconnect is tried before an exception is thrown. Note: There is a two-second pause between a failure and a retry. Therefore, a retry count of 4 means a possible wait time of up to 8 seconds before an exception is thrown.

Meaning: -1 = endless, else retry count.

Valid values: -1 - 2147483647 (-1 means endless).

Default: 1

processaudittrail.async.logger.reconnect.after (Integer)

How many log commands shall be written by an asynchronous logger until the database connection is returned to the connection pool and a new one is requested. Meaning: 0 = never, else write count.

Valid values: 0 - 2147483647.

Default value: 0

processaudittrail.async.logger.transaction.size (Integer)

Sets the maximum size of a database transaction before a commit is issued. Use a low value if there are only a few commands per Process Instance left to process. The lower the number, the more updates and locks take place on the distributed map 'logCommandIndexes'. This can become expensive and end in bad throughput. As an option, you can increase the number of writer threads, but this does not reduce the load on the distributed map ('processaudittrail.async.logger.count').

Valid values: 1 - 2147483647.

Default value: 30

processaudittrail.getMaxVersions.parallel.degree.hint (String)

When evaluating the maximum version used per Process Instance, you can provide a parallel degree hint for the query. This can be necessary if you have a huge amount of data to process and want to optimize performance. Ask your DBA for more details. Example value for Oracle: /*+ PARALLEL(12) */. Example value for MS SQL Server: OPTION (MAXDOP 12). Meaning: [empty] = database or table default, otherwise the hint you provide.

Valid values: Database dependent hint. Currently available for Oracle and MS SQL Server.

Default value: [empty]

processaudittrail.business.object.caches.invalidation.time (Integer)

Business Object caches invalidation time. All Business Objects used by this extension are reloaded after this timeout [s]. Its values are cached. All Business Objects will be read using the LatestCommittedFilter, meaning it uses the latest committed version of a Business Object. All used Business Objects can be identified using the dependency view on the Appway Studio.

Meaning: Seconds.

Valid values: 1 - 2147483647.

Default: 300

processaudittrail.languages.full.locale (Boolean)

Whether to log messages with full locale support or not. For example, if you installed Appway languages de, de_CH, en, en_US, en_CA and the configuration is set to "false", it will work only with de, en. If set to "true", it will work with all of them. Note: This setting has an effect on the number of columns used in the database tables and amount of data written.

Valid values: true, false.

Default: false

processaudittrail.skip.injected.processes (Boolean)

Disable audit trail on injected processes (with or without Swimlanes).

Valid values: true, false.

Default value: true

processaudittrail.user.system (String)

String to use as system user ID.

Valid values: any string.

Default: System

processaudittrail.user.id.preference (String)

If your company's real user ID is not the same as Appway's, provide an Appway user preference name which contains the real user ID. That ID is used in log entries instead of Appway's user ID.

Valid values: An existing user preference name.

Default: [empty]

processaudittrail.group.display.text.function
processaudittrail.role.display.text.function (String)

If your role/groups use artificial names, provide script function names which can resolve a role/group ID into a preferred display text. This text is used in log entries instead of Appway's group/role names. It will always use the latest committed version of those script functions.

Valid value: A name of a script function taking the role/group ID as parameter and returns a display text string.

Default: [empty]

processaudittrail.element.name.prepend.pattern

Provide a regular expression pattern which will be applied to an element's Name field (set via the Process Editor). The regular expression's first found result is prepended to the element's Task List Name and is saved as the element's name in the database. If prepended, there will be a separator: ' - '.

Valid value: A regular expression following the rules of the Java class Pattern.

Default: [empty]

processaudittrail.shutdown.behavior (String)

Define what should happen on an extension shutdown

  • "delayedDatabase": Wait and try to write all remaining events to the database only for a certain amount of time.

  • "database": Wait and try to write all remaining events to the database.

Note: In cluster mode, other nodes can still produce entries which will be processed on this node. This blocks a shutdown on this node until the others go down, too.

processaudittrail.shutdown.behavior.delay (Integer)

If "processaudittrail.shutdown.behavior = delayedDatabase", define a delay [ms]. After that time, storage is suspended and the remaining entries are stored and processed on the next startup (if persistence is enabled).

Meaning: Milliseconds

Valid values: 0 - 2147483647

Default: 120000

Configuring Content

Labels

This extension uses the following Labels to log events. By default, they're installed in English ("en") and German ("de"). If you need more translations or if you want to change the existing message, simply add the new content the same way you would for any other Label business object. Some Labels use placeholders such as "$1" or "$2". The descriptions below are also found in Appway Studio on the Label's property tab ("Description").

  • ProcessAuditTrail_AdminPrefix
  • ProcessAuditTrail_AttachedCancelIntermediateEventTriggered
    • $1: Type
  • ProcessAuditTrail_AttachedErrorIntermediateEventTriggered
    • $1: Type
  • ProcessAuditTrail_AttachedMessageReceiveIntermediateEventTriggered
    • $1: Type
  • ProcessAuditTrail_AttachedSignalReceiveIntermediateEventTriggered
    • $1: Type
  • ProcessAuditTrail_AttachedTimerIntermediateEventTriggered
    • $1: Type
  • ProcessAuditTrail_DueDateExpired
  • ProcessAuditTrail_DynamicElementId
  • ProcessAuditTrail_PreviousTokenAttributesCopied
  • ProcessAuditTrail_ProcessCancelled
  • ProcessAuditTrail_ProcessCompleted
  • ProcessAuditTrail_ProcessDeleted
  • ProcessAuditTrail_ProcessInjected
  • ProcessAuditTrail_ProcessInstanceAttributeAdded
    • $1: Attribute name
    • $2: Value
  • ProcessAuditTrail_ProcessInstanceAttributeModified
    • $1: Attribute name
    • $2: Old value
    • $3: New value
  • ProcessAuditTrail_ProcessInstanceAttributeRemoved
    • $1: Attribute name
  • ProcessAuditTrail_ProcessInstanceRestored
    • $1: Original last modification date
  • ProcessAuditTrail_ProcessInstanceStatusChanged
    • $1: Old state
    • $2: New state
  • ProcessAuditTrail_ProcessLinkSourceIntermediateEventCancelled
  • ProcessAuditTrail_ProcessLinkSourceIntermediateEventRegistered
  • ProcessAuditTrail_ProcessLinkSourceIntermediateEventTriggered
  • ProcessAuditTrail_ProcessLinkTargetIntermediateEventCancelled
  • ProcessAuditTrail_ProcessLinkTargetIntermediateEventRegistered
  • ProcessAuditTrail_ProcessLinkTargetIntermediateEventTriggered
  • ProcessAuditTrail_ProcessMessageReceiveIntermediateEventCancelled
  • ProcessAuditTrail_ProcessMessageReceiveIntermediateEventRegistered
  • ProcessAuditTrail_ProcessMessageReceiveIntermediateEventTriggered
  • ProcessAuditTrail_ProcessMessageSendIntermediateEventCancelled
  • ProcessAuditTrail_ProcessMessageSendIntermediateEventRegistered
  • ProcessAuditTrail_ProcessMessageSendIntermediateEventTriggered
  • ProcessAuditTrail_ProcessNoneIntermediateEventCancelled
  • ProcessAuditTrail_ProcessNoneIntermediateEventRegistered
  • ProcessAuditTrail_ProcessNoneIntermediateEventTriggered
  • ProcessAuditTrail_ProcessReactivated
  • ProcessAuditTrail_ProcessSignalReceiveIntermediateEventCancelled
  • ProcessAuditTrail_ProcessSignalReceiveIntermediateEventRegistered
  • ProcessAuditTrail_ProcessSignalReceiveIntermediateEventTriggered
  • ProcessAuditTrail_ProcessSignalSendIntermediateEventCancelled
  • ProcessAuditTrail_ProcessSignalSendIntermediateEventRegistered
  • ProcessAuditTrail_ProcessSignalSendIntermediateEventTriggered
  • ProcessAuditTrail_ProcessStarted
    • $1: Starter's user ID
  • ProcessAuditTrail_ProcessTimerIntermediateEventCancelled
  • ProcessAuditTrail_ProcessTimerIntermediateEventRegistered
    • $1: Date
  • ProcessAuditTrail_ProcessTimerIntermediateEventTriggered
  • ProcessAuditTrail_ProcessTokenAdded
  • ProcessAuditTrail_ProcessTokenAssigned
    • $1: Previous user ID
    • $2: New user ID
  • ProcessAuditTrail_ProcessTokenAttributeAdded
    • $1: Attribute name
    • $2: Value
  • ProcessAuditTrail_ProcessTokenAttributeModified
    • $1: Attribute name
    • $2: Old Value
    • $3: New value
  • ProcessAuditTrail_ProcessTokenAttributeRemoved
    • $1: Attribute name
  • ProcessAuditTrail_ProcessTokenRemoved
  • ProcessAuditTrail_ProcessTokenStatusChanged
    • $1: Old state
    • $2: New state
  • ProcessAuditTrail_TakeOwnershipFromGroup
    • $1: From group
  • ProcessAuditTrail_TakeOwnershipFromRole
    • $1: From role
  • ProcessAuditTrail_VersionFilterChanged
    • $1: Old version filter
    • $2: New version filter

Process Filter

By default, the extension will record events for all persistent Appway processes, i.e., all root processes with Swimlanes. During the very first startup, the extension installs a "ProcessAuditTrail_ProcessFilter" catalog where you can define the process ID filter rules for your solution.

Process_Filter_example.png

Figure 14: Process Filter example

Filter processing rules:

  • The records are processed in the order received.
  • To explicitly add a process ID, use the keyword "allow".
  • To explicitly skip a process ID, use the keyword "deny".
  • The wildcard character ("*") adds or skips all currently existing processes.
  • The filter rules are processed and cached. The cache is cleared frequently and the rules will be re-evaluated (see configuration property "processaudittrail.business.object.caches.invalidation.time").
  • The latest committed version of this catalog is always used.
  • You can change the definition of the catalog but it must at least contain the columns "mode" and "processId".

Database

A database stores the log entries.

Supported Databases

For a list of supported databases for your Appway version, see the Technical Requirements article.

Database Configurations

Certain databases need a special configuration to make the extension work as expected.

MS SQL Server

Connect as a DBA and issue the following commands:

  • ALTER DATABASE <database name> SET READ_COMMITTED_SNAPSHOT ON
  • ALTER DATABASE <database name> SET ALLOW_SNAPSHOT_ISOLATION ON
Important! Consider installing all database objects for this extension in a separate database. The two statements mentioned above alter the isolation level of the database to a non-default behavior. As a result, other objects and depending queries/procedures might no longer work as expected.

DDLs and DMLs

All DDLs and DMLs used inside the extension are contained in a text and XML file. If you extract the extension's JAR file, go to:

  • DDL: "db/ddl/<version>/<database vendor>.sql" It is possible to have multiple versions to support an automated upgrade process. Version 1 is the initial version; all others are subsequent ones. To check the currently installed version, query the "db_info" table.
Note: You must not execute these files manually. They contain placeholders which will be automatically filled before being applied. If "processaudittrail.auto.migrate.database" is disabled, a complete SQL file is rendered, which you can then apply. After the first startup, check the debug.log to find where this file is located.
Note: For performance reasons, foreign keys are disabled by default.
  • DML: "db/sql/<database vendor>.xml"

Auto-Migration

By default, all database objects used by this extension are installed automatically. Future database changes can be automated as well. See the section on configuring the "processaudittrail.auto.migrate.database" property.

If auto-migration is enabled but the database user does not have the right to execute DDLs, the extension startup will fail. In Appway's log files, find a file system path to a temporary file containing the needed DDL statements, which you can then execute manually.

Languages

Appway supports multiple languages for solutions and the same is true for logged messages and events. The Labels used can change over time, but log messages always log using the Labels valid at the time of the event. Therefore, the Labels are translated and stored in multilingual versions in the database.

At first startup, the extension installs all database tables needed. Some tables contain columns which store data in multiple languages. In the Entity Relationship Diagram, there are table columns with the postfix placeholder "_". This placeholder is replaced by the concrete languages used in your solution (one column per language). If you add a new language to your solution, the extension tries to create additional database table columns. If auto-migration is disabled, it creates a temporary file which you can execute manually.

Entity Relationship Diagram

Entity_relationship_diagram.png

Figure 15: Entity relationship diagram

Database Tables in Detail

The "version" column name represents a simple sequence for each process instance and it starts at 1. This is necessary because timestamps do not provide enough resolution to generate unique entries. In Appway, the versions are cluster-wide and are maintained outside of the database via the distributed map "logCommandVersions".

"process_instance"

Contains basic data about the running process instance. Other tables directly or indirectly relate to this table via the process instance ID.

Column Description
idn(PK) The process instance ID
process_id The process' ID
started_by The user ID of the user that started the process
start_version Per process instance unique sequence value
start_time Timestamp when the process was started
end_version Per process instance unique sequence value
end_time Timestamp when the process was completed
value_store_id ID of the Value Store used by the process
process_description_ The process' description (multilingual)
archived The ProcessAuditTrail extension just collects data but never deletes. Use this flag when archiving or deleting collected data.

"audit_log"

Contains all generated event log messages in multiple languages.

Column Description
process_instance_idn(PK) The process instance ID
versionn(PK) Per process instance unique sequence value
version_countn(PK) There are some events which produce more than one log entry. To be able to insert unique messages, a second sequence is needed.
user_id The user ID this log message belongs to
event_id The ID of the event
event Description of the event
event_type_id The ID of the event category
event_type Description of the event category
message_ The log message. There is one column for each installed language.
timestamp Timestamp when the log message was created inside Appway
admin_event Indicator if this log message is based on an administration action or not
user_id_update_state Reserved for future use
token_id If this log message is somehow related to a token, the token ID will be stored.

"version_filter"

Contains all version filters with which a process instance was running.

Column Description
process_instance_idn(PK) The process instance ID
version Per process instance unique sequence value
timestamp Timestamp when this version filter was set
filter A string representation of the version filter used

"process_instance_status"

Contains the process instance's status.

Column Description
process_instance_idn(PK) The process instance ID
versionn(PK) Per process instance unique sequence value
timestamp Timestamp when this process instance status was set
status The process instance status

"process_instance_attribute"

Contains the process instance's attributes and its values. The values are converted into a string representation before storing them into the database.

Column Description
process_instance_idn(PK) The process instance ID
versionn(PK) Per process instance unique sequence value
timestamp Timestamp when this process instance attribute was modified
namen(PK) Timestamp when this process instance status was set
type Java type of the value, e.g., when the value is a string, the type would be "java.lang.String".
value The plain, untranslated value.
value_ The value translated into different languages
action Modification action. This is relevant if data is stored historized. If so, it will contain entries like "added", "updated", "deleted", etc.

"token"

Contains all information about a token.

Column Description
idn(PK) The token ID
process_instance_idn(PK) The process instance ID
process_id The process' ID. In this case, it will be a Sub-Process ID.
parent_token_id The parent token ID
previous_token_id The previous token ID
incomming_connection_id The ID of the connection this task was reached
stream An artificial number identifying parallel streams, e.g., after an AND gateway. If there's no parallel stream, this column is null.
depth If on the level of the root process "1"; otherwise, the nested depth of this token.
element_id The ID of the element in the Process this token was created for
element_type The type of the element in the Process this token was created for
external Indicator if the element in the Process this token was created for is completed by an external system. "external" means outside of Appway and is usually a human task (e.g., a Screen or Placeholder task).
element The element's name (defined via Process editor)
element_name_ A description of the element by using its Task List Name, Description and Name
swimlane_id The ID of the Swimlane in the Process this token was created for
swimlane The Swimlane's name (defined via Process editor)
swimlane_name_ A description of the Swimlane using its Task List Name, Description and Name
start_version Per process instance unique sequence value
start_time Timestamp when the process was started
end_version Per process instance unique sequence value
end_time Timestamp when the process was completed

"token_status"

Contains the token's status.

Column Description
process_instance_idn(PK) The process instance ID
token_idn(PK) The token ID
versionn(PK) Per process instance unique sequence value
timestamp Timestamp when this token status was set
status The token status

"token_attribute"

Contains the token's attributes and its values. The values are converted into a string representation before storing them into the database.

Column Description
process_instance_idn(PK) The process instance ID
token_idn(PK) The token ID
versionn(PK) Per process instance unique sequence value
timestamp Timestamp when this token attribute was modified
namen(PK) Timestamp when this token status was set
type Java type of the value, e.g., when the value is a string the type would be "java.lang.String".
value The plain, untranslated value
value_ The value translated into different languages
action Modification action. This is relevant if data is stored historized. If so, it will contain entries like "added", "updated", "deleted", etc.

"token_user"

Contains the token's user assignment.

Column Description
process_instance_idn(PK) The process instance ID
token_idn(PK) The token ID
versionn(PK) Per process instance unique sequence value
timestamp Timestamp when this token status was set
user_id The user assigned to the token
action Modification action. This is relevant if data is stored historized. If so, it will contain entries like "added", "updated", "deleted", etc.

"entity"

Contains the root process' data which is available after reaching its end. It describes the data and collection type.

Column Description
process_instance_idn(PK) The process instance ID
idn(PK) An artificial ID identifying this entity
type Appway data type. The root entry has the value "ProcessAuditTrail_Root".
collection The collection type: "none", "indexed" or "mapped"

"property"

Contains the root process' data which is available after reaching its end.

Column Description
process_instance_id The process instance ID
entity_id Reference to the entity this property belongs to
child_entity_id If this property has a child or not. A property value is a child.
name The property's name
key If "mapped", the key; if "indexed", the index; Otherwise, Null .
value The property's value

"event_type"

Reference data table which defines event categories.

Column Description
idn(PK) Ascending ID of an event type
type The event's type

"event"

Reference data table which defines event categories.

Column Description
idn(PK) Ascending ID of an event type
name The event's name
event_type_idn(PK) The event's type ID

"db_info"

Contains information about the currently installed version. It is used for automatic migrations and multiple language support. Do not change the content of this table.

Column Description
version The installed version of the extension's database model
installed_languages The languages currently installed in the database. If new languages will be added to Appway, those have to be installed into the database as well.

Functions

Storing Data

ProcessAuditTrail_LogCustomEvent

This function allows you to add any custom log entry to the audit database. You can use it from anywhere you like and log any message you like.

As an alternative, you can use the ProcessAuditTrail_LogCustomEvent Trackpoint.

Parameter Description
MessagesnNamed String Named collection of message texts for the corresponding languages. Use "$1" as placeholder for the user ID in the message.nExample: "{'en'='English text', 'de'='Deutscher Text', 'fr'='Texte français', 'it'='Testo italiano'}:String"
processInstanceIdnString, optional An optional process instance ID for this log entrynBy default, the current process Instance ID will be used. If you call this function from outside a process instance and do not provide a process instance ID, it will raise an exception.nExample: "123456789012"
userIdnString, optional An optional user ID for the log entrynBy default, the current user ID will be used. nExample: "superUser"
eventTypeIdnInteger, optional An optional event type ID for the log entry. By default, value 6 (“Custom”) is used. The corresponding event type entry must be present in the event_type table.
eventIdnInteger, optional An optional event ID for the log entry. By default, value 13 (“Custom event”) is used. The corresponding event entry must be present in the event table.
tokenIdnInteger, optional An optional token ID for the log entry. By default, the token ID of the current context is used (if available). Otherwise, null is used.

Reading Data

No generic viewer is available because each customer has different needs. However, some helper functions are provided to read and display data. Data is loaded from either live process instances or from the database (completed process instances). The functions return the same data structures so you only have to build a viewer once. The following are examples for loading and displaying data.

  • Load data

    Copy
        //load log data
        Indexed ProcessAuditTrail_Log $auditLog;
        $auditLog := ProcessAuditTrail_LoadAuditLog($processInstanceId, false);
        //load process instance attribute data
        ProcessAuditTrail_ProcessInstanceAttributes $piAttrs;
        If $live Then 
           $piAttrs := ProcessAuditTrail_LoadLiveProcessAttributes($processInstanceId);
        Else
           $piAttrs := ProcessAuditTrail_LoadLoggedProcessAttributes($processInstanceId);
        End
        //load process data
        ProcessAuditTrail_Scope $scope;
        If $live Then 
           $scope := ProcessAuditTrail_LoadLiveProcessData($processInstanceId);
        Else
           $scope := ProcessAuditTrail_LoadLoggedProcessData($processInstanceId);
        End
  • Prepare for log message view

    Copy
        //initialize log items
        Indexed ProcessAuditTrail_Log $logItems := ProcessAuditTrail_NestedAuditLogToList($auditLog);
        //sort items
        ProcessAuditTrail_Log $i1;
        ProcessAuditTrail_Log $i2;
        SORT($logItems, $i1.version.compareTo($i2.version), $i1, $i2, true);
  • Prepare for process instance attribute view

    Copy
        //initialize attribute items
        Indexed ProcessAuditTrail_ProcessInstanceAttributes $attributeItems := ProcessAuditTrail_NestedAttributesToList($wfAttributes);
  • Prepare for process data view

    Copy
        //initialize data items
        Indexed ProcessAuditTrail_DataItem $dataItems := new Indexed ProcessAuditTrail_DataItem;
        ProcessAuditTrail_Scope $s;
        If $scope.name == 'Root' Then 
           ForEach $s In $scope.subScopes Do 
              $dataItems.addAll(COMMON_AUDIT_NestedScopesToList($s));
           End
        Else
           $dataItems.addAll(COMMON_AUDIT_NestedScopesToList($scope));
        End
  • Helper Script Functions

    Copy

        Function ProcessAuditTrail_NestedAuditLogToList(Indexed ProcessAuditTrail_Log $nested, Indexed ProcessAuditTrail_Log $list := null, Integer $indentLevel := 0, String $pathPrefix := null) : Indexed ProcessAuditTrail_Log Begin 

           If $nested == null Then 

              Return $list;

           End

           //

           If $list == null Then 

              $list := NewIndexed(ProcessAuditTrail_Log);

           End

           //

           ProcessAuditTrail_Log $logItem;

           String $itemPath;

           Integer $i;

           //

           For $i := 1 Condition $i <= $nested.size() Step $i := $i + 1 Do 

              $logItem := $nested[$i];

              $itemPath := CONCAT($pathPrefix, '/', TOSTRING($i));

              $logItem.indentLevel := $indentLevel;

              $logItem.itemPath := $itemPath;

              $list.addElement($logItem);

              ProcessAuditTrail_NestedAuditLogToList($logItem.children, $list, $indentLevel + 1, $itemPath);

           End

           //

           Return $list;

        End

    Copy


        Function ProcessAuditTrail_NestedAttributesToList(Indexed ProcessAuditTrail_ProcessInstanceAttributes $nested, Indexed ProcessAuditTrail_ProcessInstanceAttributes $list := null, Integer $indentLevel := 0, String $pathPrefix := null) : Indexed ProcessAuditTrail_ProcessInstanceAttributes Begin 


           If $nested == null || $nested.size() == 0 Then 


              Return $list;


           End


           If $list == null Then 


              $list := NewIndexed(ProcessAuditTrail_ProcessInstanceAttributes);


           End


           ProcessAuditTrail_ProcessInstanceAttributes $attributes;


           ProcessAuditTrail_ProcessInstanceAttributes $clonedItem;


           String $itemPath;


           ForEach $attributes In $nested Do 


              $itemPath := CONCAT($pathPrefix, '/', $attributes.name);


              $clonedItem := CLONE($attributes, false, false);


              $clonedItem.indentLevel := $indentLevel;


              $clonedItem.itemPath := $itemPath;


              $list.addElement($clonedItem);


              ProcessAuditTrail_NestedAttributesToList($attributes.children, $list, $indentLevel + 1, $itemPath);


           End


           Return $list;


        End


    Copy



        Function ProcessAuditTrail_NestedScopesToList(ProcessAuditTrail_Scope $scope, Indexed ProcessAuditTrail_DataItem $list := null, Integer $indentLevel := 0, String $pathPrefix := null) : Indexed ProcessAuditTrail_DataItem Begin 



           If $scope == null Then 



              Return $list;



           End



           If $list == null Then 



              $list := NewIndexed(ProcessAuditTrail_DataItem);



           End



           String $scopePath := CONCAT($pathPrefix, '/', $scope.name);



           //



           ProcessAuditTrail_DataItem $scopeItem := new ProcessAuditTrail_DataItem;



           $scopeItem.name := $scope.name;



           $scopeItem.indentLevel := $indentLevel;



           $scopeItem.children := CLONE($scope.variables, false, false);



           $scopeItem.type := '%SCOPE%';



           $scopeItem.itemPath := $scopePath;



           $list.addElement($scopeItem);



           //



           If $scope.subScopes != null && $scope.subScopes.size() > 0 Then 



              If $scopeItem.children == null Then 



                 $scopeItem.children := new Indexed ProcessAuditTrail_DataItem;



              End



              $scopeItem.children.addElement(new ProcessAuditTrail_DataItem);



           End



           //



           ProcessAuditTrail_NestedDataitemsToList($scope.variables, $list, $indentLevel + 1, $scopePath);



           ProcessAuditTrail_Scope $subScope;



           ForEach $subScope In $scope.subScopes Do 



              ProcessAuditTrail_NestedScopesToList($subScope, $list, $indentLevel + IF($scope.variables != null && $scope.variables.size() > 0, 1, 1), $scopePath);



           End



           Return $list;



        End



ProcessAuditTrail_LoadLiveProcessData

This function returns data from a live process. The data is collected from the in-memory instance, not read from the database.

Parameter Description
processInstanceIdnString The process instance ID to load the live data from Example: "123456789012"
Return value Description
ProcessAuditTrail_Scope A data structure containing all of the process' live data

ProcessAuditTrail_LoadLoggedProcessData

This function returns data from a completed process. This data is read from the database. The data can only be returned if the "processaudittrail.process.data.strategy" configuration is set to "store". If not, no data will be saved upon process completion.

Parameter Description
processInstanceIdnString The process instance ID to load the live data from Example: "123456789012"
Return value Description
ProcessAuditTrail_Scope A data structure containing all of the process' live data

ProcessAuditTrail_LoadLiveProcessInstanceAttributes

This function returns process instance attributes from a live process. The data is collected from the in-memory instance, not read from the database.

Parameter Description
processInstanceIdnString The process instance ID to load the live data from Example: "123456789012"
Return value Description
ProcessAuditTrail_ProcessInstanceAttributes A data structure containing all of the process' live instance attributes

ProcessAuditTrail_LoadLoggedProcessInstanceAttributes

This function returns process instance attributes from a completed process. The data is read from the database. The data can only be returned if the configuration "processaudittrail.process.instance.attribute.strategy" is set to "KeepLatest" or "KeepHistory". If not, no data will be saved.

Parameter Description
processInstanceIdnString The process instance ID to load the live data from Example: "123456789012"
Return value Description
ProcessAuditTrail_Scope A data structure containing all of the process' live data

ProcessAuditTrail_LoadLog

This function returns all log messages for a given process instance ID. The data is always read from the database.

The log detail level is defined via various configurations. See Data Collection Properties for more information. Entries may offer too much detail for a quick view so this function applies a static, predefined filter which contains:

  • Version filter changed

  • Process Instance status changed

  • (Sub-)Process Token status changed

  • Previous Process Token copied

  • Automatic Take Ownership from role/group

  • Process Instance Attribute added/modified/removed

  • (Sub-)Process Token Attribute added/modified/removed

  • Task opened/retained/taken over by user

  • Link Source/Target

Parameter Description
processInstanceIdnString The process instance ID to load the live data from Example: "123456789012"
unfilterednBoolean, optional Whether to load the log messages and apply a static, predefined filter or notnDefault: falsenExample: "123456789012"
Return value Description
Indexed ProcessAuditTrail_Log A data structure containing all of the process' log messages

Trackpoint

ProcessAuditTrail_LogCustomEvent

To avoid using explicit Script tasks or hard-to-find function calls inside other scripts or Screens, use the "ProcessAuditTrail_LogCustomEvent" trackpoint to add custom log entries instead of the function.

PAT_71.png

Figure 16: Custom Event Trackpoint

Parameter Description
NamenString, optional Standard component name property
DescriptionnString, optional Standard component description property
Task List NamenString, optional Name of this element if it appears in a Task List or elsewhere in the Workspace (may contain Language Labels)
MessagesnNamed String Named collection of message texts for the corresponding languages. Use "$1" as placeholder for the user id in the message.nExample: ['en'='Now', 'de'='Jetzt']:String
Process Instance IdnString, optional An optional Process Instance ID for the log entry. By default, the current Process Instance ID will be used.nExample: '123456789012'
User IDnString, optional An optional user ID for the log entry. By default, the current user will be used.nExample: 'superuser'
Event Type IDnInteger, optional An optional event type ID for the log entry. By default, value 6 (“Custom”) is used. The corresponding event type entry must be present in the event_type table.
Event IDnInteger, optional An optional event ID for the log entry. By default, value 13 (“Custom event”) is used. The corresponding event entry must be present in the event table.

GDPR Compliance

The General Data Protection Regulation (GDPR) is an EU regulation aimed at protecting the privacy of natural persons regarding the processing of personal data and the free movement of such data. The purpose of GDPR, among others, is to strengthen and to unify data protection laws within the EU for individuals, ensuring individuals have control of their personal data.

Important! The ProcessAuditTrail extension stores data and as such it can be used to persist information on data subjects (natural persons). Implementing a data persistence strategy that is compliant with the regulations stipulated by GDPR must be guaranteed by the implementation of the Solution by the customer or entity using this product. Compliance is not inherent in any way to the product itself.

This document and the content thereof and any information derived from it is not to be construed as legal, regulatory and/or tax advice. Appway is not permitted to and will not provide any legal, regulatory and/or tax advice. Appway is also not providing an interpretation of any laws or regulations that may be applicable to any customer, prospect, or anyone, and they shall be responsible for clarifying and stipulating the legal and regulatory requirements applicable to its business. While Appway personnel providing the services may, through experience or specialized training or both, be familiar with the general regulatory environment in their capacity as information-technology and management-consulting professionals, they will work under the direction of the customer and its legal counsels regarding the specific legal and regulatory requirements under which the customer operates. The compliance of the services and work results with the applicable laws and requirements remains the sole responsibility of customer and their legal, regulatory and/or tax advisors.