DataAuditTrail Extension

Introduction

The purpose of Data Audit Trail extension is to track changes made on data entities in the Value Store during a Process on a property level. This is usually relevant for data mutation processes, where the performed changes should be reviewed, approved, and/or archived at the end of the Process.

Change tracking is mostly transparent for the solution. All operations that change data, such as assignments on properties or add/remove/update operations on collections, are tracked in the background. All you have to do is configure which data classes and properties to track, activate the tracking in your process, and finally retrieve the changes to display, further process, or archive.

Every change that is performed, such as a client’s phone number being updated to their new number, is represented by an instance of the DataAuditChange Data Class. The Data Class contains:

  • The subject of the change (the entity whose property was changed – in our example, the client)
  • The previous, and new values
  • Information about who performed the change
  • Information about when the change took place

Figure 1 shows an example of a data table displaying changes collected with the DataAuditTrail extension. In this example, a person’s date of birth has been changed, and a new address in New York has been added.

changes-screenshot.png

Figure 1 : Data table displaying changes collected with the DataAuditTrail extension

Configuring Tracked Classes and Properties

Configure which Data Classes and which of their properties should be tracked. To this end, the DataAuditTrail extension provides an administration interface in the Studio under Solution Maintenance > Data Audit Trail > Data Class Configuration.

Each Data Class that you want to track has to be configured accordingly. After selecting the Data Class, specify a Data Class Display Expression used to create a textual representation of a changed entity at the time of the change for display purposes. The variable $this can be used to access the entity.

For example, a person could be represented by that person’s name, followed by date of birth in brackets. The corresponding expression would be:

Copy 
$this.name & ‘ (’ & DATEFORMAT($this.dateOfBirth, 'MM-dd-yyyy') & ‘)’

Each property declared in the Data Class has the following configuration options:

  • Track changes — Activate to track changes on this property.
  • Field display name — A user-readable name of the property. This value is used in the path of the change, a user-readable string indicating where in an entity graph a change occurred.
    • For collections, this value can be an expression, where you can use the $index (indexed collection) or $key (named collection) variables. The field display name for an indexed collection of addresses, for example, could be something like this: ‘Addresses[‘ & $index & ‘]’
  • Field display value expression — An expression to create a textual representation of a changed property at the time of the change. This expression is evaluated for the old and new value when a change happens and is stored for every change.
    • If the property’s type is a Data Class, the Field display value expression takes precedence over the "Data Class Display Expression" defined on that Data Class.
  • Reference catalog — A catalog that defines the display value for every possible value of the property. It is an arbitrary catalog that has a column containing all possible values of the property, and a column to which these values should be mapped for display purposes. The columns can be selected in the two dropdowns next to the catalog selection. If a Reference catalog is defined, it takes precedence over the Field display value expression.

After configuring the change tracking for a Data Class, press Save to persist the changes.

The change tracking configuration is stored as metadata attributes on the data class, so it is versioned. You have to commit the Data Class to make changes available in the latest committed version, and also part of exports of the Data Class.

For inherited properties, the configuration of the corresponding parent class is used.

Change Tracking

What is a Change?

There are three types of changes: UPDATE, ADD, and REMOVE.

  • An UPDATE change on a data object is an update of an entity property, which is usually caused by an assignment in the Script Language or the execution of a binding during Screen rendering. The update only counts as change, however, if the previous and new value of the property are not equal, where equality is assessed using the equals method of the old and new data entities.
    An UPDATE change on an indexed or named collection is caused by the replacement of a value in the collection. Again, a replacement only counts as change if the old and new values are not equal.
  • An ADD change is an insertion in a named or indexed collection.
  • A REMOVE change is removal of an element from a named or indexed collection.

Note that the changes represent the actual actions performed, not the minimal diff of the old and new entity graphs. If, for example, you assign to a person’s address property a new address object that is equal to the old address object except for the name of the street where the person lives, this is tracked as a complete replacement of the value in the address property, not just a change of the street.

Furthermore, for indexed collections, only explicit insertions and deletions are represented as changes, not the shift of elements caused by insertions and deletions.

Where are the Changes Stored?

The changes collected while change tracking is activated are stored in a hidden scope in the Value Store, and managed by the DataAuditTrail extension.

Note: The DataAuditTrail extension does not persist any changes beyond those described above. If you want to archive the changes, you have to retrieve them at the end of your Process and archive them using a mechanism of your choice.

Start Change Tracking: DataAuditActivateTracking function

Tracking changes made on entities in a process requires activating tracking using the DataAuditActivateTracking function. This activates the change tracking on all entities reachable from the supplied entity. If, for example, you activate tracking on a person that contains an indexed collection of addresses, changes on the addresses are also tracked. On entities added to the graph at a later point, change tracking is activated automatically.

Copy 
DataAuditActivateTracking($object, $active, $consolidate)
Parameter Description
$object Any The root entity of the entity graph to track. Activation/deactivation of the tracking is performed on each entity reachable from the given entity.
$active Boolean True to activate, false to deactivate.
$consolidate Boolean, Optional True to merge changes on the same property into one, false to keep every single change. Default is false.n See Section 3.4 Consolidation for more explanation.

Returns nothing.

Consolidation

As mentioned in Section Start Change Tracking:, there is the option to consolidate changes when activating change tracking. If consolidation is activated, multiple UPDATEs on the same property or collection element are merged into one, and ADD and REMOVE operations that cancel each other are removed.
The consolidation mechanism is especially useful if there are potentially many changes per property, because of an AJAX update area that submits a user’s changes as that person types, for example. Using the mechanism, intermediate changes are removed to save value store size, and the final list of changes displayed to the user or archived does not show unnecessary intermediate changes.

You might want to use consolidation but have specific points in your process beyond which you don’t want any consolidations to span, for example, swimlane transitions. If a user makes a change to a person’s address, and another user changes this address again, you might not want to merge these two changes. To define such a point, you can use the Data Audit Consolidation Savepoint trackpoint, or the DataAuditSetConsolidationSavepoint function:

Copy 
DataAuditSetConsolidationSavepoint($object)

The function has the following parameter:

  • $object (Any) — The root entity of the entity graph that is affected by the operation. A consolidation savepoint is set on all entities reachable from the $object. There will be no consolidation of collected changes with new changes on these entities. Returns nothing.

The trackpoint works analogously.

Retrieve Changes

To retrieve the collected changes, three functions are available:

  • DataAuditGetChangeEntriesForObject
  • DataAuditGetChangeEntriesForList
  • DataAuditGetChangeEntriesForMap

When calling one of these functions, supply the root of the entity graph for which you want to retrieve the tracked changes. The functions return an indexed collection of all the changes on that graph. The changes are retrieved in a breadth-first search traversal of the entity graph, starting from the supplied entity. The indexed collection is sorted according to entity, from higher level down to lower level. Changes for the same entity are in the order they were performed.

DataAuditChange Data Class

The DataAuditChange Data Class represents tracked changes. It is deployed by the extension and has the following properties:

Property Description
objectValueAny The changed entity. Note that this is just a reference to the changed entity, not a snapshot at the time of the change.
action String The change action performed. One of "UPDATE", "ADD", or "REMOVE".

key

String
  • For changes on data class properties: The name of the property: "firstname" for a change of the "firstname" property of a person, for example.
  • For changes on indexed collections: The index at which the change was performed, e.g. "1" for an insertion of an element at the first position. Note that this is the index at the time of the change. The inserted element may have been shifted afterwards due to additional insertions or removals.
  • For changes on named collections: The key on which the change was performed, e.g. "someKey" for an insertion, removal, or update of the value with key "someKey" in the collection.

path

String

 A user-readable path to the location of the change relative to the root entity supplied when retrieving the changes.nFor a change to the first line of a person’s address, for example, it could be "Address [1] / Street." The property names in the paths are the ones configured as "Field display name" (see Configuration of Tracked Classes and Properties)

oldValue

Any

 The old value (relevant only for UPDATEs and REMOVEs). Note that this is just a reference to the old value, not a snapshot at the time of the change.

oldDisplayValue

String

 Display value of the old value (relevant only for UPDATEs and REMOVEs) evaluated at the time of the change based on the configured "Field display value expression" or "Data Class Display Expression" (see Configuration of Tracked Classes and Properties).

newValue

Any

 The new value (relevant only for UPDATEs and ADDs). Note that this is just a reference to the new value, not a snapshot at the time of the change. |**newDisplayValue**n String | Display value of the new value (relevant only for UPDATEs and ADDs) evaluated at the time of the change based on the configured "Field display value expression" or "Data Class Display Expression" (see Configuration of Tracked Classes and Properties ).

changerId

String

 The user ID of the user who performed the change. n May be null if the change has been performed by a background activity (e.g. a scheduled job).

entityType

String

 User-supplied description of the type of the tracked root entity (see DataAuditGetChangeEntriesForObject )

entityId

String

 User-supplied ID of the tracked root entity (see DataAuditGetChangeEntriesForObject )

objectType

String

 User-readable type of the changed entity ("Person" or "Indexed Address", for example)
objectDisplayValue Display value of the changed entity at the time when the changes are retrieved (after all changes are applied). nComputed based on the configured "Data Class Display Expression" (see Configuration of Tracked Classes and Properties ).

 

DataAuditGetChangeEntriesForObject

Collect the changes for all entities reachable from the supplied data object.

Copy 
DataAuditGetChangeEntriesForObject($object, $entityType, $entityId, $date)
Parameter Description
$object n Any The root entity of the entity graph for which to retrieve the changes. All tracked changes performed on entities reachable from the $object are returned.
$entityType nString A String that specifies what the $object represents, e.g. "person", "party" or "account".n This String is set on each of the returned changes and can be useful for display purposes, such as when displaying a concatenated collection of changes for different root entities. Can be null.
$entityId nString A String that specifies a user-readable ID for the $object. Useful together with $entityType. Can be null.
$date nDate, optional An optional date to be set on every change. If omitted or null, the actual change dates are kept.
Return value Description
Indexed DataAuditChange List of changes grouped by the subject of the change (entity on which a property/element was changed). The primary sort order is breadth-first search order of the subjects in the entity graph (i.e. entities on higher levels come before nested entities), and the secondary sort order is the time of the change (i.e. changes for the same subject are sorted chronologically).

DataAuditGetChangeEntriesForList / DataAuditGetChangeEntriesForMap

Collect the changes on the supplied collection and all contained entity graphs. The following description refers to the DataAuditGetChangeEntriesForList function. The DataAuditGetChangeEntriesForMap function is completely analogous.

Copy 
DataAuditGetChangeEntriesForList($object, $entityType, $entityId, $loopVariable, $date)
Parameter Description
$collectionn Indexed Any The root collection of the entity graph for which to retrieve the changes. All tracked changes performed on entities reachable from the $collection are returned.
$entityType n Script A Script that specifies what each element in the $collection represents, e.g., “person”, “party”, or “account”. This String is set on each of the returned changes and can be useful for display purposes when displaying a concatenated collection of changes for different root entities. Use the $loopVariable for dynamic expressions. Can be null.
$entityId nScript A Script that specifies a user-readable ID for each element inside the $collection. Use the $loopVariable for dynamic expressions. Useful together with $entityType. Can be null.
$loopVariablen Variable A variable used as a container for the current value (not the index) while iterating through the collection. Can be used for dynamic entityTypes and entityIds.
$daten Date, optional An optional date to be set on every change. If omitted or null, the actual change dates are kept.
Return value Description
Indexed DataAuditChange List of changes grouped by the subject of the change (entity on which a property/element was changed). The primary sort order is breadth-first search order of the subjects in the entity graph (i.e. entities on higher levels come before nested entities), and the secondary sort order is the time of the change (i.e. changes for the same subject are sorted chronologically).

Impact and Limitations

While collecting the changes, the DataAuditTrail extension has a small performance impact. Changes and old values are kept in the value store, which is why the extension has a certain overhead in value store size. The change objects are small, however, and consolidation can be used to avoid generating too many changes.

Retrieving the changes requires traversing all entities reachable from the tracked root entity. Try to avoid calling the corresponding functions often. Instead, try to retrieve all changes at once by using a common parent as the root to start the collection.

A current limitation is that changes cannot be removed. The extension is therefore better suited for mutation processes where the changes should be collected up to the end of the process, but not for use cases where changes are needed only temporarily and are then removed.

GDPR Compliance

The General Data Protection Regulation (GDPR) is an EU regulation aimed at protecting the privacy of natural persons regarding the processing of personal data and the free movement of such data. The purpose of GDPR, among others, is to strengthen and to unify data protection laws within the EU for individuals, ensuring individuals have control of their personal data.

Important! The DataAuditTrail extension stores data and as such it can be used to persist information on data subjects (natural persons). Implementing a data persistence strategy that is compliant with the regulations stipulated by GDPR must be guaranteed by the implementation of the Solution by the customer or entity using this product. Compliance is not inherent in any way to the product itself.

This document and the content thereof and any information derived from it is not to be construed as legal, regulatory and/or tax advice. Appway is not permitted to and will not provide any legal, regulatory and/or tax advice. Appway is also not providing an interpretation of any laws or regulations that may be applicable to any customer, prospect, or anyone, and they shall be responsible for clarifying and stipulating the legal and regulatory requirements applicable to its business. While Appway personnel providing the services may, through experience or specialized training or both, be familiar with the general regulatory environment in their capacity as information-technology and management-consulting professionals, they will work under the direction of the customer and its legal counsels regarding the specific legal and regulatory requirements under which the customer operates. The compliance of the services and work results with the applicable laws and requirements remains the sole responsibility of customer and their legal, regulatory and/or tax advisors.