Merge topic 'implicit-array-review'

aa00e965d0 review(ImpArr): improve changelog
650ca6f387 doc(dispatch): add doc on Indexed and Composite array dispatch
33155d6843 improve(ImplicitArray): disable inheritance of certain backends
1f21217cb4 group(changelog): for `vtkImplicitArray` types into a single .md

Acked-by: Kitware Robot <kwrobot@kitware.com>
Acked-by: buildbot <buildbot@kitware.com>
Acked-by: Mathieu Westphal <mathieu.westphal@kitware.com>
Merge-request: !9720
dev
Julien Fausty 2 months ago committed by Kitware Robot
commit 2622e4a0d5

@ -41,7 +41,7 @@
* ```
*/
template <typename ValueType>
struct vtkAffineImplicitBackend
struct vtkAffineImplicitBackend final
{
/**
* A non-trivially constructible constructor

@ -40,7 +40,7 @@
* ```
*/
template <typename ValueType>
struct vtkConstantImplicitBackend
struct vtkConstantImplicitBackend final
{
/**
* A non-trivially contructible constructor

@ -394,6 +394,12 @@ currently exist for use with the VTK dispatch mechanism:
`vtkConstantArray` as part of the implicit array framework
* `VTK_DISPATCH_STD_FUNCTION_ARRAYS` (default `OFF`): includes dispatching for arrays with
an `std::function` backend `vtkStdFunctionArray` as part of the implicit array framework
* `VTK_DISPATCH_COMPOSITE_ARRAYS` (default `OFF`): includes dispatching of arrays with a
`vtkCompositeImplicitBackend` backend, `vtkCompositeArray`, as part of the implicit array
framework
* `VTK_DISPATCH_INDEXED_ARRAYS` (default `OFF`): includes dispatching of arrays with a
`vtkIndexedImplicitBackend` backend, `vtkIndexedArray`, as part of the implicit array
framework
The outlier in terms of dispatch support is the family of arrays derived from
`vtkScaledSOADataArrayTemplate` which are automatically included in dispatch when built setting

@ -1,26 +0,0 @@
## `vtkCompositeArray`: a new implicit array that concatenates other arrays together
The new `vtkCompositeArray` is a family of `vtkImplicitArray`s that can concatenate arrays together to interface a group of arrays as if they were a single array. This concatenation operates in the "tuple" direction and not in the "component" direction.
This new array relies on the `vtkCompositeImplicitBackend` template class to join two `vtkDataArray`s at a time. Creating a hiearchy of `vtkCompositeArray`s can generate a binary tree on the indexes of the composite array leading to access with $O(log_2(m))$ time where $m$ is the number of leaves (or base `vtkDataArray`s) composing the composite (or alternatively $O(l)$ where $l$ is the number of levels in the tree).
To facilitate the creation of `vtkCompositeArray`s in practice, a templated utility function `vtkCompositeArrayUtilities::Concatenate` has been made available to users that can take an `std::vector` of `vtkDataArray`s and turn them into a single concatenated `vtkCompositeArray` whose tree structure should be balanced with respect to number of arrays (a possible improvement would be to balance with respect to number of tuples following a "Huffman coding" approach).
A code snippet using this type of array:
```
std::vector<vtkDataArray*> baseArrays(16); // 16 == 2^4, will make 4 levels in binary tree
vtkNew<vtkDoubleArray> baseArray;
baseArray->SetNumberOfComponents(3);
baseArray->SetNumberOfTuples(10);
baseArray->Fill(0.0);
std::fill(baseArrays.begin(), baseArrays.end(), baseArray);
vtkSmartPointer<vtkCompositeArray<double>> composite = vtkCompositeArrayUtilities::Concatenate<double>(baseArrays); // nTuples = 160
CHECK(composite->GetComponent(42, 1) == 0.0); // always true
```
> **WARNINGS**
>
> * Any two arrays composited into a `vtkCompositeArray` using the `vtk::ConcatenateDataArrays` method must have the same number of components.
> * Iteration over the composited array incurs a lot of overhead compared to an explicit memory array (~3x slower with only 1 level). The use case is truly when memory efficiency is more important than compute performance.

@ -1,12 +1,16 @@
# New vtkImplicitArrays!
## New vtkImplicitArrays!
## Description
### Description
VTK now offers new flexible `vtkImplicitArray` template class that implements a **read-only** `vtkGenericDataArray` interface. It essentially transforms an implicit function mapping integers to values into a practically zero cost `vtkDataArray`. This is helpful in cases where one needs to attach data to data sets and memory efficiency is paramount.
## Usage
### Philosophy
The `vtkImplicitArray` is templated on the backend that is "duck typed" so that it can be any const functor/closure object or anything that has a `map(int) const` method. Here is a small example using a constant functor in an implicit array:
In order to reduce the overhead of these containers as much as possible, `vtkImplicitArray`s are templated on a "Backend" type. This backend is "duck typed" so that it can be any const functor/closure object or anything that has a `map(int) const` method to provide a certain degree of flexibility and compute performance that is easily obtained through the static dispatching native to templates. As such, developers can use tried and tested backends in the VTK framework when it fits their needs and also develop their own backends on the fly for specific use cases. `vtkImplicitArray`s can then be packed into data sets to be transmitted through the data treatment pipeline with the idea that calls to the "read-only" API of `vtkGenericDataArray` should be inlined through the implicit array and generate close to zero overhead with respect to calling the backend itself.
### Usage
Here is a small example using a constant functor in an implicit array:
```
struct ConstBackend
@ -20,11 +24,13 @@ arr42->SetNumberOfTuples(100);
CHECK(arr42->GetValue(77) == 42); // always true
```
For convenience, a number of backends have been pre-packed into the `vtkImplicitArray` framework and can be included into the dispatch mechanism with the relevant `VTK_DISPATCH_*_ARRAYS`. They are, in no particular order:
For convenience, a number of backends have been pre-packed into the `vtkImplicitArray` framework. They are, in alphabetical order:
- `vtkStdFunctionArray<ValueType>`: using a `std::function<ValueType(int)>` backend capable of covering almost any function one might want to use
- `vtkConstantArray<ValueType>`: using the `vtkConstantImplicitBackend<ValueType>` closure backend that gets constructed with a given value and then returns that same value regardless of the index queried
- `vtkAffineArray<ValueType>`: using the `vtkAffineImplicitBackend<ValueType>` closure backend that gets constructed with a slope and intercept and then returns values linearly depending on the queried index
- `vtkCompositeArray<ValueType>`: using the `vtkCompositeImplicitBackend<ValueType>` closure backend that takes an `std::vector<vtkDataArray*>` at construction and returns values as if the list has been concatenated into one array
- `vtkConstantArray<ValueType>`: using the `vtkConstantImplicitBackend<ValueType>` closure backend that gets constructed with a given value and then returns that same value regardless of the index queried
- `vtkStdFunctionArray<ValueType>`: using a `std::function<ValueType(int)>` backend capable of covering almost any function one might want to use
- `vtkIndexedArray<ValueType>`: using the `vtkIndexedImplicitBackend<ValueType>` closure backend that takes an indexing array (either `vtkIdList` or `vtkDataArray`) and a base `vtkDataArray` at construction and returns values indirected using the indexing array to give access to a shuffled array without the memory cost
Here is a small code snippet example to illustrate the usage of the `vtkConstantArray`:
@ -44,6 +50,76 @@ The read-only parts of the `vtkDataArray` API work out of the box for any `vtkIm
* Standard library like ranges and iterators using `vtkDataArrayRange` functionalities
* `vtkArrayDispatch`, provided the correct compilation options have been set and the correct type list is used for dispatching (see below)
## Building and Dispatch
### Focus on `vtkCompositeArrays`
The `vtkCompositeArray` is a family of `vtkImplicitArray`s that can concatenate arrays together to interface a group of arrays as if they were a single array. This concatenation operates in the "tuple" direction and not in the "component" direction.
This new array relies on the `vtkCompositeImplicitBackend` template class to join an `std::vector` of `vtkDataArray`s. The `vtkCompositeArray`s use internal address referencing and indirection to implement binary search on the indexes of the composite array leading to access with $O(log_2(m))$ time where $m$ is the number of leaves (or base `vtkDataArray`s) composing the composite.
To facilitate the creation of `vtkCompositeArray`s in practice, a templated utility function `vtkCompositeArrayUtilities::Concatenate` has been made available to users that can take an `std::vector` of `vtkDataArray`s (each with the same number of components) and turn them into a single concatenated `vtkCompositeArray` with the same number of components as the base array and a number of tuples being the sum of all the base arrays tuples.
A code snippet using this type of array:
```
std::vector<vtkDataArray*> baseArrays(16);
vtkNew<vtkDoubleArray> baseArray;
baseArray->SetNumberOfComponents(3);
baseArray->SetNumberOfTuples(10);
baseArray->Fill(0.0);
std::fill(baseArrays.begin(), baseArrays.end(), baseArray);
vtkSmartPointer<vtkCompositeArray<double>> composite = vtkCompositeArrayUtilities::Concatenate<double>(baseArrays); // nTuples = 160
CHECK(composite->GetComponent(42, 1) == 0.0); // always true
```
> **WARNINGS**
>
> * Any two arrays composited into a `vtkCompositeArray` using the `vtk::ConcatenateDataArrays` method must have the same number of components.
> * Iteration over the composited array incurs a lot of overhead compared to an explicit memory array (~3x slower with only 1 level). The use case is truly when memory efficiency is more important than compute performance
> * This array has no relationship with the `VTKCompositeDataArray` present in the `numpy_interface.dataset_adapter` module of the python wrapping of VTK
### Focus on `vtkIndexedArray`
The family of `vtkIndexedArray`s allow you to wrap an existing `vtkDataArray` with a layer of indirection through a list of indexes (`vtkIdList` or another `vtkDataArray`) to create a derived subset data array without any excess memory consumption. As such, by providing a `vtkIndexedImplicitBackend` with an indexation array and a `vtkDataArray`, one can effectively construct a reduced and reordered view of the base array.
While using this type of feature to create only one indexed array can be counter productive (allocation of the index array more expensive than an explicit copy of the data might be), using this feature you can share the same index list amoungst multiple indexed arrays effectively using less memory total.
Here is an example use case:
```
vtkNew<vtkIntArray> baseArr;
baseArr->SetNumberOfComponents(3);
baseArr->SetNumberOfComponents(100);
auto range = vtk::DataArrayValueRange<3>(baseArr);
std::iota(range.begin(), range.end(), 0);
vtkNew<vtkIdList> indexes;
indexes->SetNumberOfIds(30);
for (idx = 0; idx < 30; idx++)
{
indexes->SetId(ids, 10*idx);
}
vtkNew<vtkIndexedArray<int>> indexed;
indexed->SetBackend(std::make_shared<vtkIndexedImplicitBackend<int>>(indexes, baseArr));
indexed->SetNumberOfComponents(1);
indexed->SetNumberOfComponents(indexes->GetNumberOfIds());
CHECK(indexed->GetValue(13) == 130); // always true
```
> **WARNINGS**
>
> * Significant access performance hits can be incurred due to cache missing cause by the inherent indirection in the array.
### Implementing a `vtkImplicitArray` in VTK
Implementing a new `vtkImplicitArray` in the VTK library usually passes through the following steps:
* Implementing the backend: this is the step where the underlying functionality of the implicit array needs to be developed.
* Creating an alias and instantiating the concrete types: once the backend is ready, one can create an alias for the array in its own header (`using MySuperArray = vtkImplicitArray<MySuperBackend>`) and use this header and the au tomatic instantiation mechanisms present in `Common/ImplicitArrays/CMakeLists.txt` to make sure that the vtk library gets shipped with pre-compiled objects for your arrays.
* Hooking up the dispatch mechanism: this step ensures that your new arrays can be included in the `vtkArrayDispatch` mechanism when the correct compilation option is set.
* Testing: be sure to set up some tests in order to both verify that your implementation is functioning correctly and to avoid regressions in the future.
An example merge request that was used for including the `vtkIndexedArray`s can be found here: [!9703](https://gitlab.kitware.com/vtk/vtk/-/merge_requests/9703)
### Building and Dispatch
The entire implicit array framework is included in the `CommonImplicitArrays` module. Support for dispatching implicit arrays can be enforced by including type lists from `vtkArrayDispatchImplicitTypeList.h` and compiling VTK with the correct `VTK_DISPATCH_*_ARRAYS` option.

Loading…
Cancel
Save