Struct StrokeModeler

pub struct StrokeModeler { /* private fields */ }

This class models a stroke from a raw input stream. The modeling is performed in several stages

• Wobble smoothing : dampens high-frequency noise from quantization error
• Position modeling : models the pen tip as a mass, connected by a spring, to a moving anchor
• Stylus state modeling : constructs stylus states for modeled positions by interpolating over the raw input

Additional, this class provides prediction of the modeled stroke

StrokeModeler is unit-agnostic

Implementations

impl StrokeModeler

Notations

We denote :

• Pressure : $\nu \in \lbrack 0,1\rbrack$

• time : $t \geq 0$

• point : defined by position $x$ and $y$ (or $p = (x,y) \in {\mathbb{R}}^{2}$)

• Raw inputs are denoted by a tuple $i\lbrack k\rbrack = \left( \nu\lbrack k\rbrack,t\lbrack k\rbrack,x\lbrack k\rbrack,y\lbrack k\rbrack \right)$ with $k \in {\mathbb{N}}$.

An input stream is a sequence of raw inputs $i = (\nu,t,x,y)$

• with time $t\lbrack k\rbrack$ $\nearrow \nearrow$ strictly increasing

• starts with a Down event

• contains Move for $k \geq 1$

• ends either with a Move or a Up . If it is a Up we say the input stream is complete

We addition define

• $v_{x},v_{y},a_{x},a_{y}$ as the velocity and acceleration.

With the vector shorthand $v = \left( v_{x},v_{y} \right) \in {\mathbb{R}}^{2}$ and $a = \left( a_{x},a_{y} \right) \in {\mathbb{R}}^{2}$

Resampling

Algorithm : Upsampling
Input : $\left\{ v\lbrack k\rbrack = \left( x\lbrack k\rbrack,y\lbrack k\rbrack,t\lbrack k\rbrack \right),0 \leq k \leq n \right\}$ and a target rate sampling_min_output_rate, Interpolate time and position between each $k$ by linearly adding interpolated values
This is done by adding linearly interpolated values so that the output stream is $$\left\{ v\lbrack 0\rbrack,\underset{\text{ interpolated}}{\underbrace{u_{0}\lbrack 1\rbrack,\ldots,u_{0}\left\lbrack n_{0} - 1 \right\rbrack}},v\lbrack 1\rbrack,\underset{\text{ interpolated}}{\underbrace{u_{1}\lbrack 1\rbrack,\ldots,u_{1}\left\lbrack n_{1} - 1 \right\rbrack}},v\lbrack 2\rbrack,\ldots \right\}$$ Each $n_{i}$ is the minimum integer such that $$\begin{aligned} & \frac{t\lbrack i\rbrack - t\lbrack i - 1\rbrack}{n_{i}} < \Delta_{\text{target}} \\ \Leftrightarrow & n_{i} = \left\lceil \frac{t\lbrack i + 1\rbrack - t\lbrack i\rbrack}{\Delta_{\text{target}}} \right\rceil \end{aligned}$$ and the linear interpolation means $$u_{j}\lbrack k\rbrack = \left( 1 - \frac{k}{n_{j}} \right)v\lbrack j\rbrack + \frac{k}{n_{i}}v\lbrack j + 1\rbrack$$
Output : $\left\{ \left( x\prime\lbrack k\prime\rbrack,y\prime\lbrack k\prime\rbrack,t\prime\lbrack k\prime\rbrack \right),0 \leq k; \leq n\prime \right\}$ the upsampled position and times. This verifies $$\forall k\prime,t\prime\lbrack k\prime\rbrack - t\prime\lbrack k\prime - 1\rbrack < \Delta_{\text{target }} = \frac{1}{\text{sampling_min_output_rate}}$$
Remark : As this is a streaming algorithm, we only calculate this interpolation with respect to the latest stroke position.

Position modeling

The raw input is processed as follow

raw input $\rightarrow$ wobble smoother $\rightarrow$ upsampled $\rightarrow$ position modeling

The position of the pen is modeled as a weight connected by a spring to an anchor.

The anchor moves along the resampled dewobbled inputs, pulling the weight along with it across a surface, with some amount of friction. Euler integration is used to solve for the position of the pen.

The physical model that is used to model the stroke is the following $$\frac{d^{2}s}{dt^{2}} = \frac{\Phi(t) - s(t)}{k_{\text{spring }}} - k_{\text{drag }}\frac{ds}{dt}$$ where

• $t$ is time

• $s(t)$ is the position of the pen

• $\Phi(t)$ is the position of the anchor

• $k_{\text{spring}}$ and $k_{\text{drag}}$ are constants that sets how the spring and drag occurs

$k_{\text{spring}}$ is given by position_modeler_spring_mass_constant and $k_{\text{drag}}$ by position_modeler_drag_constant

We will thus have as input the upsampled dewobbled inputs taking the role of discretized $\Phi(t)$ and $s(t)$ will be the output

Modeling a stroke.

• Input : input stream $\left\{ \left( p\lbrack k\rbrack,t\lbrack k\rbrack \right),0 \leq k \leq n \right\}$

• Output : smoothed stream $\left\{ \left( p_{f}\lbrack k\rbrack,v_{f}\lbrack k\rbrack,a_{f}\lbrack k\rbrack \right),0 \leq k \leq n \right\}$

We define $\Phi\lbrack k\rbrack = p\lbrack k\rbrack$. An euler scheme integration scheme is used with the initial conditions being $v\lbrack 0\rbrack = 0$ and $p_{f}\lbrack 0\rbrack = p\lbrack 0\rbrack$ (same initial conditions)

Update rule is simply $$\begin{array}{r} a_{f}\lbrack j\rbrack = \frac{p\lbrack j\rbrack - p_{f}\lbrack j - 1\rbrack}{k_{\text{spring }}} - k_{\text{drag }}v_{f}\lbrack j - 1\rbrack \\ v_{f}\lbrack j\rbrack = v_{f}\lbrack j - 1\rbrack + \left( t\lbrack j\rbrack - t\lbrack j - 1\rbrack \right)a_{f}\lbrack j\rbrack \\ p_{f}\lbrack j\rbrack = p_{f}\lbrack j - 1\rbrack + \left( t\lbrack j\rbrack - t\lbrack j - 1\rbrack \right)v_{f}\lbrack j\rbrack \end{array}$$ The position $s\lbrack j\rbrack$ is the main thing to export but we can also export speed and acceleration if needed. We denote $$q\lbrack j\rbrack = \left( p_{f}\lbrack j\rbrack,v_{f}\lbrack j\rbrack,a_{f}\lbrack j\rbrack,t\lbrack j\rbrack \right)$$ and this will be our output with $0 \leq j \leq n$

Stylus state modeler

Up till now we have only used the raw input stream to create a new smoothed stream of positions, leaving behind the pressure attribute. This is what’s done here, to model the state of the stylus for these new position based on the pressure data of the raw input strokes.
Algorithm Stylus state modeler
Input :

• input stream with pressure information $\left\{ \left( p\lbrack k\rbrack = \left( x\lbrack k\rbrack,y\lbrack k\rbrack \right),\nu\lbrack k\rbrack \right),0 \leq k \leq n \right\}$

• query position $q = (x,y)$

• search window $n_{\text{search}}$ (From stylus_state_modeler_max_input_samples),

initialize $d = \infty$, $\text{index} = \text{None}$, $\text{interp } = \text{ None}$
for $i = n - n_{\text{search}}$ to $n - 1$ do
- Find $q_{i}$ the position that’s closest to $q$ on the segment $\left\lbrack p\lbrack i\rbrack,p\lbrack i + 1\rbrack \right\rbrack$ and denote $r \in \lbrack 0,1\rbrack$ the value such that $$q_{i} = (1 - r)p\lbrack i\rbrack + rp\lbrack i + 1\rbrack$$
- if $\left. \parallel{q - q_{i}} \right.\parallel < d$
- $\begin{array}{r} d \leftarrow \left. \parallel{q - q_{i}} \right.\parallel < d \\ \text{index } = i \\ \text{interp } = r \end{array}$
- endif
endfor
calculate $$\nu = (1 - r)\nu\left\lbrack \text{index} \right\rbrack + r\nu\left\lbrack \text{index } + 1 \right\rbrack$$
Output : interpolated pressure $\nu$

Stroke end

The position modeling algorithm will lag behind the raw input by some distance. This algorithm iterates the previous dynamical system a few additional time using the raw input position as the anchor to allow a catch up of the stroke (though this prediction is only given by predict, so is not part of the results and becomes obsolete on the next input).

Algorithm :Stroke end
Input:

• Final anchor position $p\left\lbrack \text{end} \right\rbrack = \left( x\left\lbrack \text{end} \right\rbrack,y\left\lbrack \text{end} \right\rbrack \right)$ (From the original input stream)

• final tip state $q_{f}\left\lbrack \text{end} \right\rbrack = \left( p_{f}\left\lbrack \text{end} \right\rbrack = \left( x_{f}\left\lbrack \text{end} \right\rbrack,y_{f}\left\lbrack \text{end} \right\rbrack \right),v_{f}\left\lbrack \text{end} \right\rbrack,a_{f}\left\lbrack \text{end} \right\rbrack \right)$ (returned from the physical modeling from the last section, $\cdot_{f}$ signifies that we are looking at the filtered output)

• $K_{\text{max}}$ max number of iterations (sampling_end_of_stroke_max_iterations)

• $\Delta_{\text{target}}$ the target time delay between stroke (1/sampling_min_output_rate)

• $d_{\text{stop}}$ stopping distance (sampling_end_of_stroke_stopping_distance)

• $k_{\text{spring}}$ and $k_{\text{drag}}$ the modeling coefficients

initialize the vector $q_{o}$ with $q_{0}\lbrack 0\rbrack = \left. \left( \underset{p_{o}\lbrack 0\rbrack}{\underbrace{p_{f}\left\lbrack \text{end} \right\rbrack}},\underset{v_{o}\lbrack 0\rbrack}{\underbrace{v_{f}\left\lbrack \text{end} \right\rbrack}},\underset{a_{0}\lbrack 0\rbrack}{\underbrace{a_{f}\left\lbrack \text{end} \right\rbrack}} \right) \right.$
initialize $\Delta t = \Delta_{\text{target}}$
for $1 \leq k \leq K_{\text{max}}$
- calculate the next candidate $$\begin{aligned} a_{c} & = \frac{p\left\lbrack \text{end} \right\rbrack - p_{o}\left\lbrack \text{end} \right\rbrack}{k_{\text{spring}}} - k_{\text{drag }}v_{0}\left\lbrack \text{end} \right\rbrack \\ v_{c} & = v_{o}\lbrack 0\rbrack + \Delta ta_{c} \\ p_{c} & = p_{o}\lbrack 0\rbrack + \Delta v_{c} \end{aligned}$$

- if $\left. \parallel{p_{c} - p\left\lbrack \text{end} \right\rbrack} \right.\parallel < d_{\text{stop}}$ (further iterations won’t be able to catch up and won’t move closer to the anchor, we stop here), - return $q_{0}$
- endif

• if $\langle p_{c} - p_{o}\left\lbrack \text{end} \right\rbrack,p\left\lbrack \text{end} \right\rbrack - p_{o}\left\lbrack \text{end} \right\rbrack\rangle < \left. \parallel{p_{c} - p_{o}\left\lbrack \text{end} \right\rbrack} \right.\parallel$ (we’ve overshot the anchor, we retry with a smaller step)

  
  

  • $\Delta t \leftarrow \frac{\Delta t}{2}$,

  • continue (this candidate will be discarded, try again with a smaller time step instead),

- else
- $q_{0}\lbrack end\ +1\rbrack = \left( p_{c},v_{c},q_{c} \right)$ (We append the result to the end of the $q_{0}$ vector)
- endif

• if $\left. \parallel{p_{c} - p\left\lbrack \text{end} \right\rbrack} \right.\parallel < d_{\text{stop}}$,

  • return (We are within tolerance of the anchor, we stop iterating),

• endif,

Output : $\left\{ q_{o}\lbrack k\rbrack = \left( s_{o}\lbrack k\rbrack,v_{o}\lbrack k\rbrack,a_{o}\lbrack k\rbrack \right),0 \leq k \leq n\left( \leq K_{\text{max }} - 1 \right) \right\}$

pub fn new(params: ModelerParams) -> Result<Self, String>

pub fn reset(&mut self)

Clears any in-progress stroke, keeping the same model parameters

pub fn reset_w_params(&mut self, params: ModelerParams) -> Result<(), String>

Clears any in-progress stroke, and re initialize the model with the given parameters

pub fn update( &mut self, input: ModelerInput, ) -> Result<Vec<ModelerResult>, ModelerError>

Updates the model with a raw input, and appends newly generated Results to the results vector. Any previously generated Result values remain valid. (This does not require that any previous results returned remain in the results vector, as it is appended to without examining the existing contents)

If this does not return an error, results will contain at least one Result, and potentially more if the inputs are slower than the minimum output rate

pub fn predict(&mut self) -> Result<Vec<ModelerResult>, String>

Models the given input prediction without changing the internal model state

Returns an error if the model has not yet been initialized, if there is no stroke in progress

Trait Implementations

impl Default for StrokeModeler

fn default() -> Self

Returns the “default value” for a type.