TimeseriesUtilities

TimeseriesUtilities

A collection of utilities to simplify common time series analysis.

From data cleaning to arithmetic operations (e.g. linear algebra) to common time series operations (e.g. resampling, filtering).

Data Cleaning

• find_outliers, find_outliers_median, find_outliers_mean
• replace_outliers, replace_outliers!

Query

• times, time_grid
• timerange, common_timerange

(Windowed) Statistics

• tstat
• tmean
• tmedian
• tsum
• tvar
• tstd
• tsem

Algebra

• tcross
• tdot
• tnorm
• tsproj, tproj, toproj
• tsubtract
• tderiv

Time-Domain Operations

• tselect
• tclip, tclips
• tview
• tmask and tmask!
• tshift
• tsplit
• tgroupby

Time-Frequency Domain Operations

• tfilter

TimeseriesUtilities

A collection of utilities to simplify common time series analysis.

From data cleaning to arithmetic operations (e.g. linear algebra) to common time series operations (e.g. resampling, filtering).

Data Cleaning

• find_outliers, find_outliers_median, find_outliers_mean
• replace_outliers, replace_outliers!

Query

• times, time_grid
• timerange, common_timerange

(Windowed) Statistics

• tstat
• tmean
• tmedian
• tsum
• tvar
• tstd
• tsem

Algebra

• tcross
• tdot
• tnorm
• tsproj, tproj, toproj
• tsubtract
• tderiv

Time-Domain Operations

• tselect
• tclip, tclips
• tview
• tmask and tmask!
• tshift
• tsplit
• tgroupby

Time-Frequency Domain Operations

• tfilter

DiffQ(v, t; dim=1)

Difference quotient of v with respect to t.

To avoid undefined behavior for division by Date/DateTime, we convert the time difference to a Unitful.Quantity if eltype(v) is not a Unitful.Quantity.

common_timerange(x1, xs...)

Get the common time range (intersection) across multiple arrays. If there is no overlap, returns nothing.

dimnum(x, query)

Get the number(s) of Dimension(s) as ordered in the dimensions of an object.

Extend the function for custom type x. By default, we fall back to DimensionalData.dimnum.

dropna(A; dim=nothing)
dropna(A::AbstractDimArray; dim=nothing, query=nothing)

Remove slices containing NaN values along along the dim dimension.

find_outliers(A, [method, window]; dim = 1, kw...)

Find outliers in data A along the specified dim dimension.

Returns a Boolean array whose elements are true when an outlier is detected in the corresponding element of A.

The default method is :median (other option is :mean), which uses the median absolute deviation (MAD) to detect outliers. When the length of A is greater than 256, it uses a moving window of size 16.

See also: find_outliers_median, find_outliers_mean, isoutlier - MATLAB

find_outliers_mean(x::AbstractVector, window; threshold = 3)

Find outliers that are defined as elements more than three standard deviations from the mean.

This method is faster but less robust than find_outliers_median.

find_outliers_median(x, window; threshold=3)

Find outliers that are defined as elements more than threshold=3 times the scaled median absolute deviation (MAD) from the median.

When window is set to a integer, a moving window of that size is used to compute local MAD. Otherwise, global statistics are used.

References

• Median absolute deviation - Wikipedia

Group x into windows based on every and period.

interpolate_outliers!(x, t, outliers)

Interpolate outliers in x using interpolation of neighboring non-outlier values.

Vector rejection

proj(a, b)

Vector projection of a vector a on (or onto) a nonzero vector b.

References: Wikipedia

See also: sproj, oproj

Rectify the time step of a DimArray to be uniform.

replace_outliers!(A, s, [find_method, window]; kwargs...)

Finds outliers in A and replaces them with s (by default: NaN).

See also: find_outliers, filloutliers - MATLAB

replace_outliers!(A, method, [find_method, window]; kwargs...)
replace_outliers!(A, method, outliers; kwargs...)

Replaces outliers in A with values determined by the specified method.

Outliers can be detected using find_outliers with optional find_method and window parameters or specified directly as a Boolean array outliers.

method can be one of the following:

• :linear: Linear interpolation of neighboring, nonoutlier values
• :previous: Previous nonoutlier value
• :next: Next nonoutlier value
• :nearest: Nearest nonoutlier value

See also: filloutliers - MATLAB

replace_outliers(A; args...; kw...)

Non-mutable version of replace_outliers!.

smooth(da::AbstractDimArray, window; dim=Ti, suffix="_smoothed", kwargs...)

Smooths a time series by computing a moving average over a sliding window.

The size of the sliding window can be either:

• Quantity: A time duration that will be converted to number of samples based on data resolution
• Integer: Number of samples directly

Arguments

• dims=Ti: Dimension along which to perform smoothing (default: time dimension)
• suffix="_smoothed": Suffix to append to the variable name in output
• kwargs...: Additional arguments passed to RollingWindowArrays.rolling

Scalar projection

tclip(A, t0, t1; query=nothing, sort=false)

Clip a Dimension or DimArray A to a time range [t0, t1].

For unordered dimensions, the dimension should be sorted before clipping (see tsort).

tclips(xs...; trange=common_timerange(xs...))

Clip multiple arrays to a common time range trange.

If trange is not provided, automatically finds the common time range across all input arrays.

tcross(x, y; dim = nothing, query=nothing)

Compute the cross product of two (arrays of) vectors along the specified dimension dim or query.

References:

• https://docs.xarray.dev/en/stable/generated/xarray.cross.html

tderiv(A, times; dim=1)
tderiv(A; dim = nothing, query = nothing)

Compute the time derivative of A. Set lazy=true for lazy evaluation.

See also: deriv_data - PySPEDAS

tdot(x, y; dim=nothing, query=nothing)

Dot product of two arrays x and y along the dim dimension.

tfilter(da, Wn1, Wn2=samplingrate(da) / 2; designmethod=nothing)

By default, the max frequency corresponding to the Nyquist frequency is used.

References

• https://docs.juliadsp.org/stable/filters/
• https://www.mathworks.com/help/signal/ref/filtfilt.html
• https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.filtfilt.html

Issues

• DSP.jl and Unitful.jl: https://github.com/JuliaDSP/DSP.jl/issues/431

tgroupby(x, every, period = every, start_by = :window)

Group x into windows based on every and period.

time_grid(x, dt)

Create a time grid from the minimum to maximum time in x with the step size dt.

Examples

# Create hourly time grid
time_grid(x, Hour(1))
time_grid(x, 1u"hr")

# Create 1-s intervals
time_grid(x, Second(1))
time_grid(x, 1u"second")
time_grid(x, 1u"Hz")

timerange(times)
timerange(x1, xs...)

Get the time range (minimum and maximum) of time series data.

For a single argument, returns a tuple (tmin, tmax) containing the minimum and maximum times. For multiple arguments, returns the common time range (intersection) across all arrays - equivalent to common_timerange(x1, xs...).

Examples

# Single time series
times = [1, 2, 3, 4, 5]
timerange(times)  # (1, 5)

# Multiple time series - find common range
x1_times = [1, 2, 3, 4]
x2_times = [2, 3, 4, 5]
timerange(x1_times, x2_times)  # (2, 4)

See also: common_timerange, tminimum, tmaximum

Returns the time indices of x.

tmask!(da, t0, t1)
tmask!(da, it::Interval)
tmask!(da, its)

Mask all data values within the specified time range(s) (t0, t1) / it / its with NaN.

tmask(da, args...; kwargs...)

Non-mutable version of tmask!. See also tmask!.

tmaximum(x)

Get the maximum timestamp of x.

tmean(x, [dt]; dim=nothing, query=nothing)

Calculate the arithmetic mean of x along the dim dimension, optionally grouped by dt.

It returns a value if x is a vector along the dim dimension, otherwise returns a DimArray with the specified dimension dropped.

If dim is not specified, it defaults to the query dimension (dimension of type TimeDim by default).

tmedian(x, [dt]; dim=nothing, query=nothing)

Calculate the median of x along the dim dimension, optionally grouped by dt.

It returns a value if x is a vector along the dim dimension, otherwise returns a DimArray with the specified dimension dropped.

If dim is not specified, it defaults to the query dimension (dimension of type TimeDim by default).

tminimum(x)

Get the minimum timestamp of x.

tnorm(A; dim=nothing, query=nothing)

Compute the norm of each slice in A along the specified dimension dim or query.

See also: tnorm_combine

tnorm_combine(x; dim=nothing, name=:magnitude)

Calculate the norm of each slice along query dimension and combine it with the original components.

toproj(A, B; dim=nothing, query=nothing)

Compute vector rejection (orthogonal projection) of array A from B along specified dimension dim or query.

tproj(A, B; dim=nothing, query=nothing)

Compute vector projection of A onto B along specified dimension dim or query.

tselect(A, t, [δt]; query=nothing)

Select the value of A closest to time t within the time range [t-δt, t+δt].

Similar to DimensionalData.Dimensions.Lookups.At but choose the closest value and return missing if the time range is empty.

tsem(x, [dt]; dim=nothing, query=nothing)

Calculate the standard error of the mean of x along the dim dimension, optionally grouped by dt.

It returns a value if x is a vector along the dim dimension, otherwise returns a DimArray with the specified dimension dropped.

If dim is not specified, it defaults to the query dimension (dimension of type TimeDim by default).

tshift(x; dim=TimeDim, t0=nothing, new_dim=nothing)

Shift the dim of x by t0.

tsort(A; query=nothing, rev=false)

Sort a DimArray A along the query dimension.

tsplit(da::AbstractDimArray, dim=Ti)

Splits up data along dimension dim.

tsproj(A, B; dim=nothing, query=nothing)

Compute scalar projection of A onto B along specified dimension dim or query.

tstat(f, x, [dt]; dim = nothing)

Calculate the statistic f of x along the dim dimension, optionally grouped by dt.

See also: groupby_dynamic

tstd(x, [dt]; dim=nothing, query=nothing)

Calculate the standard deviation of x along the dim dimension, optionally grouped by dt.

It returns a value if x is a vector along the dim dimension, otherwise returns a DimArray with the specified dimension dropped.

If dim is not specified, it defaults to the query dimension (dimension of type TimeDim by default).

tsubtract(x, f=nanmedian; dims=timedim(x))

Subtract a statistic (default function f: nanmedian) along dimensions (default: time dimension) from x.

tsum(x, [dt]; dim=nothing, query=nothing)

Calculate the sum of x along the dim dimension, optionally grouped by dt.

It returns a value if x is a vector along the dim dimension, otherwise returns a DimArray with the specified dimension dropped.

If dim is not specified, it defaults to the query dimension (dimension of type TimeDim by default).

tvar(x, [dt]; dim=nothing, query=nothing)

Calculate the variance of x along the dim dimension, optionally grouped by dt.

It returns a value if x is a vector along the dim dimension, otherwise returns a DimArray with the specified dimension dropped.

If dim is not specified, it defaults to the query dimension (dimension of type TimeDim by default).

tview(d, t0, t1)

View a dimension or DimArray in time range [t0, t1].

unwrap(x)

Return the innermost object of the wrapped object x with similar behavior as x (e.g. same size, same type, etc.)

window_bf_sizes(window)

Converts a window specification to backward and forward window sizes.

When window is a positive integer scalar, the window is centered about the current element and contains window-1 neighboring elements. If window is even, then the window is centered about the current and previous elements.