TimeseriesUtilities

TimeseriesUtilities

A collection of utilities to simplify common time series analysis.

From data cleaning to arithmetic operations (e.g. linear algebra) to common time series operations (e.g. resampling, filtering).

Data Cleaning

• find_outliers, find_outliers_median, find_outliers_mean
• replace_outliers, replace_outliers!

Query

• times, time_grid
• timerange, common_timerange
• cadence — robust modal-cluster Δt, survives gaps, jitter, and dropout

(Windowed) Statistics

• Base: tstat - tstat(f, x, [dt]; dim)
• NaNStatistics wrappers: tmean, tmedian, tsum, tvar, tstd, tsem

Algebra

• tcross, tdot, tnorm
• tsproj, tproj, toproj
• tsubtract, tderiv

Time-Domain Operations

• tselect
• tclip, tclips
• tview
• tmask and tmask!
• tshift
• tsplit
• tgroupby
• Resampling: tinterp, tsync

Time-Frequency Domain Operations

• tfilter, pspectrum

TimeseriesUtilities

A collection of utilities to simplify common time series analysis.

From data cleaning to arithmetic operations (e.g. linear algebra) to common time series operations (e.g. resampling, filtering).

Data Cleaning

• find_outliers, find_outliers_median, find_outliers_mean
• replace_outliers, replace_outliers!

Query

• times, time_grid
• timerange, common_timerange
• cadence — robust modal-cluster Δt, survives gaps, jitter, and dropout

(Windowed) Statistics

• Base: tstat - tstat(f, x, [dt]; dim)
• NaNStatistics wrappers: tmean, tmedian, tsum, tvar, tstd, tsem

Algebra

• tcross, tdot, tnorm
• tsproj, tproj, toproj
• tsubtract, tderiv

Time-Domain Operations

• tselect
• tclip, tclips
• tview
• tmask and tmask!
• tshift
• tsplit
• tgroupby
• Resampling: tinterp, tsync

Time-Frequency Domain Operations

• tfilter, pspectrum

ContinuousTimeRanges(times, max_dt)

Iterator of (t_start, t_stop) spans over times, splitting wherever consecutive values differ by more than max_dt.

DiffQ(v, t; dim=1)

Difference quotient of v with respect to t.

Date/DateTime differences are converted to seconds before division.

GroupByDynamic(times, every, period=every, start_by=:window)

Iterator over (index_range, window_start) pairs for sliding windows on sorted times.

IntervalRange{T, D}

Lazy iterator of (t_start, t_end) pairs splitting [t0, t1) into dt-sized windows.

WindowedView(data, coords, windows; dim=1)
WindowedView(data, coords, before, after; dim=1)

AbstractVector of views into data, one per time window. wv[i] returns the slice of data along dimension dim whose coords fall within the i-th window.

windows is any indexable source of (t_start, t_stop) pairs (e.g. TimeRanges). The before/after form builds a symmetric window around each point in coords. Requires coords to be sorted.

cadence(x; tol=2, rtol=exp10(-tol-1), check=true)

Robust sampling cadence of times(x) via IQR-trimmed modal clustering on positive diffs, refined by the median of values near the modal center. With check=true, warns on gaps or non-integer-multiple intervals.

common_timerange(x1, xs...)

Get the common time range (intersection) across multiple arrays. If there is no overlap, returns nothing.

dimnum(x, dim)

Get the ordinal of the dimension dim in x.

dropna(A; dim=nothing)

Remove slices containing NaN values along dimension dim.

find_continuous_timeranges(x, max_dt)

Find continuous time ranges for x, where max_dt is the maximum time gap between consecutive times.

find_outliers(A, [method, window]; dim = nothing, kw...)

Find outliers in data A along the specified dim dimension.

Returns a Boolean array whose elements are true when an outlier is detected in the corresponding element of A.

The default method is :median (other option is :mean), which uses the median absolute deviation (MAD) to detect outliers. When the length of A is greater than 256, it uses a moving window of size 16.

See also: find_outliers_median, find_outliers_mean, isoutlier - MATLAB

find_outliers_mean(x::AbstractVector, window; threshold = 3)

Find outliers that are defined as elements more than three standard deviations from the mean.

This method is faster but less robust than find_outliers_median.

find_outliers_median(x, window; threshold=3)

Find outliers that are defined as elements more than threshold=3 times the scaled median absolute deviation (MAD) from the median.

When window is set to a integer, a moving window of that size is used to compute local MAD. Otherwise, global statistics are used.

References

• Median absolute deviation - Wikipedia

groupby_dynamic(x, every, period=every, start_by=:window)

Eagerly collect (group_idx, starts) vectors for sorted x.

interpolate_outliers!(x, t, outliers)

Interpolate outliers in x using interpolation of neighboring non-outlier values.

Vector rejection

proj(a, b)

Vector projection of a vector a on (or onto) a nonzero vector b.

References: Wikipedia

See also: sproj, oproj

pspectrum(A, times; dim=ndims(A), nfft=256, noverlap=div(nfft, 2), window=DSP.hamming)
pspectrum(A; dim=nothing, freqdim=:frequency, kwargs...)

Compute a time-frequency power spectrum with DSP.spectrogram.

For containers that implement the common axis API, times are inferred from the selected axis and the result is rebuilt with frequency and spectrogram time axes followed by the remaining axes.

replace_outliers!(A, s, [find_method, window]; kwargs...)

Finds outliers in A and replaces them with s (by default: NaN).

See also: find_outliers, filloutliers - MATLAB

replace_outliers!(A, method, [find_method, window]; kwargs...)
replace_outliers!(A, method, outliers; kwargs...)

Replaces outliers in A with values determined by the specified method.

Outliers can be detected using find_outliers with optional find_method and window parameters or specified directly as a Boolean array outliers.

method can be one of the following:

• :linear: Linear interpolation of neighboring, nonoutlier values
• :previous: Previous nonoutlier value
• :next: Next nonoutlier value
• :nearest: Nearest nonoutlier value

See also: filloutliers - MATLAB

replace_outliers(A; args...; kw...)

Non-mutable version of replace_outliers!.

smooth(data, times, window; dim=ndims(data))
smooth(data, window; dim=ndims(data))

Smooths a time series by computing a moving average over a sliding window. Edge windows are truncated, so the output has the same size as the input.

Each window covers the half-open interval [coord - before, coord + after). A scalar window is interpreted as a coordinate span along the smoothed axis.

Arguments

• dim=ndims(data): Dimension along which to perform smoothing
• op=nanmean: Function used to aggregate each window

Scalar projection

tclip(A, t0, t1; dim=nothing)

Clip A to time range [t0, t1] along dimension dim.

dim should be sorted before clipping (see tsort).

tclips(xs...; trange=common_timerange(xs...))

Clip multiple arrays to a common time range trange.

If trange is not provided, automatically finds the common time range across all input arrays.

tcross(x, y; dim=nothing)

Compute the cross product of two (arrays of) vectors along dimension dim.

tderiv(A, times; dim=1)
tderiv(A; dim=nothing)

Compute the time derivative of A. Set lazy=true for lazy evaluation.

See also: deriv_data - PySPEDAS

tdot(x, y; dim=nothing)

Dot product of two arrays x and y along dimension dim.

tfilter(A, times, Wn1, Wn2=nothing; dim=ndims(A), designmethod=nothing)
tfilter(A, Wn1, Wn2=nothing; dim=nothing, designmethod=nothing)

Bandpass filter A between Wn1 and Wn2. The upper cutoff defaults to the Nyquist frequency.

References

• https://docs.juliadsp.org/stable/filters/
• https://www.mathworks.com/help/signal/ref/filtfilt.html
• https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.filtfilt.html

tgroupby(x, every, period = every, start_by = :window)

Group x into windows based on every and period.

time_grid(x, dt)

Create a time grid from the minimum to maximum time in x with the step size dt.

Examples

# Create hourly time grid
time_grid(x, Hour(1))
time_grid(x, 1u"hr")

# Create 1-s intervals
time_grid(x, Second(1))
time_grid(x, 1u"second")
time_grid(x, 1u"Hz")

timerange(x)
timerange(x1, xs...)

Get the time range of time series data x.

For a single argument, returns a tuple (tmin, tmax) containing the minimum and maximum times. For multiple arguments, returns the common time range (intersection) across all arrays - equivalent to common_timerange(x1, xs...).

Examples

# Single time series
times = [1, 2, 3, 4, 5]
timerange(times)  # (1, 5)

# Multiple time series - find common range
x1_times = [1, 2, 3, 4]
x2_times = [2, 3, 4, 5]
timerange(x1_times, x2_times)  # (2, 4)

See also: common_timerange, tminimum, tmaximum

times(x)

Get time coordinate of x.

tinterp(A, old_times, new_times; interp=LinearInterpolation)

Interpolate time series A at new time points new_times.

The interp constructor must accept (u, t; kws...) and return a callable object. Its syntax is compatible with DataInterpolations.jl.

Examples

# Interpolate at a single time point
tinterp(time_series, DateTime("2023-01-01T12:00:00"))

# Interpolate at multiple time points using cubic spline interpolation
new_times = DateTime("2023-01-01"):Hour(1):DateTime("2023-01-02")
tinterp(time_series, new_times; interp = CubicSpline)

tinterp_nans(A; dim = nothing, kwargs...)

Interpolate only the NaN values in A along dimension dim.

tmask!(A, t0, t1; dim=nothing)
tmask!(A, its; dim=nothing)

Mask all data values within the specified time range(s) (t0, t1) / its with NaN.

tmask(A, args...; kwargs...)

Non-mutable version of tmask!. See also tmask!.

tmaximum(x)

Get the maximum timestamp of x.

tmean(x, [dt]; dim=nothing)

Calculate the arithmetic mean of x along dimension dim, optionally grouped by dt.

dim accepts integer index, dimension type/instance, or nothing (defaults to the time dimension).

tmedian(x, [dt]; dim=nothing)

Calculate the median of x along dimension dim, optionally grouped by dt.

dim accepts integer index, dimension type/instance, or nothing (defaults to the time dimension).

tminimum(x)

Get the minimum timestamp of x.

tnorm(A; dim=nothing)

Compute the norm of each slice in A along dimension dim.

See also: tnorm_combine

tnorm_combine(x; dim=nothing, name=:magnitude)

Calculate the norm of each slice along dim and combine it with the original components.

toproj(A, B; dim=nothing)

Compute vector rejection (orthogonal projection) of A from B along dimension dim.

tproj(A, B; dim=nothing)

Compute vector projection of A onto B along dimension dim.

tresample(A, old_times, freq; kw...)

Resample time series A onto a regular time grid with the specified frequency freq.

See also: tinterp, time_grid

tselect(A, t; dim=nothing)
tselect(A, t, δt; dim=nothing)

Select the value of A at time nearest to t.

With δt, restricts the search to [t-δt, t+δt] and returns missing if that window is empty.

tsem(x, [dt]; dim=nothing)

Calculate the standard error of the mean of x along dimension dim, optionally grouped by dt.

dim accepts integer index, dimension type/instance, or nothing (defaults to the time dimension).

tshift(x, t0 = nothing; dim=nothing)

Shift the dim of x by t0.

tsort(A; dim=nothing, rev=false)

Sort A along dimension dim.

tsplit(t0, t1, n::Int)
tsplit(t0, t1, dt)
tsplit(t0, t1, dtType::Type{<:Period})

Split the range from t0 to t1 into n parts, dt-sized parts, or by period type (e.g., Month, Day).

tsproj(A, B; dim=nothing)

Compute scalar projection of A onto B along dimension dim.

tstat(f, x, [dt]; dim = nothing)

Calculate the statistic f of x along the dim dimension, optionally grouped by dt.

See also: groupby_dynamic

tstd(x, [dt]; dim=nothing)

Calculate the standard deviation of x along dimension dim, optionally grouped by dt.

dim accepts integer index, dimension type/instance, or nothing (defaults to the time dimension).

tsubtract(x, op=nanmedian; dim=nothing)

Subtract a statistic op along time dimension (or dim) from x.

tsum(x, [dt]; dim=nothing)

Calculate the sum of x along dimension dim, optionally grouped by dt.

dim accepts integer index, dimension type/instance, or nothing (defaults to the time dimension).

tsync(A, Bs...)

Synchronize multiple time series to have the same time points.

This function aligns time series Bs... to match time points of A by:

1. Finding common time range between all time series
2. Extracting subset of A within common range
3. Interpolating each series in Bs... to match the time points of the subset of A

Examples

A_sync, B_sync, C_sync = tsync(A, B, C)

See also: tinterp, common_timerange

tvar(x, [dt]; dim=nothing)

Calculate the variance of x along dimension dim, optionally grouped by dt.

dim accepts integer index, dimension type/instance, or nothing (defaults to the time dimension).

tview(A, t0, t1; dim=nothing)
tview(f, t0, t1)

View A in time range [t0, t1] along dimension dim.

When f is callable, returns f(t0, t1).

unwrap(x)

Return the innermost object (array) of wrapped x with similar behavior (e.g. same size, same type, etc.)

window_bf_sizes(window)

Converts a window specification to backward and forward window sizes.

When window is a positive integer scalar, the window is centered about the current element and contains window-1 neighboring elements. If window is even, then the window is centered about the current and previous elements.