Simon Knight

Data Analytics & Visualization Ecosystem

Tools for processing massive datasets, discovering patterns in time series, and creating information-dense visualizations that transform raw data into actionable insights.

Contents

The Vision

Modern data analysis fragments across disconnected tools: extract data with one tool, analyze with another, visualize with a third. This ecosystem provides an integrated workflow built on Rust and Polars with a focus on three key problems:

  1. Scale: Process millions of rows interactively, not in overnight batch jobs
  2. Signal: Find patterns and anomalies automatically, not through manual exploration
  3. Clarity: Generate visualizations that reveal structure, not just plot points

Core Philosophy:

How They Work Together

flowchart TD DS["Raw Data Sources
Geospatial · Time Series · Large CSVs · Streams"] DS --> TS["Tileserver Polars
Geospatial analytics"] DS --> MP["matrix-profile-rs
Time series patterns"] TS --> FE["Interactive Frontends
Kepler.gl · Dashboards"] MP --> FE

Typical Workflow:

  1. Ingest: Load massive datasets (geospatial points, time series) into Polars DataFrames
  2. Analyze: Use matrix-profile-rs for pattern discovery or Tileserver for spatial queries
  3. Visualize: Render interactive visualizations with sub-second query latency
  4. Iterate: Refine analysis based on visual feedback without waiting for batch jobs

The Tools

Tileserver Polars — Geospatial Analytics at Scale

Phase 7 — Complete · Full Details →

What It Is: Tile server that renders vector tiles (MVT) from Polars DataFrames for interactive geospatial visualization.

Key Features:

Example Workflow:

Load data:

import polars as pl
from tileserver_polars import TileServer

df = pl.read_csv("earthquakes_10M.csv")
server = TileServer(df, lon_col="longitude", lat_col="latitude")
server.start(port=8080)
Configure Kepler.gl + dynamic filtering ```javascript // Add custom tile layer { type: "mvt", url: "http://localhost:8080/tiles/{z}/{x}/{y}.mvt", renderSubLayers: true } ``` ```python # Filter by magnitude on the fly server.set_filter(pl.col("magnitude") > 5.0) # Tiles regenerate automatically with filtered data ```

Performance:

Use Cases:

Current Status: Production-ready for point geometries, adding polygon/line support.

Tech Stack: Rust, Polars, protobuf for MVT encoding, Actix-web for HTTP

matrix-profile-rs — Time Series Pattern Discovery

Phase 2/5 (16%) · Full Details →

What It Is: A Rust implementation of Matrix Profile algorithms for time series analysis. Automatically discovers repeating patterns (motifs) and anomalies (discords) in univariate time series without domain knowledge or parameter tuning.

The Problem: Time series analysis traditionally requires:

The Solution: Matrix Profiles provide a universal representation:

Key Features:

Polars Integration (Planned):

Matrix profile as DataFrame operations ```python import polars as pl df = pl.read_csv("sensor_data.csv") # Compute matrix profile as DataFrame operation result = df.with_columns([ pl.col("vibration").mp.stomp(window=100).alias("mp_distance"), pl.col("vibration").mp.motifs(k=3).alias("top_motifs"), pl.col("vibration").mp.discords(k=3).alias("anomalies") ]) ```

See the full project page → for Rust API examples and performance benchmarks.

Use Cases:

Current Status: Phase 2 (Discovery Ergonomics) - building high-level APIs for motif/discord extraction.

Tech Stack: Rust, ndarray, rayon for parallelization, PyO3 for Python bindings

Philosophy: Why This Approach?

Performance Enables Interactivity

Sub-second query latency transforms the analysis workflow. Instead of “run batch job, wait, inspect results, adjust, repeat,” you get “adjust filter, see results immediately.” This tight feedback loop enables exploratory analysis that’s impossible with slow tools.

Rust + Polars for the Data Layer

Polars provides:

Algorithms, Not Heuristics

Matrix Profiles are mathematically sound—they guarantee finding the true nearest neighbor for every subsequence. This eliminates “tune epsilon until it looks right” parameter hell common in clustering/anomaly detection.

Composable Tools

Each tool solves one problem well:

Use the full stack or just the pieces you need. All built on Arrow for interoperability.

Open Source & Contributions

Active development, contributions welcome:

← Back to Projects Network Automation Signal Processing Photography Autonomous Systems