Matyan

Experiment tracking and ML observability — Aim-compatible UI and SDK with a scalable backend.

Matyan (մատյան, book of records in Armenian) is a fork of Aim (aimhub) that reimplements the backend while keeping the frontend (UI) and Python SDK API unchanged. You get the same workflows and dashboards, with a modern stack: FoundationDB, Kafka, and stateless services.

✨ Features

Aim-compatible API — Same Run, Repo, and track() surface as the Aim SDK; minimal code changes to switch.
Lightweight client — matyan-client has minimal core dependencies (no ML frameworks by default); image, audio, figure, and framework adapters (PyTorch, Keras, etc.) are optional extras so you only install what you need.
Scalable storage — FoundationDB instead of RocksDB/SQLite; horizontally scalable, transactional.
Decoupled UI — React-based Aim UI served separately; backend is a stateless REST API.
Async ingestion — Training clients send data to a frontier (WebSocket + presigned S3); Kafka and workers write to FDB. No direct DB access from clients.
Index-accelerated search — Tier 1 (experiment, tag, archived, etc.) and Tier 2 (hyperparameter) indexes; MatyanQL runs as index scan + in-memory filter.
Full artifact support — Metrics, hyperparameters, images, audio, figures, distributions, text, terminal logs, and structured log records.
Python-only backend — FastAPI, aiokafka, FoundationDB Python bindings, boto3; no Cython, no embedded C.

📦 Installation

Run the full stack (Docker Compose)

Infrastructure (FoundationDB, Kafka, MinIO) plus backend, frontier, and workers:

git clone https://github.com/4gt-104/matyan-core.git
cd matyan-core
./dev/compose-cluster.sh up -d

Then open the UI (default: backend on port 53800, UI may be served separately — see Getting started).

Python client (track from your training code)

Install the Matyan client in your project:

python3 -m pip install matyan-client
# or with uv:
uv add matyan-client

Optional: set MATYAN_FRONTIER_URL and MATYAN_BACKEND_URL (e.g. http://localhost:53801, http://localhost:53800) if not using defaults.

🚀 Quick Start

With the Python client

Same API as Aim: create a run, set hyperparameters, track metrics and custom objects, then close. Full example: Getting started.

from matyan_client import Run

run = Run(experiment="baseline")
run["hparams"] = {"lr": 0.01, "batch_size": 32}
run.track(0.5, name="loss", step=0, context={"subset": "train"})
run.close()

Query runs and metrics via Repo (same as Aim):

from matyan_client import Repo

repo = Repo("http://localhost:53800")  # backend URL
for run in repo.iter_runs():
    print(run.hash, run["hparams"])

With Docker Compose (full stack)

Start infrastructure and services (see Installation).
Ensure the frontier is reachable at MATYAN_FRONTIER_URL (e.g. http://localhost:53801) and the backend at MATYAN_BACKEND_URL (e.g. http://localhost:53800).
Use the client as above; tracking goes through the frontier (WebSocket), metadata and queries go to the backend (REST).

See Getting started for detailed setup and Architecture for data flow and components.

🏗️ Architecture

Component	Role
matyan-backend	REST API: reads and control operations (delete run, rename experiment). Reads from FoundationDB; control writes emit Kafka events for async side effects (e.g. S3 cleanup).
matyan-frontier	Ingestion gateway: WebSocket for metrics/params, presigned S3 URLs for large blobs. Publishes to Kafka only.
Workers	Kafka consumers: ingestion (data → FDB), control (control-events → S3 cleanup, etc.).
matyan-client	Python SDK; same API as Aim. Uses frontier for tracking, backend for metadata and queries.
matyan-ui	Aim React UI; talks to matyan-backend.

Data flow: Reads: UI → backend → FoundationDB. Writes: client → frontier → Kafka → workers → FoundationDB (and S3 for blobs).

📚 Documentation

Getting started — Install client, run the stack (Docker Compose), first run, open UI, env vars, smoke tests; integrations, supported types.
Using Matyan — Manage runs, configure, query, search (MatyanQL), remote tracking, logging, artifacts.
Matyan UI — Explorers and run management (same as Aim UI).
Understanding Matyan — Concepts, data storage.
Architecture — Components and data flow in detail.
Advanced — Component-level design: Backend, Frontier, Workers, FoundationDB, Kafka, S3 and blobs, UI, Client SDK.
Production deployment — Deploy to Kubernetes with the Helm chart (values, secrets, scaling).
API — REST and streaming endpoints.
References — Glossary (key terms: Run, Repo, sequence, context, MatyanQL), CLI (backend, frontier, workers), SDK (matyan_client), Environment variables (all components).

🤝 Contributing

Contributions are welcome. For large changes, open an issue first to align on design. See Contributing and the repository for development setup and tests.

📄 License

See the LICENSE file in the repository.