Introduction
Nexus is the open-source Enterprise Intelligence Framework — a Python library for building secure, governed AI applications on your own data. It is released under the Apache License 2.0, so every component described here is readable, auditable, and free to use commercially.
It turns documents, databases and live events into clean, grounded chunks and semantic embeddings; answers questions from them behind guardrails, with citations; keeps indexes current from PostgreSQL, Kafka and webhooks without losing changes; and turns data into governed, explainable actions — reminders under contact rules, work orders for degrading equipment — each sent exactly once.
Everything runs in your process, on your infrastructure: parsing, PII masking, embeddings (FastEmbed / ONNX) and decisions are local by default, and nothing leaves the process unless you configure a provider.
Release Notes
Nexus follows Semantic Versioning. Only the latest release line (3.1.x) receives fixes. The full history is in CHANGELOG.md.
Search scoring, guardrail defaults, relevance gate
- Added —
RagConfig.min_semantic_score(off by default):ask()refuses questions whose sources are not close enough.inspect_prompt(..., include_leakage=False). - Changed — keyword scoring is the IDF-weighted share of the query's meaningful words; stopword-only matches no longer score 1.0. Leakage terms apply to the question only; answers are still re-checked for blocked patterns. An off-topic gate with no keywords no longer blocks every question.
- Fixed — ServiceNow correlation display now says "Nexus"; README, reference and security-policy corrections.
- Upgrade — no code changes; re-tune thresholds stored on
search()scores.
Operations, condition monitoring, work orders, Kafka
- Added —
nexus.operations: expression language, decision tables, strategies, scorecards, contact policy, case planner with approvals and holdout,OperationsEngine, messaging providers, reply intents with a learned model, experiments,AssetMonitoranddiagnose, ServiceNow / Maximo / Teams work orders, outbound SSRF protection, TRAI DLT fields. - Added —
KafkaSource([kafka]): at-least-once batches, SASL_SSL and mutual TLS, rebalance-safe commits, lag, Debezium / JSON / Avro decoding. - Upgrade — queue actions on
PlannedAction.ready, notsendable: actions awaiting approval are not ready.
Semantic embeddings, re-ranking, CDC, PostgreSQL, Slack
- Added —
embed_texts,embed_query,embedding_info(FastEmbed, 384 dimensions),rerank; CDC normalization; PostgreSQLpgoutputstreaming; webhook signing; Slack; pluggable OCR, transcription and video; heading-aware chunking; DDL builders. - Removed — vectors from processing and the 3,072-dimension hashing projection:
NexusClient.embed(),ProcessedChunk.embedding, hashing embedders, thesemanticextra, legacy CDC helpers and DDL constants. - Upgrade — embed at storage time, migrate the vector column to 384 dimensions and re-embed stored chunks once.
Apache-2.0, multimodal ingestion
- Changed — open source under the Apache License 2.0 with reserved trademarks; one name, the Enterprise Intelligence Framework; documentation consolidated.
- Added — pure-Python Word, Excel, PowerPoint, audio, video, image, code, OpenAPI, SQLite, e-mail and chat processing; universal routing in
process_document; DCO sign-off; thenexusconsole script. - Fixed — retrieval indexes were not persisted by
build_indexes(); README examples; namespace leaks.
Execution trace
- Added — a five-stage execution trace on every processed document;
enable_guardrailsonprocess_document().
NexusClient
- Added — the in-process
NexusClient;nexus.*namespace packaging;nexus.database; PyPI Trusted Publishing. - Changed — locked in-memory stores,
in_memory_only, tenant-bound key derivation, no implicit environment lookups for secrets.
Seven layers
- Added — the seven independently installable layers, the
nexusCLI and offline test suites, with authenticated encryption, constant-time API keys, pluggable RBAC, SSRF defense, subprocess hardening and Unicode-normalized guardrails.
Upgrade guide
| From → to | Effort | What to do |
|---|---|---|
| 3.1.0 → 3.1.1 | None in code | Review thresholds on search() scores and any reliance on the old guardrail blocking |
| 3.0.1 → 3.1.0 | Small | Use PlannedAction.ready when queueing actions |
| 3.0.0 → 3.0.1 | Breaking | Embed with embed_texts() / embed_query(); migrate to vector(384); re-embed; replace removed APIs |
| 2.4.0 → 3.0.0 | None in code | License changes to Apache-2.0 |
| 0.1.0 → 2.3.0 | Moderate | Adopt NexusClient; pass secrets explicitly; check existing ciphertext |
$ pip install --upgrade "veloxs-nexus==3.1.1"Overview
One entry point, NexusClient, wires the processing, retrieval, guardrails and experience layers together in memory. Each capability is also a focused module — nexus.processing, nexus.retrieval, nexus.guardrails, nexus.operations, nexus.security, nexus.database — usable on its own.
In a service deployment, each of the seven layers runs as its own project with its own configuration, tests and CLI, integrating only through configuration, JSONL contracts, CLIs and HTTP — never by importing another layer's code. That is what makes any layer replaceable.
Offline by default
Parsing, masking, embeddings, search and decisions run locally. Models download once into a cache you control.
Safe with untrusted input
Rules and templates are data that cannot run code; prompts and retrieved text are screened; outbound calls are SSRF-checked.
Explainable
Five-stage processing traces, citations on every answer, reason codes and rule versions on every decision.
High-Level Architecture Overview
Five sequential pipeline layers and two cross-cutting layers — Security & Governance and Observability & Monitoring — that integrate with every stage. nexus.operations sits beside them and turns facts into decisions and actions.
Component-Level Design
The framework is composed of seven layered components plus the operations module. Each card lists what ships today in the veloxs-nexus package.
Enterprise Data Sources
The systems of record Nexus reads from, in place:
- Documents — PDF, Word, Excel, PowerPoint, e-mail, images, audio, video
- Databases — PostgreSQL (logical replication), MySQL rows, MongoDB collections, SQLite files
- Event streams — Apache Kafka topics, Debezium and Maxwell change events, MongoDB change streams
- Applications — REST APIs, signed webhooks, Slack workspaces, source code and OpenAPI specifications
Data Ingestion & Integration
Moves data from source systems into Nexus reliably, with at-least-once delivery: nothing is acknowledged until the caller has durably applied it.
Key capabilities
- REST connectors — pagination, bearer tokens from the environment, cross-origin
nextlinks refused, redirects refused. - Batch drops — JSON, JSONL and CSV files validated and deduplicated by primary key.
- PostgreSQL change data capture — the built-in
pgoutputprotocol, an exact initial copy through an exported snapshot, publication column lists and row filters, slot monitoring. - Apache Kafka — consumer groups,
read_committed, commits after apply, SASL_SSL and mutual TLS, rebalance-safe commits, Schema Registry Avro. - Webhooks & Slack — HMAC-SHA256 signatures with a replay window; Slack Events API verification.
Data Processing & Enrichment
Turns raw content into retrieval-ready chunks and records exactly what it did.
Key capabilities
- Format-aware parsing — 30+ formats in pure Python, with optional OCR (EasyOCR), transcription (faster-whisper) and video demuxing (PyAV).
- Chunking — heading sections with breadcrumbs for Markdown and Word, pages for PDF, rows with headers for tables, time windows for media, threads for chat.
- Metadata extraction — classification, tags, entities, dates, amounts and e-mails.
- PII masking at ingestion — before anything is stored; switchable per call.
- Execution trace — five stages with status, duration and summary on every payload.
Embedding & Retrieval Intelligence
Semantic representations and precise retrieval over them.
Key components
- Semantic embeddings —
BAAI/bge-small-en-v1.5(384 dimensions) through FastEmbed / ONNX, asymmetric query and passage encoding; OpenAI as an alternative provider. - Hybrid search — semantic similarity, IDF-weighted keyword coverage and a shared-entity graph, blended with configurable weights.
- Cross-encoder re-ranking —
Xenova/ms-marco-MiniLM-L-6-v2for ordering a short candidate list. - Storage schemas — PostgreSQL + pgvector (HNSW,
halfvecabove 2,000 dimensions, generatedtsvector), MySQL and MongoDB Atlas Vector Search.
AI Orchestration & Guardrails
The control point every question passes through before an answer is returned.
- Prompt screening — blocked patterns and leakage terms, checked on NFKC-normalized text with zero-width and bidi characters removed.
- Indirect-injection defense — the composed answer is re-checked for blocked patterns, because retrieved text is untrusted.
- PII detection & masking — API keys, JWTs, e-mail, SSNs, Luhn-checked card numbers, phone numbers.
- Grounded answers — retrieval, answer composition, grounding verification with a confidence score, and citations; no context means
blocked. - Relevance gate — optional
min_semantic_scorerefuses questions the documents cannot answer. - Policies & topics — blocked terms, required citations, required masking, and a keyword-based off-topic gate.
Experience & Engagement Layer
How applications and people use Nexus.
- Python SDK —
NexusClient, in process, thread-safe. - REST service —
/health,/v1/ask,/v1/sessionswith constant-time API-key auth, principal-bound identity and owned sessions. - CLI —
nexusplus one CLI per layer. - Pluggable authorization — an
Authorizerprotocol for your own policy engine.
Governed Operations & Condition Monitoring
Turns facts into accountable actions, with the controls regulated teams require.
- Rules as data — a safe expression language, versioned decision tables, strategies and points scorecards with reason codes.
- Case planning — a pure planner: next state, actions with permanent idempotency keys, contact-rule timing, channel fallback, approvals and a holdout arm.
- Contact rules — hours in the recipient's time zone, caps, consent, do-not-disturb;
IN_RBIandUS_REG_Fpresets. - Messaging & replies — placeholder-only templates, signed webhooks, SMTP, WhatsApp Cloud, Twilio; multilingual reply intents with a learned model.
- Condition monitoring & work orders — explainable anomaly scores, failure-mode diagnosis, ServiceNow, Maximo and Teams.
- Experiments — Wilson intervals, two-proportion tests and conclusiveness guards against a holdout.
Security, Governance & Observability
Consulted by every other layer, so that all data and AI interactions are protected, policy-compliant and auditable.
- Role-Based Access Control — permissions, tenant match (or
cross_tenant:read), tenant and role data scopes. - Tenant-bound encryption — Fernet with keys derived by HKDF-SHA256 per tenant and key ID; fails closed without key material.
- Outbound SSRF protection — every provider call resolves and checks each address, pins the connection and refuses redirects.
- Audit logging — access decisions and events as append-only JSONL.
- Observability — metrics, structured logs, traces, AI interaction events and alert evaluation, with declarative exporter configuration.
Compliance & regulatory alignment
Designed to support controls under industry and regulatory standards, including:
API Reference
Every name below is importable from the installed veloxs-nexus 3.1.1 package.
NexusClient
NexusClient(
config=None, base_dir=None, tenant_id="default", in_memory_only=True,
processing_engine=None, retrieval_engine=None,
guardrails_engine=None, experience_service=None,
)| Method | Description |
|---|---|
| process_document | (document_id, name, text, file_type=None) — text or bytes, routed by extension and file header |
| process_pdf | (pdf_id, name, pdf_bytes=None, file_path=None) — one chunk per page |
| process_word | (document_id, name, docx_bytes=None, file_path=None) — headings, lists, tables |
| process_spreadsheet | (spreadsheet_id, name, spreadsheet_bytes=None, file_path=None) — rows with sheet and column names |
| process_presentation | (presentation_id, name, presentation_bytes=None, file_path=None) — slides and speaker notes |
| process_email | (email_id, name, email_bytes=None, file_path=None) — headers, body, attachments |
| process_image | Metadata; OCR with [ocr], or pass ocr_text / caption |
| process_audio | Time windows (window_seconds=10.0); transcription with [audio-ml] |
| process_video | Scene windows (scene_interval_seconds=10.0); demuxing, transcript and OCR with extras |
| process_sqlite | Schema and rows per table (max_rows_per_table=500) |
| process_mysql_table | (table_name, rows, primary_key=None, rows_per_chunk=1) |
| process_mongo_collection | (collection_name, documents, flatten_nested=True) |
| process_chat | Conversations as turn windows (turns_per_window=8) |
| process_code | Functions and classes with signatures |
| process_openapi | OpenAPI 3.0 / 3.1 and Swagger 2.0 endpoints and schemas |
| process_slack_export | Workspace .zip or channel .json: threads and day windows |
| embed_texts | Passage embeddings, batched, L2-normalized |
| embed_query | Query embedding (asymmetric encoding) |
| embedding_info | Provider, model and dimensions of the active embedder |
| rerank | (query, passages) — cross-encoder scores in (0, 1) |
| index_document | (payload, collection="general") — add chunks to the in-memory indexes |
| search | (query, limit=10) — hybrid search, best first |
| ask | (query, channel, user_id, tenant_id, session_id) — guarded, grounded AskResponse |
All processing methods accept metadata, enable_guardrails=True and shallow_mode=False.
Payloads, Results & Responses
| Model | Fields |
|---|---|
| ProcessedDocumentPayload | document_id, name, file_type, file_size_bytes, content_hash, classification, chunks, metadata, execution_trace, summary |
| ProcessedChunk | chunk_id (<document_id>:<index>), document_id, chunk_index, text, metadata — no vector |
| ProcessingStageTrace | step_number, stage_name, status, duration_ms, summary, details |
| SearchResult | id, collection, text, score, semantic_score, lexical_score, graph_score, metadata |
| AskResponse | request_id, decision (allowed / blocked), answer, citations (source_id, collection, score), channel, tenant_id, session_id, metadata, created_at |
| GuardrailResponse | decision, query, masked_query, answer, confidence, citations, findings (category, message, severity) |
Live Sources & Media Providers
| Name | Description |
|---|---|
| normalize_change_events | cdc — Debezium, Maxwell, MongoDB change streams, generic webhooks → ChangeEvent; single, list, {"events": [...]} or NDJSON; at most 1,000 |
| ChangeEvent | operation, source_format, database, table, key, record, removed_fields, partial, timestamp_ms, is_delete |
| change_event_text | cdc — one self-describing chunk for the record's current state |
| verify_webhook_signature | cdc — HMAC-SHA256 with timestamp binding and a 300-second tolerance; sign_webhook_payload for senders |
| PostgresLogicalStream | pgoutput — (dsn, slot_name, publication): prerequisites, ensure_slot, create_slot_with_snapshot, snapshot_session, published_tables, iter_table, transactions, confirm, slot_info, identify_system, drop_slot |
| KafkaSource | kafka — (KafkaSettings): batches, commit, check, lag, before_revoke, assignment_version, close |
| decode | kafka — (record, fmt) to change events or JSON documents; avro_decoder(url, …) for Schema Registry |
| slack_export | parse_slack_export, messages_to_chunks, verify_slack_signature, slack_event_message |
| release_idle_models | ml_providers — unload OCR and speech models idle for longer than max_idle_seconds=600; providers EasyOCRProvider, FasterWhisperTranscriber, PyAVDemuxer |
Retrieval Engine, Embedders & Re-ranker
| Name | Description |
|---|---|
| RetrievalEngine | (in_memory_only=True, …) — add_entry, search, embed, embed_batch; with in_memory_only=False indexes persist under base_dir |
| create_text_embedder | (provider=None, model_name=None) — process-wide cached SemanticEmbedder (FastEmbed) or OpenAIEmbedder |
| create_reranker | (model_name=None) — cross-encoder; score(query, passages) |
GuardrailsEngine & Configuration
GuardrailsEngine(config=None, base_dir=None, retrieval_engine=None, search_provider=None) — evaluate(query), inspect_prompt(text), detect_pii(text), mask_pii(text).
| Section | Fields and model defaults |
|---|---|
| prompt_security | blocked_patterns=[], leakage_terms=[] — leakage terms apply to the question |
| pii | enabled, mask, detectors=[api_key, jwt, email, ssn, credit_card, phone] |
| policies | id, description, blocked_terms, require_citations, require_pii_masking, action (warn / block) |
| off_topic | enabled=True, min_keyword_overlap=1, allowed_keywords=[] — no keywords, no restriction |
| rag | top_k=3, min_context_score=0.05, require_citations=True, min_semantic_score=None |
| verification | min_confidence=0.2, require_grounded_terms=True |
The engine's own defaults (no config): blocked patterns ignore previous instructions, reveal system prompt, exfiltrate; leakage terms api key, password, secret, token; off-topic gate off.
Operations Building Blocks
| Area | Names |
|---|---|
| Expressions | evaluate(source, facts, today=None), compile_expression, referenced_fields, ExpressionError |
| Decisions | DecisionTable (first / priority / collect / unique), Rule, DecisionResult, DecisionError, evaluate_strategy, StrategyResult |
| Scorecards | Scorecard(name, characteristics, base_points=600, base_odds=50, pdo=20, bands, max_reasons=4, outputs), ScoreResult |
| Contact rules | ContactPolicy (preset, from_dict, check), ContactDecision, FrequencyCap |
| Cases | plan_case, CasePlan, PlannedAction (ready, needs_approval), OperationsEngine (upsert, evaluate_due, dispatch, approve), CaseMachine, DEFAULT_MACHINE, idempotency_key, assign_arm |
| Messaging | Message, SendResult, DryRunProvider, WebhookProvider, SmtpEmailProvider, WhatsAppCloudProvider, TwilioSmsProvider, provider_from_config, render_template, mask_recipient, dlt_problems |
| Replies | IntentClassifier, IntentResult, IntentModel, evaluate_model, redact, INTENTS |
| Experiments | rate, compare, summarize_arms |
| Monitoring | SignalSpec, specs_from_dict, AssetMonitor, Assessment, diagnose, window_stats |
| Work orders | WorkItem, WorkResult, TicketStatus, ServiceNowProvider, MaximoProvider, TeamsProvider, DryRunWorkProvider, work_provider_from_config |
| Outbound | outbound.check_url, check_address, resolve_checked, urlopen, pinned_smtp, OutboundBlocked |
Expression language
Literals, names (unknown fields are None), nested fields (a.b, a['b']), and / or / not, comparisons including chained and in, arithmetic, conditionals, and the functions min, max, abs, round, len, lower, upper, coalesce, date, today, days_until, days_since, add_days, str. Comparisons with None are False; attribute access on objects, imports, lambdas, comprehensions and dunder names are rejected; at most 2,000 characters.
Security & Storage Schemas
| Name | Description |
|---|---|
| EncryptionConfig | enabled, key_id, tenant_id, secret_key, key_material_env="NEXUS_SECURITY_KEY", require_tls, allowed_tls_versions |
| encrypt_text | Fernet with HKDF-SHA256 per tenant and key ID; fails closed without key material |
| decrypt_text | EncryptionError on tampering, a wrong key or a wrong tenant |
| authorize | (SecurityConfig, AccessRequest) — role, permission, tenant match, tenant scope, role scope → AccessDecision |
| pgvector_ddl | (dim) — PostgreSQL + pgvector schema with HNSW and full-text indexes |
| mysql_ddl | (dim) — MySQL schema, vectors as JSON arrays |
| mongo_atlas_vector_index | (dim) — Atlas Vector Search definition (cosine) |
| get_pgvector_column_type | (dim) — VECTOR(n) up to 2,000 dimensions, HALFVEC(n) above |
Formats, Extras & Environment
| Category | Formats |
|---|---|
| Text | .txt, .md (heading sections); .json, .html, .xml, .yaml as text; .csv (row packs with the header) |
| Office & mail | .pdf, .docx, .xlsx, .pptx, .eml |
| Media | .png, .jpg, .jpeg, .bmp; .wav, .mp3, .aiff; .mp4, .mov |
| Code & APIs | Python (AST), TypeScript, JavaScript, Go, Rust, Java, C++, C#; OpenAPI 3.0 / 3.1, Swagger 2.0 |
| Data | SQLite files, MySQL rows, MongoDB documents, chat exports, Slack exports, change events |
Optional extras: postgres, kafka, kafka-avro, ocr, audio-ml, video, all-ml, yaml, dev. Environment variables are listed in the Configuration Reference.
Design Notes
Security model
Nexus handles untrusted input: documents, prompts, retrieved text, rules written by business users and provider URLs typed into a settings page. PII is masked at ingestion and again in answers; prompts are screened before retrieval and answers re-screened after it; rules are parsed against an allow-list and templates only substitute placeholders; outbound calls are resolved, checked and pinned; secrets are passed explicitly or named with env:VAR_NAME, never read silently. There is no eval, exec, pickle, os.system or shell=True, and no side effect at import time.
Platform configuration is treated as trusted. Rate limits, request sizes and TLS belong at your ingress; key management at your KMS; audit logs in tamper-evident storage.
At-least-once delivery
A consumer that acknowledges before its write commits can lose data silently on a crash. Every Nexus source acknowledges after: stream.confirm(txn.end_lsn) for PostgreSQL, source.commit(batch) for Kafka. A crash replays; idempotent writes — upserts by key, merges for partial events — make replays harmless. Actions go further: permanent idempotency keys (subject|cycle|stage) in a unique outbox column, passed to providers, give exactly-once effects.
Explainable decisions
Rules are data with content versions; every decision carries reason codes naming the step and rule; plan_case is a pure function of strategy, facts, contact context and an explicit clock, so any past decision can be recomputed. Transparent models come first; learned models are consulted only when rules are unsure, with capped confidence, and an LLM only ever sees redacted text. Sensitive actions wait for a person's approval, and a holdout that is evaluated but never contacted measures what the actions add.
Deployment
Use Nexus as a library in your own service, run the experience layer as a REST microservice behind your ingress, or deploy each layer as its own container — cross-layer integration is configuration, JSONL and HTTP, with no shared runtime.
In containers, mount a volume for NEXUS_MODEL_CACHE_DIR and warm it during the image build; pass secrets through your secret store; terminate TLS at your ingress; ship the structured logs and audit events to your log pipeline.
For installation, configuration, and integration patterns, see the Nexus User & Integrator Guide.