kaman.ai

Docs

Documentation

Guides, use cases & API reference

  • Overview
    • Getting Started
    • Platform Overview
  • Features
    • Features Overview
    • AI Assistant
    • Workflow Automation
    • Intelligent Memory
    • Data Management
    • Universal Integrations
    • Communication Channels
    • Security & Control
  • Use Cases Overview
  • Financial Services
  • Fraud Detection
  • Supply Chain
  • Technical Support
  • Software Development
  • Smart ETL
  • Data Governance
  • ESG Reporting
  • TAC Management
  • Reference
    • API Reference
  • Guides
    • Getting Started
    • Authentication
  • Endpoints
    • Workflows API
    • Tools API
    • KDL (Data Lake) API
    • OpenAI-Compatible API
    • A2A Protocol
    • Skills API
    • Knowledge Base (RAG) API
    • Communication Channels

Data Management

Centralized Data Control with Complete Traceability

Kaman's Data Management capabilities provide a unified, version-controlled data lake (KDL - Kaman Data Lake) powered by DuckDB with full lineage tracking via OpenLineage. Know exactly where your data comes from, how it's transformed, and where it flows throughout your organization.

The Challenge of Enterprise Data

Organizations struggle with:

Kaman solves these challenges with:

  • Centralized data lake with unified access
  • Complete data lineage from source to destination
  • Version control with time-travel queries
  • Impact analysis before changes
  • Full audit trails for compliance

Core Capabilities

KDL - Kaman Data Lake

Built on DuckDB and DuckLake, KDL provides high-performance analytics with enterprise features:

Benefits:

  • Single source of truth for your organization
  • High-performance analytics with DuckDB
  • Efficient columnar storage with Parquet
  • S3/MinIO compatible object storage
  • Simplified access management

Time-Travel Queries

Query your data as it existed at any point in time:

CapabilityDescription
Point-in-Time AccessQuery data as it existed at any historical moment
Change HistorySee exactly what changed between versions
RollbackRestore previous data states when needed
AuditProve what data looked like at any compliance checkpoint
ComparisonCompare data across different time periods

Full Data Lineage with OpenLineage

Track every piece of data from origin to consumption using the OpenLineage standard:

Lineage tracking shows:

  • Where data originated
  • What transformations were applied
  • When changes occurred
  • Who made the changes
  • Where the data is used

Impact Analysis

Before making changes, understand the consequences:

Answer questions like:

  • What will break if I change this table?
  • Who uses this data?
  • What reports depend on this field?
  • Which processes will be affected?

Schema Management

Control how your data structures evolve:

  • Schema Versioning - Track changes to data structures
  • Compatibility Checks - Ensure changes don't break consumers
  • Migration Support - Automated data migration when schemas change
  • Documentation - Automatic schema documentation
  • Schema Detection - Auto-discover schemas from MCP sources

Data Governance Features

Data Catalog

Discover and understand your organization's data assets:

  • Search - Find data by name, description, or content
  • Browse - Navigate data hierarchies and relationships
  • Documentation - Rich descriptions and usage examples
  • Metadata - Technical details, statistics, and quality metrics

Data Quality

Ensure your data meets organizational standards:

Quality checks include:

  • Completeness validation
  • Format and type verification
  • Referential integrity
  • Business rule compliance
  • Anomaly detection

Access Control

Secure your data with granular permissions:

LevelDescription
Data LakeControl who can access entire lakes
SchemaPermissions on schema namespaces
TableAccess control on specific tables
ColumnHide sensitive columns from certain users
RowShow only relevant records to each user

Audit & Compliance

Meet regulatory requirements with comprehensive tracking:

  • Access Logs - Who accessed what data and when
  • Change Logs - All modifications with before/after values
  • Query History - Complete record of data queries
  • Export Tracking - Record of data extracts and destinations
  • OpenLineage Events - Standard-format lineage events

Integration with MCP Connectors

KDL integrates seamlessly with all 23+ MCP data connectors:

Sync Modes:

ModeDescription
Full RefreshComplete data replacement
IncrementalOnly changed records
CDCReal-time change data capture
AppendAdd new records only

Business Benefits

Digital Resilience

Your data operations remain stable:

  • Recover from errors with version rollback
  • Time-travel to any historical state
  • Continue operations during system issues
  • Maintain data consistency across failures

Regulatory Compliance

Meet compliance requirements easily:

  • OpenLineage for standardized audit trails
  • Access controls for data protection
  • Retention management for data lifecycle
  • Complete documentation for auditors

Faster Decision Making

Get answers quickly:

  • DuckDB provides sub-second query performance
  • Unified data access eliminates hunting
  • Self-service analytics reduces bottlenecks
  • Trusted data improves decision confidence

Reduced Risk

Minimize data-related problems:

  • Impact analysis prevents breaking changes
  • Quality checks catch issues early
  • Lineage enables rapid troubleshooting
  • Time-travel provides safety net

How Data Flows Through KDL

Ingestion

Data enters the lake through multiple methods:

  1. MCP Sync - Automated sync from 23+ connectors
  2. Batch Import - Scheduled data loads from source systems
  3. Real-Time Streaming - Continuous data capture from events
  4. File Upload - Manual file imports with validation
  5. API Integration - Direct connections to external systems

Storage

Data is stored with:

  • Parquet Format - Optimized columnar storage
  • Compression - Reduced storage costs
  • Partitioning - Fast access to relevant data subsets
  • Encryption - Data protected at rest and in transit
  • S3/MinIO - Scalable object storage backend

Access

Consumers access data through:

  • SQL Interface - Standard SQL queries via DuckDB
  • API Access - REST API for programmatic retrieval
  • AI Assistant - Natural language queries
  • Export - Controlled data extraction
  • Visualization - Built-in charting and dashboards

Getting Started with Data Management

Step 1: Inventory Your Data

Identify existing data sources and their importance to your organization.

Step 2: Connect Sources

Set up MCP connectors to your priority data sources.

Step 3: Define Governance

Establish data ownership, quality standards, and access policies.

Step 4: Configure Sync

Set up synchronization schedules and transformation rules.

Step 5: Enable Users

Grant appropriate access and train users on self-service capabilities.

Data Management - Control, trace, and trust your data with KDL

On this page

  • Centralized Data Control with Complete Traceability
  • The Challenge of Enterprise Data
  • Core Capabilities
  • KDL - Kaman Data Lake
  • Time-Travel Queries
  • Full Data Lineage with OpenLineage
  • Impact Analysis
  • Schema Management
  • Data Governance Features
  • Data Catalog
  • Data Quality
  • Access Control
  • Audit & Compliance
  • Integration with MCP Connectors
  • Business Benefits
  • Digital Resilience
  • Regulatory Compliance
  • Faster Decision Making
  • Reduced Risk
  • How Data Flows Through KDL
  • Ingestion
  • Storage
  • Access
  • Getting Started with Data Management
  • Step 1: Inventory Your Data
  • Step 2: Connect Sources
  • Step 3: Define Governance
  • Step 4: Configure Sync
  • Step 5: Enable Users