LangDB Logo

The Command Center
for AI Workflows

https://langdb.ai

Agenda

  • 1
    The Challenge of Uncontrolled LLM Deployments
  • 2
    The Need for Centralized AI Management
  • 3
    AI Gateway Architecture
  • 4
    Why LangDB?
  • 5
    Own Your Data, Own Your AI
  • 6
    Deploying LLMs in the Enterprise
  • 7
    How to Use It & LangDB Editions

About Us

Matteo Pelati

Matteo Pelati

Co-founder

Veteran software architect and data engineering leader with 20+ years of experience. Expert in building large-scale data analytics and ML platforms.

  • Former Head of APAC Data Engineering @ Goldman Sachs
  • Former Head of Data Platform @ DBS Bank
  • Former Chief Architect @ DataRobot
Vivek Gudapuri

Vivek Gudapuri

Co-founder

Technology leader with proven track record in scaling startups. Expert in building innovative platforms in data, logistics, and fintech.

  • Chief Technology Officer @ OpenFabric
  • Chief Technology Officer @ Yojee
  • Member of Founding Team @ PaySense

Backed by

Sequoia Capital Gradient Ventures January Capital

Uncontrolled LLM Deployments

LLM Hallucinations

LLM Hallucinations

  • Air Canada faced legal action after its AI-powered chatbot provided a passenger with incorrect information regarding refund policies
  • Dow Jones and the New York Post filed a lawsuit against AI startup Perplexity, alleging that its platform generated fake news content by fabricating sections of articles and misattributing them to reputable publishers
Prompt Injection

LLM Jailbreaks

  • A Stanford student exploited a vulnerability in Microsoft's Bing Chat, causing it to reveal internal guidelines and its codename, "Sydney"

The Need for Centralized AI Management

AI Management
  • Monitor what AI applications and AI agents are doing across the organization
  • Enforce policies, ensuring no data is leaked and wrong information is not communicated
  • Manage costs at a granular level with detailed usage tracking and controls
  • Measure ROI to ensure AI investments are generating the desired business outcomes

AI Gateway Architecture

AI Gateway Architecture

An AI Gateway is a service that simplifies, secures, and governs access to LLMs.

Acts as a centralized platform for managing AI workflows, giving developers a consistent interface (or endpoint) to interact with models from different providers

Key Features

  • 1
    Logging and Tracing
    Centralized request tracking and audit trails
  • 2
    Routing and Model Selection
    Intelligent request routing across providers
  • 3
    Monitoring & Cost Management
    Real-time spend tracking and budget controls
  • 4
    Evaluations & Experimentation
    A/B testing across different model versions
  • 5
    Guardrailing & Policy Management
    Content filtering and compliance enforcement

Why LangDB?

LangDB Logo
bolt

Fastest Enterprise AI Gateway

Built in Rust for unmatched performance and scalability

shield

Guardrailing & Policy Management

Define guardrails and policies to make sure every request is compliant

api

Access to 250+ Models

Unified API for seamless integration with hundreds of models

money

Cost Control & Billing

Comprehensive cost management and and user usage tracking

hub

Multi-Agent Tracing

Advanced debugging and tracing for complex agent interactions

analytics

Advanced Analytics

Deep insights into your AI operations with Clickhouse

Own your Data, Own your AI

ClickHouse

ClickHouse as a Storage Layer

We partnered with ClickHouse to provide a fast & reliable storage layer for LLM observability
  • 1
    Observability Powerhouse
    ClickHouse is widely used as a storage for observability tools. An AI gateway needs to capture and store a lot of analytical and conversational data, making ClickHouse ideal.
  • 2
    Customer Control
    Data remains fully accessible for database customers. They can even choose to host their own ClickHouse instance and run LangDB on top of it.
  • 2
    Advanced Analytics
    LangDB provides their users with the ability to access LLM models directly from ClickHouse's native SQL to perform adanved analytics on the collected data

Deploying LLMs in the Enterprise

settings

Flexibility

  • Access to multiple models, tools, and data sources
  • Seamless integration with existing infrastructure
  • Ability to adapt to new models and changing business needs
visibility

Observability

  • Understand what happens behind the scenes
  • Ensure full traceability of AI interactions
  • Debug and address issues in real-time
security

Security and Cost Control

  • Manage access to different models
  • Limit usage and set cost caps to control expenses
  • Protect sensitive data through strict access controls
shield

Safety

  • Put guardrails in place to prevent harmful outputs
  • Ensure compliance with data privacy regulations
  • Monitor outputs to avoid inappropriate AI behavior
copyright

Ownership

  • Retain ownership of LLM interaction data
  • Enable full access to logs for auditing and compliance
  • Secure storage while ensuring accessibility
science

Experimentation and Improvement

  • Run controlled experiments to optimize performance
  • Collect data from experiments to fine-tune models
  • Continuously improve based on real-world feedback

How to use it?

  • 1
    Get a LangDB API key from the LangDB dashboard
  • 2
    Replace the OpenAI URL, the API key, and the model
from openai import OpenAI

client = OpenAI(
    base_url="https://api.us-east-1.langdb.ai/9c7ac2c8-b76f-453b-914d-39eaaccec092/v1",  # LangDB API base URL
    api_key=""
)

response = client.chat.completions.create(
    model="openai/gpt-4o-mini",
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant."
        },
        {
            "role": "user",
            "content": "What are the earnings of Apple in 2022?"
        }
    ]
)

print("Assistant:", response.choices[0].message)

LangDB Dashboard

LangDB Dashboard Screenshot

LangDB Editions

cloud

LangDB Hosted

Available at https://langdb.ai/

  • 1
    Access to 250+ Models
    Use any model from our extensive collection
  • 2
    Free Signup + Credits
    Start for free, use credits or bring your own key
  • 3
    Intelligent Routing
    Smart request distribution and load balancing
  • 4
    Monitoring & Cost Control
    Track usage and manage expenses in real-time
code

LangDB Open Source

Available at github.com/langdb/ai-gateway

  • 1
    Built in Rust
    High performance and reliability
  • 2
    Self-Hosted
    Deploy in your own environment
  • 3
    Apache-2.0 License
    Free to use and modify
  • 4
    Active Community
    Join us on Slack, contributions welcome