Local and private AI

Use local AI when the requirement justifies the responsibility.

Local models and self-hosted tools can keep more data and inference under company control. They can also create a new server, network service, model supply chain, backup plan, and support obligation. Vertex Authority compares cloud, managed API, local, private-cloud, and hybrid architecture without treating any one option as automatically safer.

Evaluate private AI options See engagement options

Useful outcomes

What the work should improve.

  • A documented reason for choosing cloud, managed API, local, self-hosted, or hybrid
  • A realistic hardware and hosting plan
  • Private document search with citations and access controls
  • A patching, backup, logging, and recovery plan
  • Measured model quality, latency, throughput, and operating cost
  • A support boundary employees can understand

How it works

A practical scope with clear boundaries.

Architecture assessment

Evaluate the data classification, model-quality requirement, user count, expected volume, latency, offline needs, identity system, and support budget.

  • Cloud business plan or managed API
  • Hosted private cloud or dedicated inference
  • Local workstation or small office server
  • Hybrid private retrieval with approved cloud generation

Model sizing before model branding

Open-weight does not mean office-local. Current frontier examples such as the 2.8-trillion-parameter Kimi K3 and the GLM 5.2 family can be valuable to evaluate, but managed inference or major accelerator infrastructure is more realistic than a normal business workstation.

  • Separate compact local models from frontier-scale open models
  • Confirm whether weights are actually available before designing self-hosting
  • Compare managed API cost with hardware, power, networking, and support

Local document assistant

Ingest approved documents, create local embeddings, retrieve relevant passages, and show citations while respecting user permissions and document lifecycle rules.

  • Ollama or another local inference runtime when appropriate
  • Open WebUI or a custom interface
  • Vector search, source links, and document updates

Agent and device access

OpenClaw is designed around a personal-assistant trust boundary. It can be useful for a single owner or a tightly controlled prototype, but it should not be treated as a hostile multi-tenant security boundary for unrelated employees or customers.

  • Use one trusted operator boundary per host or deployment
  • Sandbox tools and dedicate browser profiles when practical
  • Limit channels, host access, remote exposure, and recovery authority

Hardware and operations

Size hardware for the models and concurrency the business actually needs. Consumer GPUs can work for limited office workloads, but they are not a substitute for operations planning.

  • GPU memory, system memory, storage, networking, and power
  • Remote access, monitoring, and replacement planning
  • Model updates, rollback, and performance testing

Security boundaries

Keep inference and agent endpoints off the public internet unless there is a carefully designed need, use identity and least privilege, protect secrets, validate model and container sources, and log access without capturing unnecessary sensitive content.

  • Network segmentation and authenticated access
  • Patch, dependency, plugin, and model provenance review
  • Backups, incident response, shutdown, and secure disposal

Good fit

A real privacy, offline, control, or predictable-volume requirement.

  • Sensitive internal documents should not leave an approved environment
  • The company needs offline or low-latency access at a known location
  • A predictable workload can justify hardware and support costs
  • The business accepts responsibility for operations and security controls

Usually not a good first project

What Vertex Authority will push back on.

  • Choosing local AI only because cloud AI feels vaguely unsafe
  • Expecting a small office computer to run frontier-scale Kimi or GLM models economically
  • Exposing Ollama, Open WebUI, OpenClaw, or another inference or agent endpoint directly to the internet
  • Running a private system with no patching, backup, identity, logging, or owner

Next step

Describe the workflow before choosing the tool.

Share the current process, software, volume, and the time or revenue impact. Zach will reply with the most sensible next step.

Evaluate private AI options