Expose an inference API

Key concepts

• The API is how an application calls the model. OpenAI-compatible servers commonly expose a /v1 base path.
• Check GET /v1/models and POST /v1/chat/completions where supported. Verify streaming, tool calls, embeddings, and errors individually.
• Test connectivity, authentication, timeouts, and latency from the application or user network.

Checkpoint

Coding needs interactive streaming, while document jobs may run for ten minutes. How should clients access inference?

1. Private-network pilot

   Fits a small set of controlled development clients.

   Tradeoffs · Simple deployment leaves client configuration and error handling to each application.

   Verify · Verify authorization, disconnects, streaming, and tool calls from the real laptop.

2. Shared API gateway

   Fits multiple applications needing credential rotation, quotas, and traceability.

   Tradeoffs · Adds an operational layer whose streaming and timeout behavior needs testing.

   Verify · Test quotas, interrupted streams, denied requests, and sensitive-log redaction.

3. Separate interactive and queued jobs

   Fits long documents, overnight batches, and retryable jobs.

   Tradeoffs · Requires job state, cancellation, retries, and deduplication.

   Verify · Interrupt a worker and verify recovery without duplicate external effects.

Hands-on check

Make one authorized inference request from the laptop and record the base URL, model ID, and supported features.

References

• vLLM API
• Ollama API

Illustration credits

• PRIVATE API · AI-generated concept illustration. Verify actual specifications in product documentation.
• API GATEWAY · AI-generated concept illustration. Verify actual specifications in product documentation.
• JOB QUEUE · AI-generated concept illustration. Verify actual specifications in product documentation.

← Back to the field lab