So, there has recently been a surge in entrepreneurs offering open-source inference as a service. There are days when I get more emails from vendors offering inference at discounted prices, than actual customer sign-ups.
While this is great, I can't help but notice that bootstrapped entrepreneurs are competing with GPU rich adapter companies which can serve open-source image inference for free all day long, stateless video inference is probably not sustainable either because large adapters can give subsidies on both open and closed models and sign larger contracts directly with the vendor, thus undercutting your price offering.
Most inference APIs are geared towards editors, developers or designers and none towards the end-user. At the current level of capabilities, with state-of-the-art scaffolding, consumer should just be able to prompt for a video they want with exactly the content they want, in any theme they want, instead of having to individually prompt and iterate for image, video and narration for individual scenes.
High level commercial APIs when available, are often not model composable at a fundamental level. For when such an API is provided, the user cannot choose the internal implementation hard-coded to proprietary vendor models.
An API that serves both Consumers and Professionals
The Samar API and the client app are designed to support users of all skills, backgrounds and experience levels, both in the usage method as well as in the render output.
The high level endpoints can be used directly by the consumer to generate a video from an idea on demand. (Inference provider can also make simpler custom interfaces, mobile apps over the API server which is more constrained and simpler to use)
Professional editors can also use the high-level endpoint as the near-perfect first draft and then go into Studio to make detailed customizations and edits.
Most users are already familiar with the Vidgenie Agent and Studio UI in the deployed production app. However, something most don't notice is that the standalone docker build also comes with a built in API server running by default on http://localhost:3002 which anyone can use to host their own local open router, serving 1-shot Text-to-video, Image-list-to-video API calls.
The all-in-one agent, studio and API surface designed for vendors who want to offer something unique, useful out-of-the-box and something no one else is already offering. (a super-set of all routers)
The Prompt-to-video pipeline offers the full matrix of 4 inference models, 5 image models and 5 video models. You can also add your own model adapters or constrain the offerings based on your business use-case for your deployed instance.
The API further provides "external" routes for one-off image, video generation, assistant calls as well as routes for transcribe audio and moderation with a large range of supported models. (this is a side-effect and not the main API offering, however can be used by all registered users via metered usage, production instance does not run inference locally but rather routes to the correct optimized adapter for the corresponding model request)
There are also APIs for high level tasks like search, recommendations, OCR etc. which can be enabled by adding config for corresponding model adapters.
For the more technically inclined, you can create compatible custom adapters for each part of the pipeline using your own local or third party model endpoints.
For curious hackers, dig into the source code (It is MIT licensed) and update the validation logic and system prompts to suit your own use-case. Here's how default vs. custom installation compare-
Use default adapters
When you start the docker installation, the setup container opens at http://localhost:8089 by default, you can pick and choose your favorite model adapters and add config for those. Choose from native adapters, third-party adapters or simply add a samsar-js API key as a universal fallback which allows inference via all the providers. User only needs to add config for the models they'd like to enable.
Minimal model configurations to run the text-to-video pipeline, require adapter for at-least an inference model, a text-to-image model, an image-to-video model , an eligible sound effect model and an eligible lip-sync model.
Build your own custom adapters
You can serve open-source inference models according to your GPU capacity while offloading remaining tasks to state-of-the-art public adapters via a native or a samsar-js adapter key.
Offloading tasks to native or local adapters can cut costs/run while still delivering state-of-the-art performance and agent run via the Samsar generic pipeline. There is already some scaffolding present for each stage to add custom adapters.
This enables bootstrapped inference providers to serve a unique production-grade offering with their custom clients, available no-where else in the market currently.
Users of the inference provider, can further create themed content with dedicated channels for various content types for distribution, or just to consume on their for information. Dedicated themed content pipelines could be created, connected to the API, to populate content in niche apps in various categories like - Anime creator, Food , Travel , Marketing, Ads , Documentary , Cinema etc.
If you are serving your model via local inference, simply ensure that the model endpoint is accessible via Fal API compatible request and response and payload structure, then simple add an additional adapter to the respective service (image-generator, video-generator etc.)
Preparing your deployment for hosting for External users
The default standalone docker build is meant for single or team usage and doesn't support handling external users yet, since it doesn't provide billing interface (User is billed at the various providers they configured for their instance) or a register UI, with a minimal bit of updates and running the docker container in "production" mode, inference provider can switch to the same view and state that production server uses and can even bill their external clients for usage. When opening up port access via Nginx / reverse-proxy, ensure that you have a strong admin password setup and setup SSL cert as well from within the deplyment launcher if you would like to put your instance behind your own domain (All configurable via startup wizard)
Since the code is fully open-source, external inference-provider can also set custom pricing for their adapter calls by applying their own multiplier over the underlying API adapter call pricing.
Check out the docs for the hosted endpoint here. The self-hosted endpoint will follow similar request, polling and result pattern, however you can set your own pricing with a little bit of code edits)
You can checkout the code here, including all the services available here-
samsarone
Hope folks enjoyed reading a brief introduction to the Samsar API service that is bundled as part of the monorepo.
I am also accepting contracts to build this or similar systems for enterprises and organization small or large this are interested in having their own technical knowledge-base/marketing content warehouse fully catalogued, editable and searchable. Send an email to hello@samsar.one to know more, or try creating some videos with default model settings in the hosted app here.