Geniusrise is a modular, loosely-coupled AI-microservices framework.<p>It can be used to perform various tasks, including hosting inference endpoints, performing bulk inference, fine tune etc with open source models or closed source APIs.<p>- The framework provides structure for modules and operationalizes and orchestrates them.
- The modular ecosystem provides a layer of abstraction over the myriad of models, libraries, tools, parameters and optimizations underlying the operationalization of modern AI models.<p>1. Install geniusrise and libs<p><pre><code> pip install torch
pip install geniusrise
pip install geniusrise-vision # vision multi-modal models
pip install geniusrise-text # text models, LLMs
pip install geniusrise-audio # audio models
</code></pre>
2. Create YAML file<p><pre><code> version: '1'
bolts:
my_multimodal_api:
name: VisualQAAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: 'llava-hf/bakLlava-v1-hf'
model_class: 'LlavaForConditionalGeneration'
processor_class: 'AutoProcessor'
device_map: 'cuda:0'
use_cuda: True
precision: 'bfloat16'
quantization: 0
max_memory: None
torchscript: False
compile: False
flash_attention: False
better_transformers: False
endpoint: '*'
port: 3000
cors_domain: 'http://localhost:3000'
username: 'user'
password: 'password'
</code></pre>
3. Launch API service<p><pre><code> genius rise
MY_IMAGE=/path/to/test/image
(base64 -w 0 $MY_IMAGE | awk '{print "{\"image_base64\": \""$0"\", \"question\": \"<image>\nUSER: Whats the content of the image?\nASSISTANT:\", \"do_sample\": false, \"max_new_tokens\": 128}"}' > /tmp/image_payload.json)
curl -X POST http://localhost:3000/api/v1/answer_question \
-H "Content-Type: application/json" \
-u user:password \
-d @/tmp/image_payload.json | jq
</code></pre>
More: <a href="https://docs.geniusrise.ai/guides/usage/" rel="nofollow">https://docs.geniusrise.ai/guides/usage/</a> and <a href="https://github.com/geniusrise/examples">https://github.com/geniusrise/examples</a>