Oobabooga (LLM webui)

a large language model(llm) learns to predict the next word in a sentence by analyzing the patterns and structures in the text it has been trained on this enables it to generate human like text based on the input it receives there are many popular open source llms falcon 40b, guanaco 65b, llama and vicuna hugging face maintains a leaderboard https //huggingface co/spaces/huggingfaceh4/open llm leaderboard of the most popular open source models that they have available oobabooga https //github com/oobabooga/text generation webui is a front end that uses gradio to serve a simple web ui for interacting with the open source model in this guide, we will show you how to run an llm using oobabooga on vast 1\) setup your vast account the first thing to do if you are new to vast is to create an account and verify your email address then head to the billing tab and add credits vast uses stripe to processes credit card payments and also accepts major cryptocurrencies through coinbase or crypto com $20 should be enough to start you can setup auto top ups so that your credit card is charged when your balance is low 2\) pick the oobabooga template go to the templates tab and search for "oobabooga" among recommended templates and select it 3\) allocate storage the default storage amount will not be enough for downloading an llm use the slider under the instance configuration to allocate more storage 100gb should be enough ooobaboogasize 4\) pick a gpu offer you will need to understand how much gpu ram the llm requires before you pick a gpu for example, the falcon 40b instruct https //huggingface co/tiiuae/falcon 40b instruct model requires 85 100 gb of gpu ram falcon 7b only requires 16gb other models do not have great documentation on how much gpu ram they require if the instance doesn't have enough gpu ram, there will be an error when trying to load the model you can use multiple gpus in a single instance and add their gpu ram together for this guide, we will load the falcon 40b instruct model on a 2x a6000 instance, which has 96gb of gpu ram in total oobaboogasearch click on the rent button to start the instance which will download the docker container and boot up 5\) open oobabooga once the instance boots up, the open button will open port 7860 in a new browser window this is the oobabooga web interface the web gui can take an additional 1 2 minutes to load if the button is stuck on "connecting" for more than 10 minutes, then something has gone wrong you can check the log for an error and/or contact us on website chat support for 24/7 help 6\) download the llm click on the model tab in the interface enter the hugging face username/model path, for instance tiiuae/falcon 40b instruct to specify a branch, add it at the end after a " " character like this tiiuae/falcon 40b instruct the download will take 15 20 minutes depending on the machine's internet connection oob downloading to check the progress of the download, you can click on the log button on the vast instance card on cloud vast ai/instances/ https //cloud vast ai/instances/ which will show you the download speed for each of the llm file segments 7\) load the llm if you are using multiple gpus such as the 2x a6000 selected in this guide, you will need to move the memory slider all the way over for all the gpus you may also have to select the "trust remote code" option if you get that error once those items are fixed, you can reload the model oob model load any errrors loading the model will appear under the download button 8\) start chatting! navigate to the text generation tab to start chatting with the model this is the most basic way to use oobabooga, there are many other settings and things you can do with the interface 9\) done? destroy the instance if you stop the instance using the stop button, you will no longer pay the hourly gpu charges however you will still incur storage charges because the data is still stored on the host machine when you hit the start button to restart the instance, you are also not guaranteed that you can rent the gpu as someone else might have rented it while it was stopped to incur no other charges you have to destroy the instance using the trash can icon we recommend you destroy instances so as not to incur storage charges while you are not using the system have fun!