How to set context_size for a model in Docker Desktop?

[gotofritz]

Hi. I am using Docker Desktop with the Open-WebUI extention to run models, but I hit the exceed_context_size_error. I believe this is due to the context_size being quite low. But in the documentation I only see ways to edit the context_size in docker compose, not the Desktop. Is it possible at all? And if not, any change you could make it possible? Thanks!

[doringeman]

Hey @gotofritz, you can use docker model package --from <oiriginal_model> --context-size <size> <final_model_id> (ref.).

[ilopezluna]

@gotofritz you can also use the cli:

docker model configure ai/smollm2 --context-size 100

docker model run ai/smollm2 "tell me a story of more than 100 tokens"
Once upon a time, in a small town called Codeville, lived a young programmer named Timmy. Timmy was a dreamer, always looking for ways to solve complex problems and bring them to life. One day, he decided to create a unique game that would challenge the best programmers in the world and inspire them to think creatively.

Timmy, along with his friends, embarked on a journey to find a game that would be challenging but rewarding. They studied the best games in the world and analyzed their structures, mechanics, and challenges. After many late nights and early mornings, they finally created a game called CodeMaster.

CodeMaster was a game that required players to write code in a specific format to solve a series of puzzles. The game was designed to be challenging yet rewarding, as each level would gradually increase the complexity of the puzzles. However, unlike traditional games, CodeMaster didn't just provide a series of answers, but instead, presented the solutions in a narrative format, which encouraged players to think creatively.

When CodeMaster was released,

// here we exceeded context size

docker model configure ai/smollm2 --context-size 1000

docker model run ai/smollm2 "tell me a story of more than 100 tokens"
Once upon a time, in a faraway land, there was a magical kingdom called "Lexiconia." The kingdom was inhabited by two peculiar creatures, the Lingua-Vox and the Lexicon-Gabriel. The Lingua-Vox were the master of words, while the Lexicon-Gabriel was the guardian of words.

One day, the Lingua-Vox and the Lexicon-Gabriel met in a grand ballroom filled with curious visitors. The guests were filled with excitement as they had never seen such a magnificent pair.

The Lingua-Vox, named Silvanus, stood on one side of the room with a long, slender sword. The Lexicon-Gabriel, named Felix, stood on the other side with a soft, golden touch. The guests were captivated by the two of them and wanted to learn more.

As the guests began to ask questions, Silvanus, who was the master of words, started to speak in a way that was just a little too poetic for the crowd. The guests were confused, but Silvanus was patient and explained what he meant.

Meanwhile, Felix, who was the guardian of words, tried to explain what he was doing but couldn't because he was too busy looking at the guests' hands. Silvanus noticed this and decided to show Felix his powers.

Silvanus handed Felix a small, soft book. Felix was amazed to see that the book was filled with many words. Felix then asked Silvanus, "What is the purpose of this book?"

Silvanus replied, "The book is a key to understanding the meanings of words. It shows the relationship between different words and their connections."

Felix was amazed and asked Silvanus, "How can you see all these words?"

Silvanus smiled and explained, "The book is a magical book that contains a special power called 'Knowledge.' It can show you the relationships between different words and how they can be combined to create new meanings."

Felix was amazed and asked Silvanus, "How can you use it to learn from others?"

Silvanus smiled and replied, "The book can be used to learn from others. By reading the words in the book, you can learn from their experiences and wisdom. It's like having a guide to the depths of knowledge!"

Felix was fascinated and asked Silvanus, "How can you use it to make decisions?"

[gotofritz]

Thanks, I will try that. I was hoping I could do it all from Docker Desktop without having to mess with the CLI.

[gotofritz]

Oh I get it, the command docker model configure ai/smollm2 --context-size 100 will change the settings once and Docker Desktop will pick them up immediately and use them. Thanks, that worked!