Skip to content

Commit b4317ee

Browse files
authored
Merge pull request #632 from AsakusaRinne/master
Release version 0.11.0
2 parents f1cb0e3 + 53092eb commit b4317ee

File tree

202 files changed

+11822
-5740
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

202 files changed

+11822
-5740
lines changed

.github/prepare_release.sh

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,5 +77,8 @@ do
7777
nuget pack $nuspec -version $updated_version
7878
done
7979

80+
# write the version to the file
81+
echo $updated_version > version.txt
82+
8083
cd ..
8184
exit 0

.github/workflows/release-minor.yml

Lines changed: 24 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,11 @@ jobs:
1515
runs-on: ubuntu-latest
1616

1717
steps:
18-
- uses: actions/checkout@v3
18+
- uses: actions/checkout@v4
19+
- name: Configure Git Credentials
20+
run: |
21+
git config user.name Rinne
22+
git config user.email [email protected]
1923
- name: Setup NuGet
2024
uses: nuget/setup-nuget@v1
2125
with:
@@ -51,3 +55,22 @@ jobs:
5155

5256
- name: Push LLamaSharp packages to nuget.org
5357
run: dotnet nuget push ./temp/LLamaSharp*.nupkg --source https://www.nuget.org -k ${{ secrets.LLAMA_SHARP_NUGET_KEY }} --skip-duplicate
58+
59+
# Deploy the documentation to GitHub Pages
60+
- uses: actions/setup-python@v5
61+
with:
62+
python-version: 3.x
63+
- run: echo "cache_id=$(date --utc '+%V')" >> $GITHUB_ENV
64+
- uses: actions/cache@v4
65+
with:
66+
key: mkdocs-material-${{ env.cache_id }}
67+
path: .cache
68+
restore-keys: |
69+
mkdocs-material-
70+
- run: pip install mkdocs==1.4.3 mkdocs-material mike==1.1.2 setuptools
71+
- run: |
72+
git fetch origin gh-pages --depth=1
73+
version=$(cat ./temp/version.txt)
74+
mike deploy --push --update-aliases --force $version latest
75+
mike set-default --push --force latest
76+

.github/workflows/release-patch.yml

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,11 @@ jobs:
1515
runs-on: ubuntu-latest
1616

1717
steps:
18-
- uses: actions/checkout@v3
18+
- uses: actions/checkout@v4
19+
- name: Configure Git Credentials
20+
run: |
21+
git config user.name Rinne
22+
git config user.email [email protected]
1923
- name: Setup NuGet
2024
uses: nuget/setup-nuget@v1
2125
with:
@@ -52,3 +56,21 @@ jobs:
5256
- name: Push LLamaSharp packages to nuget.org
5357
run: dotnet nuget push ./temp/LLamaSharp*.nupkg --source https://www.nuget.org -k ${{ secrets.LLAMA_SHARP_NUGET_KEY }} --skip-duplicate
5458

59+
# Deploy the documentation to GitHub Pages
60+
- uses: actions/setup-python@v5
61+
with:
62+
python-version: 3.x
63+
- run: echo "cache_id=$(date --utc '+%V')" >> $GITHUB_ENV
64+
- uses: actions/cache@v4
65+
with:
66+
key: mkdocs-material-${{ env.cache_id }}
67+
path: .cache
68+
restore-keys: |
69+
mkdocs-material-
70+
- run: pip install mkdocs==1.4.3 mkdocs-material mike==1.1.2 setuptools
71+
- run: |
72+
git fetch origin gh-pages --depth=1
73+
version=$(cat ./temp/version.txt)
74+
mike deploy --push --update-aliases --force $version latest
75+
mike set-default --push --force latest
76+

Assets/LLamaSharp-Integrations.png

28.8 KB
Loading

Assets/LLamaSharp-Integrations.vsdx

24.7 KB
Binary file not shown.

Assets/llava_demo.gif

6.98 MB
Loading

CONTRIBUTING.md

Lines changed: 58 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -2,21 +2,65 @@
22

33
Hi, welcome to develop LLamaSharp with us together! We are always open for every contributor and any format of contributions! If you want to maintain this library actively together, please contact us to get the write access after some PRs. (Email: [email protected])
44

5-
In this page, we'd like to introduce how to make contributions here easily. 😊
5+
In this page, we introduce how to make contributions here easily. 😊
66

7-
## Compile the native library from source
7+
## The goal of LLamaSharp
88

9-
Firstly, please clone the [llama.cpp](https://github.com/ggerganov/llama.cpp) repository and following the instructions in [llama.cpp readme](https://github.com/ggerganov/llama.cpp#build) to configure your local environment.
9+
At the beginning, LLamaSharp is a C# binding of [llama.cpp](https://github.com/ggerganov/llama.cpp). It provided only some wrappers for llama.cpp to let C#/.NET users could run LLM models on their local device efficiently even if without any experience with C++. After around a year of development, more tools and integrations has been added to LLamaSharp, significantly expanding the application of LLamaSharp. Though llama.cpp is still the only backend of LLamaSharp, the goal of this repository is more likely to be an efficient and easy-to-use library of LLM inference, rather than just a binding of llama.cpp.
1010

11-
If you want to support cublas in the compilation, please make sure that you've installed the cuda.
11+
In this way, our development of LLamaSharp is divided into two main directions:
1212

13-
When building from source, please add `-DBUILD_SHARED_LIBS=ON` to the cmake instruction. For example, when building with cublas but without openblas, use the following instruction:
13+
1. To make LLamaSharp more efficient. For example, `BatchedExecutor` could accept multiple queries and generate the response for them at the same time, which significantly improves the throughput. This part is always related with native APIs and executors in LLamaSharp.
14+
2. To make it easier to use LLamaSharp. We believe the best library is to let users build powerful functionalities with simple code. Higher-level APIs and integrations with other libraries are the key points of it.
15+
16+
17+
## How to compile the native library from source
18+
19+
If you want to contribute to the first direction of our goal, you may need to compile the native library yourself.
20+
21+
Firstly, please follow the instructions in [llama.cpp readme](https://github.com/ggerganov/llama.cpp#build) to configure your local environment. Most importantly, CMake with version higher than 3.14 should be installed on your device.
22+
23+
Secondly, clone the llama.cpp repositories. You could manually clone it and checkout to the right commit according to [Map of LLamaSharp and llama.cpp versions](https://github.com/SciSharp/LLamaSharp?tab=readme-ov-file#map-of-llamasharp-and-llama.cpp-versions), or use clone the submodule of LLamaSharp when cloning LLamaSharp.
24+
25+
```shell
26+
git clone --recursive https://github.com/SciSharp/LLamaSharp.git
27+
```
28+
29+
If you want to support cublas in the compilation, please make sure that you've installed it. If you are using Intel CPU, please check the highest AVX ([Advanced Vector Extensions](https://en.wikipedia.org/wiki/Advanced_Vector_Extensions)) level that is supported by your device.
30+
31+
As shown in [llama.cpp cmake file](https://github.com/ggerganov/llama.cpp/blob/master/CMakeLists.txt), there are many options that could be enabled or disabled when building the library. The following ones are commonly used when using it as a native library of LLamaSharp.
32+
33+
```cpp
34+
option(BUILD_SHARED_LIBS "build shared libraries") // Please always enable it
35+
option(LLAMA_NATIVE "llama: enable -march=native flag") // Could be disabled
36+
option(LLAMA_AVX "llama: enable AVX") // Enable it if the highest supported avx level is AVX
37+
option(LLAMA_AVX2 "llama: enable AVX2") // Enable it if the highest supported avx level is AVX2
38+
option(LLAMA_AVX512 "llama: enable AVX512") // Enable it if the highest supported avx level is AVX512
39+
option(LLAMA_BLAS "llama: use BLAS") // Enable it if you want to use BLAS library to acclerate the computation on CPU
40+
option(LLAMA_CUDA "llama: use CUDA") // Enable it if you have CUDA device
41+
option(LLAMA_CLBLAST "llama: use CLBlast") // Enable it if you have a device with CLBLast or OpenCL support, for example, some AMD GPUs.
42+
option(LLAMA_VULKAN "llama: use Vulkan") // Enable it if you have a device with Vulkan support
43+
option(LLAMA_METAL "llama: use Metal") // Enable it if you are using a MAC with Metal device.
44+
option(LLAMA_BUILD_TESTS "llama: build tests") // Please disable it.
45+
option(LLAMA_BUILD_EXAMPLES "llama: build examples") // Please disable it.
46+
option(LLAMA_BUILD_SERVER "llama: build server example")// Please disable it.
47+
```
48+
49+
Most importantly, `-DBUILD_SHARED_LIBS=ON` must be added to the cmake instruction and other options depends on you. For example, when building with cublas but without openblas, use the following instruction:
1450
1551
```bash
52+
mkdir build && cd build
1653
cmake .. -DLLAMA_CUBLAS=ON -DBUILD_SHARED_LIBS=ON
54+
cmake --build . --config Release
1755
```
1856

19-
After running `cmake --build . --config Release`, you could find the `llama.dll`, `llama.so` or `llama.dylib` in your build directory. After pasting it to `LLamaSharp/LLama/runtimes` you can use it as the native library in LLamaSharp.
57+
Now you could find the `llama.dll`, `libllama.so` or `llama.dylib` in your build directory (or `build/bin`).
58+
59+
To load the compiled native library, please add the following code to the very beginning of your code.
60+
61+
```cs
62+
NativeLibraryConfig.Instance.WithLibrary("<Your native library path>");
63+
```
2064

2165

2266
## Add a new feature to LLamaSharp
@@ -39,19 +83,19 @@ You could use exactly the same prompt, the same model and the same parameters to
3983

4084
If the experiment showed that it worked well in llama.cpp but didn't in LLamaSharp, a search for the problem could be started. While the reason of the problem could be various, the best way I think is to add log-print in the code of llama.cpp and use it in LLamaSharp after compilation. Thus, when running LLamaSharp, you could see what happened in the native library.
4185

42-
After finding out the reason, a painful but happy process comes. When working on the BUG fix, there's only one rule to follow, that is keeping the examples working well. If the modification fixed the BUG but impact on other functions, it would not be a good fix.
43-
44-
During the BUG fix process, please don't hesitate to discuss together when you stuck on something.
86+
During the BUG fix process, please don't hesitate to discuss together when you are blocked.
4587

4688
## Add integrations
4789

48-
All kinds of integration are welcomed here! Currently the following integrations are under work or on our schedule:
90+
All kinds of integration are welcomed here! Currently the following integrations have been added but still need improvement:
91+
92+
1. semantic-kernel
93+
2. kernel-memory
94+
3. BotSharp (maintained in SciSharp/BotSharp repo)
95+
4. Langchain (maintained in tryAGI/LangChain repo)
4996

50-
1. BotSharp
51-
2. semantic-kernel
52-
3. Unity
97+
If you find another library that is good to be integrated, please open an issue to let us know!
5398

54-
Besides, for some other integrations, like `ASP.NET core`, `SQL`, `Blazor` and so on, we'll appreciate it if you could help with that. If the time is limited for you, providing an example for it also means a lot!
5599

56100
## Add examples
57101

LLama.Examples/Examples/ChatChineseGB2312.cs

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33

44
namespace LLama.Examples.Examples;
55

6+
// This example shows how to deal with Chinese input with gb2312 encoding.
67
public class ChatChineseGB2312
78
{
89
private static string ConvertEncoding(string input, Encoding original, Encoding target)

LLama.Examples/Examples/ChatSessionStripRoleName.cs

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@
22

33
namespace LLama.Examples.Examples;
44

5+
// When using chatsession, it's a common case that you want to strip the role names
6+
// rather than display them. This example shows how to use transforms to strip them.
57
public class ChatSessionStripRoleName
68
{
79
public static async Task Run()

LLama.Examples/Examples/InstructModeExecute.cs

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
namespace LLama.Examples.Examples
44
{
5+
// This example shows how to use InstructExecutor to generate the response.
56
public class InstructModeExecute
67
{
78
public static async Task Run()

LLama.Examples/Examples/InteractiveModeExecute.cs

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
namespace LLama.Examples.Examples
44
{
5+
// This is an example which shows how to chat with LLM with InteractiveExecutor.
56
public class InteractiveModeExecute
67
{
78
public static async Task Run()

LLama.Examples/Examples/LlavaInteractiveModeExecute.cs

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@
55

66
namespace LLama.Examples.Examples
77
{
8+
// This example shows how to chat with LLaVA model with both image and text as input.
9+
// It uses the interactive executor to inference.
810
public class LlavaInteractiveModeExecute
911
{
1012
public static async Task Run()

LLama.Examples/Examples/LoadAndSaveState.cs

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
namespace LLama.Examples.Examples
44
{
5+
// This example shows how to save/load state of the executor.
56
public class LoadAndSaveState
67
{
78
public static async Task Run()

LLama.KernelMemory/LLamaSharp.KernelMemory.csproj

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
<TargetFrameworks>net6.0;net7.0;net8.0</TargetFrameworks>
55
<ImplicitUsings>enable</ImplicitUsings>
66
<Nullable>enable</Nullable>
7-
<Version>0.8.0</Version>
7+
<Version>0.11.0</Version>
88
<Authors>Xbotter</Authors>
99
<Company>SciSharp STACK</Company>
1010
<GeneratePackageOnBuild>true</GeneratePackageOnBuild>
@@ -17,7 +17,7 @@
1717
The integration of LLamaSharp and Microsoft kernel-memory. It could make it easy to support document search for LLamaSharp model inference.
1818
</Description>
1919
<PackageReleaseNotes>
20-
Support integration with kernel-memory
20+
v0.11.0 updated the kernel-memory package and Fixed System.ArgumentException: EmbeddingMode must be true.
2121
</PackageReleaseNotes>
2222
<PackageLicenseExpression>MIT</PackageLicenseExpression>
2323
<PackageOutputPath>packages</PackageOutputPath>

LLama.SemanticKernel/LLamaSharp.SemanticKernel.csproj

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
<ImplicitUsings>enable</ImplicitUsings>
1111
<Nullable>enable</Nullable>
1212

13-
<Version>0.8.0</Version>
13+
<Version>0.11.0</Version>
1414
<Authors>Tim Miller, Xbotter</Authors>
1515
<Company>SciSharp STACK</Company>
1616
<GeneratePackageOnBuild>true</GeneratePackageOnBuild>
@@ -23,7 +23,7 @@
2323
The integration of LLamaSharp and Microsoft semantic-kernel.
2424
</Description>
2525
<PackageReleaseNotes>
26-
Support integration with semantic-kernel
26+
v0.11.0 updates the semantic-kernel package.
2727
</PackageReleaseNotes>
2828
<PackageLicenseExpression>MIT</PackageLicenseExpression>
2929
<PackageOutputPath>packages</PackageOutputPath>

LLama/LLamaSharp.csproj

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,8 @@
77
<Platforms>AnyCPU;x64;Arm64</Platforms>
88
<AllowUnsafeBlocks>True</AllowUnsafeBlocks>
99

10-
<Version>0.10.0</Version>
11-
<Authors>Yaohui Liu, Martin Evans, Haiping Chen</Authors>
10+
<Version>0.11.0</Version>
11+
<Authors>Rinne, Martin Evans, jlsantiago and all the other contributors in https://github.com/SciSharp/LLamaSharp/graphs/contributors.</Authors>
1212
<Company>SciSharp STACK</Company>
1313
<GeneratePackageOnBuild>true</GeneratePackageOnBuild>
1414
<Copyright>MIT, SciSharp STACK $([System.DateTime]::UtcNow.ToString(yyyy))</Copyright>
@@ -17,11 +17,12 @@
1717
<PackageIconUrl>https://avatars3.githubusercontent.com/u/44989469?s=200&amp;v=4</PackageIconUrl>
1818
<PackageTags>LLama, LLM, GPT, ChatGPT, NLP, AI, Chat Bot, SciSharp</PackageTags>
1919
<Description>
20-
The .NET binding of LLama.cpp, making LLM inference and deployment easy and fast. For model
21-
weights to run, please go to https://github.com/SciSharp/LLamaSharp for more information.
20+
LLamaSharp is a cross-platform library to run 🦙LLaMA/LLaVA model (and others) in your local device.
21+
Based on [llama.cpp](https://github.com/ggerganov/llama.cpp), inference with LLamaSharp is efficient on both CPU and GPU.
22+
With the higher-level APIs and RAG support, it's convenient to deploy LLM (Large Language Model) in your application with LLamaSharp.
2223
</Description>
2324
<PackageReleaseNotes>
24-
LLamaSharp 0.10.0 supports automatically device feature detection, adds integration with kernel-memory and fixes some performance issues.
25+
LLamaSharp 0.11.0 added support for multi-modal (LLaVA), improved the BatchedExecutor and added state management of `ChatSession`.
2526
</PackageReleaseNotes>
2627
<PackageLicenseExpression>MIT</PackageLicenseExpression>
2728
<PackageOutputPath>packages</PackageOutputPath>

0 commit comments

Comments
 (0)