October binary update #940
Conversation
…th `SafeLLamaSamplerChainHandle`/`SafeLLamaSamplerHandle`
…ip of the individual stages.
- Removed `llama_n_heads` (that's not in this version, it's in future versions)
|
Test application is running fine on:
I noticed seed was removed from the Edit |
| public void AddDistributionSampler(uint seed) | ||
| { | ||
| llama_sampler_chain_add(this, llama_sampler_init_dist(seed)); | ||
|
|
||
| [DllImport(NativeApi.libraryName, CallingConvention = CallingConvention.Cdecl)] | ||
| static extern IntPtr llama_sampler_init_dist(uint seed); | ||
| } |
There was a problem hiding this comment.
Individual samplers which need a seed now accept a seed parameter just for that sampler. For example llama_sampler_init_dist(uint seed);.
The seed is now set per-sampler, see my other comment for an example. I'm glad you mentioned this though, because I didn't pass that through to the higher level! I'll add a
Good catch, that method should have been removed too. |
- Exposed `Seed` property on `DefaultSamplingPipeline` - Added penalties to `DefaultSamplingPipeline` - Added sensible defaults (copied from llama.cpp "main" example) for `DefaultSamplingPipeline`
|
Can confirm the |
|
Test is running fine on:
|
Updated binaries to llama.cpp c35e586ea57221844442c65a1172498c54971cb0, built with https://github.com/SciSharp/LLamaSharp/actions/runs/10984319249
This is an unusually large change due to a redesign of the llama.cpp sampling API. The new sampling system has all been exposed in a safe way except for custom samplers, which are not implemented and will need to be tackled in a future update.
Assorted notable changes:
IInferenceParamssampling parameters, this must be done through a pipelineIInferenceParamsproperties broke it)Testing: