Feature/repeat penalty #20
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Should probably still scale by temp even if penalized
I see that numbers can go negative so a fix from a referenced commit
ggerganov
reviewed
Mar 12, 2023
|
Maybe should reconsider bringing back the Lines 75 to 89 in 129c7d1 I decided to use |
abetlen
pushed a commit
to abetlen/llama.cpp
that referenced
this pull request
Apr 10, 2023
fix: Mention of incorrect filename for MacOS cmake build artifact
SlyEcho
pushed a commit
to SlyEcho/llama.cpp
that referenced
this pull request
Jun 12, 2023
Logging changes
flowgrad
pushed a commit
to flowgrad/llama.cpp
that referenced
this pull request
Jun 27, 2023
* Cuda:
1) changed tensor split calculation to work on free vram instead of total vram (the main GPU often consumes a few GB from desktop work)
Changed the loading progress callback:
1) Added a numerical percentage to it
2) Added a status message to display what it is doing right now ("loading tensors: CPU)
3) normalized the length of the bar and designed it from dots to an actual progress bar
Example :
[============================================>-----] 88% Loading tensor (GPU-Split)
Todo:
The progressbar can be changed into 3-4 lines, so it displays important flags and changes alongside the progress.
This would replace the list of "log entries" before the progress starts counting and preceding log-lines are only printed for errors and warnings.
* 1) Added a tensor split prepare wrapper, this allows to take the tensor split given by -ts immediately
2) Created a CUDA status struct with getter and update functions.
Moved g_main_device and g_num_devices into the new struct.
typedef struct {
int num_devices;
int main_device_id;
size_t total_vram;
size_t total_free_vram;
struct cudaDeviceProp device_props[GGML_CUDA_MAX_DEVICES];
size_t device_vram_free[GGML_CUDA_MAX_DEVICES];
size_t device_vram_total[GGML_CUDA_MAX_DEVICES];
} GPUStatus;
3) Replaced the previous log output with a print function that gives better CUDA information on init
Example:
CUDA Device Summary - 2 devices found
+------------------------------------+------------+-----------+-----------+-----------+-----------+
| Device | VRAM Total | VRAM Free | VRAM Used | Split % | Device ID |
+------------------------------------+------------+-----------+-----------+-----------+-----------+
| NVIDIA GeForce RTX 4090 | 24563 MB | 23006 MB | 1557 MB | 0.0% | 0 (Main) |
+------------------------------------+------------+-----------+-----------+-----------+-----------+
| NVIDIA GeForce RTX 3090 | 24575 MB | 23318 MB | 1257 MB | 67.2% | 1 |
+------------------------------------+------------+-----------+-----------+-----------+-----------+
Total VRAM: 47.99 GB, Free VRAM: 45.24 GB
--------------------
4) cleaned up libfalcon.cpp to use the new information instead of making cuda requests and device changes
TODO: The VRAM reported by CUDA is not the same as reported by CPU-Z which also is different from HWINFO64 (this one is totally off).
* CUDA:
1) Tensor split now allows to disable either device (usage proportion of 0)
- this currently still occupies minimal VRAM and if main-device it will still be used for non split tensors
2) Corrected (reduced) total free VRAM calculation if a device proportion is disabled (does not take main device into account)
* vram_total bugfix
* status table alignment
* bugfixes
* added perf counters also into makefile
---------
phymbert
referenced
this pull request
Mar 23, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #4
I tried hacking together penalizing repeated tokens for n parameters. I'm actually not sure of the correct approach and I am not a great C++ programmer! But it appears to work somewhat. I think this can be improved much more with better sampling code.