[server] Cleanup a memory leak on exit

[stevegrubb]

There are a couple memory leaks on exit of the server. This hides others. After cleaning this up, you can see leaks on slots. But that is another patch to be sent after this.

[github-actions]

📈 llama.cpp server for bench-server-baseline on Standard_NC4as_T4_v3 for phi-2-q4_0: 547 iterations 🚀

Expand details for performance related PR only

• Concurrent users: 8, duration: 10m
• HTTP request : avg=8558.73ms p(95)=21372.37ms fails=, finish reason: stop=487 truncated=60
• Prompt processing (pp): avg=99.19tk/s p(95)=377.32tk/s
• Token generation (tg): avg=32.39tk/s p(95)=46.85tk/s
• ggml-org/models/phi-2/ggml-model-q4_0.gguf parallel=8 ctx-size=16384 ngl=33 batch-size=2048 ubatch-size=256 pp=1024 pp+tg=2048 branch=exit-leak1 commit=820fe7e66778ed4e3299b6432dcac7e8396dd7a7

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 547 iterations"
    y-axis "llamacpp:prompt_tokens_seconds"
    x-axis "llamacpp:prompt_tokens_seconds" 1715436136 --> 1715436766
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 401.08, 401.08, 401.08, 401.08, 401.08, 800.47, 800.47, 800.47, 800.47, 800.47, 499.85, 499.85, 499.85, 499.85, 499.85, 545.91, 545.91, 545.91, 545.91, 545.91, 591.46, 591.46, 591.46, 591.46, 591.46, 615.95, 615.95, 615.95, 615.95, 615.95, 622.56, 622.56, 622.56, 622.56, 622.56, 653.3, 653.3, 653.3, 653.3, 653.3, 666.77, 666.77, 666.77, 666.77, 666.77, 684.58, 684.58, 684.58, 684.58, 684.58, 714.76, 714.76, 714.76, 714.76, 714.76, 736.23, 736.23, 736.23, 736.23, 736.23, 752.32, 752.32, 752.32, 752.32, 752.32, 754.1, 754.1, 754.1, 754.1, 754.1, 750.04, 750.04, 750.04, 750.04, 750.04, 753.85, 753.85, 753.85, 753.85, 753.85, 753.53, 753.53, 753.53, 753.53, 753.53, 775.23, 775.23, 775.23, 775.23, 775.23, 774.73, 774.73, 774.73, 774.73, 774.73, 773.41, 773.41, 773.41, 773.41, 773.41, 781.43, 781.43, 781.43, 781.43, 781.43, 784.6, 784.6, 784.6, 784.6, 784.6, 772.39, 772.39, 772.39, 772.39, 772.39, 772.84, 772.84, 772.84, 772.84, 772.84, 771.31, 771.31, 771.31, 771.31, 771.31, 779.04, 779.04, 779.04, 779.04, 779.04, 783.43, 783.43, 783.43, 783.43, 783.43, 783.49, 783.49, 783.49, 783.49, 783.49, 782.37, 782.37, 782.37, 782.37, 782.37, 785.67, 785.67, 785.67, 785.67, 785.67, 787.84, 787.84, 787.84, 787.84, 787.84, 786.27, 786.27, 786.27, 786.27, 786.27, 791.31, 791.31, 791.31, 791.31, 791.31, 804.59, 804.59, 804.59, 804.59, 804.59, 800.56, 800.56, 800.56, 800.56, 800.56, 806.66, 806.66, 806.66, 806.66, 806.66, 808.87, 808.87, 808.87, 808.87, 808.87, 806.99, 806.99, 806.99, 806.99, 806.99, 806.38, 806.38, 806.38, 806.38, 806.38, 809.54, 809.54, 809.54, 809.54, 809.54, 811.91, 811.91, 811.91, 811.91, 811.91, 815.49, 815.49, 815.49, 815.49, 815.49, 817.34, 817.34, 817.34, 817.34, 817.34, 798.97, 798.97, 798.97, 798.97, 798.97, 798.8, 798.8, 798.8, 798.8, 798.8, 796.74, 796.74, 796.74, 796.74, 796.74, 800.89, 800.89, 800.89, 800.89, 800.89, 803.25, 803.25, 803.25, 803.25, 803.25, 803.03, 803.03, 803.03, 803.03, 803.03, 808.57, 808.57, 808.57, 808.57, 808.57, 809.25, 809.25, 809.25, 809.25, 809.25, 806.64, 806.64, 806.64, 806.64, 806.64, 810.11, 810.11, 810.11, 810.11, 810.11, 809.92, 809.92, 809.92, 809.92, 809.92, 810.79, 810.79, 810.79, 810.79, 810.79, 812.36, 812.36, 812.36, 812.36, 812.36, 812.5, 812.5, 812.5, 812.5, 812.5, 812.84, 812.84, 812.84, 812.84, 812.84, 813.12, 813.12, 813.12, 813.12, 813.12, 812.89, 812.89, 812.89, 812.89, 812.89, 815.43, 815.43, 815.43, 815.43, 815.43, 815.43]
                    

Loading

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 547 iterations"
    y-axis "llamacpp:predicted_tokens_seconds"
    x-axis "llamacpp:predicted_tokens_seconds" 1715436136 --> 1715436766
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 41.85, 41.85, 41.85, 41.85, 41.85, 42.12, 42.12, 42.12, 42.12, 42.12, 37.1, 37.1, 37.1, 37.1, 37.1, 31.57, 31.57, 31.57, 31.57, 31.57, 31.53, 31.53, 31.53, 31.53, 31.53, 31.81, 31.81, 31.81, 31.81, 31.81, 33.2, 33.2, 33.2, 33.2, 33.2, 34.05, 34.05, 34.05, 34.05, 34.05, 34.16, 34.16, 34.16, 34.16, 34.16, 34.14, 34.14, 34.14, 34.14, 34.14, 34.14, 34.14, 34.14, 34.14, 34.14, 33.9, 33.9, 33.9, 33.9, 33.9, 33.37, 33.37, 33.37, 33.37, 33.37, 33.22, 33.22, 33.22, 33.22, 33.22, 32.24, 32.24, 32.24, 32.24, 32.24, 32.26, 32.26, 32.26, 32.26, 32.26, 32.39, 32.39, 32.39, 32.39, 32.39, 32.16, 32.16, 32.16, 32.16, 32.16, 31.49, 31.49, 31.49, 31.49, 31.49, 31.23, 31.23, 31.23, 31.23, 31.23, 31.18, 31.18, 31.18, 31.18, 31.18, 31.37, 31.37, 31.37, 31.37, 31.37, 31.47, 31.47, 31.47, 31.47, 31.47, 31.42, 31.42, 31.42, 31.42, 31.42, 31.44, 31.44, 31.44, 31.44, 31.44, 31.58, 31.58, 31.58, 31.58, 31.58, 31.23, 31.23, 31.23, 31.23, 31.23, 30.79, 30.79, 30.79, 30.79, 30.79, 30.66, 30.66, 30.66, 30.66, 30.66, 30.85, 30.85, 30.85, 30.85, 30.85, 31.05, 31.05, 31.05, 31.05, 31.05, 31.17, 31.17, 31.17, 31.17, 31.17, 31.33, 31.33, 31.33, 31.33, 31.33, 31.33, 31.33, 31.33, 31.33, 31.33, 31.25, 31.25, 31.25, 31.25, 31.25, 31.19, 31.19, 31.19, 31.19, 31.19, 30.74, 30.74, 30.74, 30.74, 30.74, 30.71, 30.71, 30.71, 30.71, 30.71, 30.8, 30.8, 30.8, 30.8, 30.8, 30.9, 30.9, 30.9, 30.9, 30.9, 30.99, 30.99, 30.99, 30.99, 30.99, 31.06, 31.06, 31.06, 31.06, 31.06, 30.99, 30.99, 30.99, 30.99, 30.99, 30.85, 30.85, 30.85, 30.85, 30.85, 29.85, 29.85, 29.85, 29.85, 29.85, 29.35, 29.35, 29.35, 29.35, 29.35, 29.29, 29.29, 29.29, 29.29, 29.29, 29.28, 29.28, 29.28, 29.28, 29.28, 29.31, 29.31, 29.31, 29.31, 29.31, 29.38, 29.38, 29.38, 29.38, 29.38, 29.52, 29.52, 29.52, 29.52, 29.52, 29.56, 29.56, 29.56, 29.56, 29.56, 29.57, 29.57, 29.57, 29.57, 29.57, 29.42, 29.42, 29.42, 29.42, 29.42, 29.41, 29.41, 29.41, 29.41, 29.41, 29.45, 29.45, 29.45, 29.45, 29.45, 29.62, 29.62, 29.62, 29.62, 29.62, 29.77, 29.77, 29.77, 29.77, 29.77, 29.88, 29.88, 29.88, 29.88, 29.88, 29.94, 29.94, 29.94, 29.94, 29.94, 30.01, 30.01, 30.01, 30.01, 30.01, 30.12]
                    

Loading

Details

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 547 iterations"
    y-axis "llamacpp:kv_cache_usage_ratio"
    x-axis "llamacpp:kv_cache_usage_ratio" 1715436136 --> 1715436766
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.12, 0.12, 0.12, 0.12, 0.12, 0.41, 0.41, 0.41, 0.41, 0.41, 0.22, 0.22, 0.22, 0.22, 0.22, 0.11, 0.11, 0.11, 0.11, 0.11, 0.23, 0.23, 0.23, 0.23, 0.23, 0.15, 0.15, 0.15, 0.15, 0.15, 0.11, 0.11, 0.11, 0.11, 0.11, 0.13, 0.13, 0.13, 0.13, 0.13, 0.18, 0.18, 0.18, 0.18, 0.18, 0.14, 0.14, 0.14, 0.14, 0.14, 0.22, 0.22, 0.22, 0.22, 0.22, 0.24, 0.24, 0.24, 0.24, 0.24, 0.1, 0.1, 0.1, 0.1, 0.1, 0.32, 0.32, 0.32, 0.32, 0.32, 0.13, 0.13, 0.13, 0.13, 0.13, 0.15, 0.15, 0.15, 0.15, 0.15, 0.18, 0.18, 0.18, 0.18, 0.18, 0.24, 0.24, 0.24, 0.24, 0.24, 0.31, 0.31, 0.31, 0.31, 0.31, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.29, 0.29, 0.29, 0.29, 0.29, 0.11, 0.11, 0.11, 0.11, 0.11, 0.16, 0.16, 0.16, 0.16, 0.16, 0.23, 0.23, 0.23, 0.23, 0.23, 0.35, 0.35, 0.35, 0.35, 0.35, 0.28, 0.28, 0.28, 0.28, 0.28, 0.13, 0.13, 0.13, 0.13, 0.13, 0.14, 0.14, 0.14, 0.14, 0.14, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.14, 0.14, 0.14, 0.14, 0.14, 0.28, 0.28, 0.28, 0.28, 0.28, 0.19, 0.19, 0.19, 0.19, 0.19, 0.33, 0.33, 0.33, 0.33, 0.33, 0.16, 0.16, 0.16, 0.16, 0.16, 0.17, 0.17, 0.17, 0.17, 0.17, 0.12, 0.12, 0.12, 0.12, 0.12, 0.13, 0.13, 0.13, 0.13, 0.13, 0.1, 0.1, 0.1, 0.1, 0.1, 0.33, 0.33, 0.33, 0.33, 0.33, 0.45, 0.45, 0.45, 0.45, 0.45, 0.51, 0.51, 0.51, 0.51, 0.51, 0.46, 0.46, 0.46, 0.46, 0.46, 0.42, 0.42, 0.42, 0.42, 0.42, 0.18, 0.18, 0.18, 0.18, 0.18, 0.17, 0.17, 0.17, 0.17, 0.17, 0.09, 0.09, 0.09, 0.09, 0.09, 0.17, 0.17, 0.17, 0.17, 0.17, 0.12, 0.12, 0.12, 0.12, 0.12, 0.17, 0.17, 0.17, 0.17, 0.17, 0.24, 0.24, 0.24, 0.24, 0.24, 0.22, 0.22, 0.22, 0.22, 0.22, 0.24, 0.24, 0.24, 0.24, 0.24, 0.13, 0.13, 0.13, 0.13, 0.13, 0.09, 0.09, 0.09, 0.09, 0.09, 0.14, 0.14, 0.14, 0.14, 0.14, 0.17, 0.17, 0.17, 0.17, 0.17, 0.11, 0.11, 0.11, 0.11, 0.11, 0.2, 0.2, 0.2, 0.2, 0.2, 0.24]
                    

Loading

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 547 iterations"
    y-axis "llamacpp:requests_processing"
    x-axis "llamacpp:requests_processing" 1715436136 --> 1715436766
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 2.0, 2.0, 2.0, 2.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 8.0, 8.0, 8.0, 8.0, 8.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 3.0, 3.0, 3.0, 3.0, 3.0, 6.0, 6.0, 6.0, 6.0, 6.0, 0.0, 0.0, 0.0, 0.0, 0.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 3.0, 3.0, 3.0, 3.0, 3.0, 2.0, 2.0, 2.0, 2.0, 2.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 2.0, 2.0, 2.0, 2.0, 2.0, 5.0, 5.0, 5.0, 5.0, 5.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 1.0, 1.0, 1.0, 1.0, 1.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 2.0, 2.0, 2.0, 2.0, 2.0, 0.0]
                    

Loading