Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Build Index from regex #125

Draft
wants to merge 21 commits into
base: main
Choose a base branch
from
Draft

Build Index from regex #125

wants to merge 21 commits into from

Conversation

torymur
Copy link
Contributor

@torymur torymur commented Dec 12, 2024 •
edited
Loading

Rust

  • Build Index from regex with regex-automata
  • Change Token type from String to Vec<u8>
  • Change type of Vocabulary's eos_token_id from Option<TokenId> to TokenId
  • Stabilize Index interface
  • Stabilize Index python binding interface
  • Stabilize Vocabulary interface
  • Stabilize Vocabulary python binding interface
  • Drop all intermediary structures related to interregular workflow
  • Introduce new Guide python binding
  • Add and/or port tests
  • Add docs

Python

  • Drop all intermediary structures related to interregular workflow
  • Fix existing tests to use different interface of Guide, Index & Vocabulary
  • Adjust benchmarks to use new interface

@torymur torymur added the enhancement New feature or request label Dec 12, 2024
@rlouf
Copy link
Member

rlouf commented Dec 13, 2024

Supersedes #38?

@torymur
Copy link
Contributor Author

torymur commented Dec 13, 2024

Yes, it will

@torymur torymur force-pushed the index-from-regex-97 branch 2 times, most recently from fa36be9 to 5b8808d Compare December 18, 2024 15:50
@torymur torymur force-pushed the index-from-regex-97 branch 3 times, most recently from 88d609b to 52edc6d Compare January 7, 2025 17:54
torymur added 21 commits January 9, 2025 18:07
@torymur
Build Index from regex
606b460
@torymur
Test Index from regex in Guide
bdc120d
@torymur
Use FxHash* as default Hash*
6c5b853
@torymur
Cleaner from_regex logic
f349404
@torymur
Use bytes as Token type, more tests for Index
15a85aa
@torymur
Drop majority of intermediate structures
f02faec
@torymur
Add PyGuide, use proper types for Index
70b4bc6
@torymur
Provide basic Guide binding, test it
b477598
@torymur
Improve Vocabulary python binding, add tests
f3266ee
@torymur
Non-optional eos_token_id
7edb831
@torymur
Stabilize vocabulary interface
03e5561
@torymur
Add tests for Guide
64f0d73
@torymur
Python vocabulary to accept pretrained params
2ab0007
@torymur
Correct interface in pyi, reprs for all python bindings
063d1c2
@torymur
Adjust benchmarks
f65d86f
@torymur
Drop unused dependencies
30e29ef
@torymur
Index by ref in Guide
e04e5be
@torymur
Extend interface of python bindings
7b6781b
@torymur
Disallow insert of eos token into Vocabulary
1fab872
@torymur
Stabilize Index interfaces
15a45c0
@torymur
Use new interface in statistical
bf6e8a6
@torymur torymur force-pushed the index-from-regex-97 branch from 1a8831a to bf6e8a6 Compare January 9, 2025 18:12
torymur
torymur commented Jan 9, 2025
View reviewed changes
tests/fsm/test_statistical.py
mask: List[int] = [1 if s in allowed else 0 for s in range(1, n_tokens + 1)]
tokens = model(tokens, mask=mask)
state = fsm.get_next_state(state, tokens[-1])
allowed = guide.read_next_token(tokens[-1])
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dpsimpson Could you, please, take a look at this failing test: https://github.com/dottxt-ai/outlines-core/actions/runs/12695987037/job/35389081344

Interface has changed and I updated it here somewhat accordingly, but it needs to be checked, for example I added third value to Vocabulary (instead of eos token before that) just for the sake of keeping the dimensions right, I suspect that could be incorrect 😅

But overall, since its statistical, it goes over my head to fully understand the intentions and adjust properly expected testing values or construction logic, so I would really appreciate your help here 🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
2 participants
@torymur @rlouf