Y
Hacker News
new
|
ask
|
show
|
jobs
by
FlyingLawnmower
264 days ago
If your masking is fast enough, you can make it easily work with spec dec too :). We manage to keep this on CPU. Some details here:
https://github.com/guidance-ai/llguidance/blob/main/docs/opt...