TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

OpenAI's Tokenizer Page for OSS Models

3 pointsby rgs224over 1 year ago

1 comment

rgs224over 1 year ago
I kept having to debug prompt issues with open models<p>So I built OpenAI&#x27;s Tokenizer page for all tokenizers on HuggingFace: Llama, Mistral, GPT2, MPT, Persimmon, T5 etc<p><a href="http:&#x2F;&#x2F;tokenwiz.rahul.gs" rel="nofollow noreferrer">http:&#x2F;&#x2F;tokenwiz.rahul.gs</a>