TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

GPT-4-Tokenizer-Visualizer

6 点作者 eleijonmarck将近 2 年前

1 comment

WilliamBerglund将近 2 年前
Hello (author here), I found the OpenAI tool (<a href="https:&#x2F;&#x2F;platform.openai.com&#x2F;tokenizer" rel="nofollow">https:&#x2F;&#x2F;platform.openai.com&#x2F;tokenizer</a> ) for visualising tokens really useful when working on token optimisation strategies (and other token related tasks). But unfortunately that site doesn&#x27;t support the tokeniser that GPT4 uses (currently).<p>So I threw together a stackblitz that uses the js-tiktoken to do the same thing.<p>Hopefully it will be useful for someone else too!<p>Try it here: <a href="https:&#x2F;&#x2F;react-ts-h9yrjh.stackblitz.io&#x2F;" rel="nofollow">https:&#x2F;&#x2F;react-ts-h9yrjh.stackblitz.io&#x2F;</a>