Ghostwriter – use the reMarkable2 as an interface to vision-LLMs

Project author here -- happy to elaborate on anything; a continuous WIP project. The biggest insight has been limitations of vision models in spacial awareness -- see <a href="https:&#x2F;&#x2F;github.com&#x2F;awwaiid&#x2F;ghostwriter&#x2F;blob&#x2F;main&#x2F;evaluation_results&#x2F;2024-12-29_21-05-47&#x2F;results.md">https:&#x2F;&#x2F;github.com&#x2F;awwaiid&#x2F;ghostwriter&#x2F;blob&#x2F;main&#x2F;evaluation_...</a> for some sketchy examples of my rudimentary eval.<p>Next top things:<p>* Continue to build&#x2F;extract into a yaml+shellscript agentic framework&#x2F;tool<p>* Continue exploring pre-segmenting or other methods of spacial awareness<p>* Write a reSvg backend that sends actual pen-strokes instead of lots of dots

This is so cool! I love to see people hacking together apps for the reMarkable tablet<p>I made a little app for reMarkable too and I shared it here some time back: <a href="https:&#x2F;&#x2F;digest.ferrucc.io&#x2F;" rel="nofollow">https:&#x2F;&#x2F;digest.ferrucc.io&#x2F;</a>

I wish the remarkable tablets weren&#x27;t so locked down.<p>It&#x27;s one of my favorite pieces of hardware and wish there were more apps for it.

Awesome.<p>I wanted to try to implement this for months. You did a really good job.

This is so cool. I’m going to try it this weekend.<p>I’ve been playing with the idea of auto creating tasks when I write todos by emailing the PDF and sending it to an LLM.<p>This just opened up a whole realm of better ways to accomplish that goal in realtime.

How about this on android driven Onyx Boox ereaders? Would it be possible?

This is a brilliant use case—handwriting input combined with LLMs makes for a much more natural workflow. I wonder how well it handles messy handwriting and if fine-tuning on personal notes would improve recognition over time.

Love this! There are some vector diffusion models out there; why not use tool calling to outsource to one of those if the model decides to draw something? Then it could specify coordinate range and the prompt.

For PDF paper readers, is the Remarkable’s 11” size sufficient? I have the Sony DPT 2nd version at 13”, and it’s perfect viewing experience. But projects like this keep drawing me to the Remarkable product.

I own a boox tablet (full fledge Android tablet with eink screen), and this sort of things would be perfect for it. I wonder if in 5 years the mobile hw would support something like that locally!

Really cool. Would this run on the remarkable paper pro too?

&gt; <i>Things that worked at least once:</i><p>I like it.

Kinda unrelated but should I go for kobo or the remarkable? I mostly want to read papers and maybe take notes. How do tthey compare in terms of hackability and freedom?

I wonder if this can be abstracted to accept interaction from a Daylight too.

@apple.com add to iPadOS Notes?

Harry potter half-blood prince vibes. Interesting just how much the medium changes the feeling of interacting with a chat model

Not to distract from the project but if anyone is interested in eink tablets with LLMs, the ViWoods tablet might be of interest to you.