So I have 10,000 paragraphs I want to individually process with a prompt.<p>Any packages that would handle running things async, doing retries, handling the occasional hung connection, and logging performance?<p>Tried helicone, but I don't think it handles the hung connections.<p>Just doing it manually for now, but there must be an existing solution?
You could set up a collection in memory/SQL with a Status + UpdatedUtc column and poll the collection for incomplete items each loop until all are in the desired state.<p>Your state machine could be as simple as: New, Processing, Failed, Succeeded. Outer loop will query the collection every ~second for items that are New or Failed and retry them. Items that are stuck Processing for more than X seconds should be forced to Failed each loop through (you'll retry them on the next pass). Each state transition is written to a log with timestamps for downstream reporting. Failures are exclusively set by the HTTP processing machinery with timeouts being detected as noted above.<p>Using SQL would make iterations of your various batch processing policies substantially easier. Using a SELECT statement to determine your batch at each iteration would permit adding constraints over aggregates. For example, you could cap the # of simultaneous in-flight requests, or abandon all hope & throw if the statistics are looking poor (aka OpenAI outage).
I'm working on a Python package for this: <a href="https://github.com/qagentur/texttunnel">https://github.com/qagentur/texttunnel</a><p>It's a wrapper for OpenAI's Python sample script plus adjacent functionality like cost estimation and binpacking multiple inputs into one request.
They have an example python script for batch jobs with retries. <a href="https://github.com/openai/openai-cookbook/blob/main/examples/api_request_parallel_processor.py">https://github.com/openai/openai-cookbook/blob/main/examples...</a>
What is the average paragraph length?<p>I would consider using one of the 32k context windows.<p>Define a delimiter for the paragraphs and prefix the prompt to process each then write out with a delimiter the result you need.<p>Maybe wrap your call in a simple try catch and do exponential back-off.
Did you try asking chatgpt to write it? :)<p>For async and retries I just asked the bot to write proper code, and it worked fine. Could do it myself, but it would take longer.<p>With hung connections - I didn't experience those, but it should also be straightforward.