The new Sonnet model definitely kills GPT-4o in the published benchmarks.
I evaluated it for real-world use cases and compared against GPT-4o. It did better in all the cases.<p>Test Case 1: Python Code Generation
Write a script to generate email address from name and domain<p>Test Case 2: Web Page creation
Create an HTML file that displays a simple personal portfolio webpage. The webpage should include a header with your name, a profile picture, a brief introduction about yourself, and a list of your skills. Use basic HTML tags to structure the content and include some inline CSS to style the elements<p>Test Case 3: API Query Generation
Write a cURL to call dall-e-3 API, and generate image of a Unicorn with a rainbow horn<p>Assessment:
- Sonnet provides a more direct response to the coding requests. When we asked for a cURL command, Claude directly gave that, whereas GPT-4o created a bash script.
- The web page created by Claude was much more aesthetically pleasing, and almost readily usable. Great for non-tech folks who want to create web pages.
- Python code generation: This one is hard to say, both perform well. GPT-4o needs a bit more detailed instructions.
- Pricing: Claude is cheaper than GPT-4o ($3 per million input tokens vs $5 per million tokens for GPT-4o)
- Speed: Claude is faster at generating the first token.<p>Here's a detailed write up: https://blog.getbind.co/2024/06/21/claude-3-5-sonnet-does-it-outperform-gpt-4o/