Why I tend not to use content negotiation

Cloudflare doesn&#x27;t obey the Accept header on anything other than image content types.<p>This means if you have an endpoint that returns HTML or JSON depending on the requested content type and you try to serve it from behind Cloudflare you risk serving cached JSON to HTML user agents or vice-versa.<p>I dropped the idea of supporting content negotiation from my Datasette project because of this.<p>(And because I personally don&#x27;t like that kind of endpoint - I want to be able to know if a specific URL is going to return JSON or HTML).

I agree with the underlying point being made here: a separate API (REST&#x2F;JSON) that serves data should be kept separate from the endpoints that serve an application, in this case, HTML-based.<p>It seems like building your application around a single API that is also used to provide data externally saves you time, but you end up polluting that API with presentation concerns needed to drive the application&#x27;s reports&#x2F;grids&#x2F;views. It&#x27;s not worth the mental energy to consider how changes you need to make for presentation might affect the &#x27;purity&#x27; of your public API. Returning hypermedia from the &#x27;internal&#x27; API just forces that separation: there&#x27;s no expectation that this &#x27;data&#x27; is being returned for consumption by anything except the app that uses it.

I think content negotiation is great when your usecase supports it, like asking for XML or JSON. Also the mappings of content types should be well defined for all to see and edit. Rails is kinda a labyrinth in that regard.<p>I do tend to prefer actual file extensions though. Friendlier for humans (curl <a href="https:&#x2F;&#x2F;endpoint&#x2F;item.json" rel="nofollow noreferrer">https:&#x2F;&#x2F;endpoint&#x2F;item.json</a> vs curl -H &quot;accept: application&#x2F;json&quot; <a href="https:&#x2F;&#x2F;endpoint&#x2F;item" rel="nofollow noreferrer">https:&#x2F;&#x2F;endpoint&#x2F;item</a>), and it is visible in logging infrastructure, sharable, etc.

I really enjoy these blog posts from htmx.org. Makes me better at creating applications faster, because they challenge my conventional wisdom that slows me down without offering a compelling advantage.<p>Side note: I guess a better title for this blog post would be &quot;<i>Don&#x27;t share endpoints between machines and humans</i>&quot;. If the consumer of an endpoint is a human, it&#x27;s probably a bad idea to make machines use the same endpoint. Content negotiation is fine, for example if I want to have an API return binary data instead of JSON (if, for example, I&#x27;m on an IoT device with sparse resources). This is fine because it&#x27;s exactly the same API, with just the data format being different. And in both cases it&#x27;s consumed by a machine&#x2F;computer.<p>In the case of the frontend&#x2F;HTML, the consumer is a human (and not a machine), which — as the article mentions — adds a bunch of constraints that are not applicable to machines: a need for pagination (because scrolling through a thousand results can be annoying), a need for ordering (because I want the most relevant results first), displaying related content (as the article explains). The machine API doesn&#x27;t necessarily need any of these features.

So, this is actually about: Why I Tend Not To Use Content Negotiation (to serve both HTML and JSON data from the same endpoints).<p>The author also suggests:<p>&gt; The alternative is to ... splitting your APIs. This means providing different paths (or sub-domains, or whatever) for your JSON API and your hypermedia (HTML) API.<p>I believe the alternative has been the norm actually. For example, many front-end frameworks encode UI states in URL, and it&#x27;s not so sustainable to keep the alignment b&#x2F;w UI states and data APIs in the long term.

Content negotiation is surely a fire composed of tires but there are sadly many scenarios where it is necessary. Deciding whether to redirect for example is often context sensitive.<p>In other words content negotiation is useful to be able to respond intelligibly. If a client asks you for json but not html, it might not make sense to return html.

A technology like htmx seems to demand a “hypermedia” API. Whereas things like angular and react can consume a data API in many cases. However once an application becomes sufficiently complex people end up building data APIs just to suit their frontend framework. And in that case doing htmx and returning html seems nicer.

Content negotiation is useful when there are multiple feasible return formats. But when is that the case?<p>For data, Json is the absolute king. For content, html is king. There is very little to negotiate.<p>The only case where I needed that feature was when data scientists wanted to download data from my API and needed a bunch of formats (parquet, CSV, TSV). But then they did not really grok content negotiation and asked for a query param. So finally I think this is like a lot of html features: half baked and from a different time. Html would do well to drop it.

&gt; Your JSON API should remain stable. You can’t be adding and removing end-points willy-nilly. Yes, you can have some end-points respond with either JSON or HTML and others only respond with HTML, but it gets messy. What if you accidentally copy-and-paste in the wrong code somewhere, for example.<p>Can&#x27;t we handle this situation by regarding each version of the JSON output as a separate content type (which is arguably the semantically correct thing anyway) and then letting the server pick the more recent output version that the client supports?

To me this just feels wasteful. Most APIs only ever return one type of thing. So the business of asking for application&#x2F;json with every request and then the API confirming that, yes, it&#x27;s still sending application&#x2F;json;charset=utf-8 just seems a pointless waste of bandwidth. Same with API versioning. Most APIs are stuck at v1 forever. It never changes. It never gets verified. It gets hard coded all over the place. All for the option to, maybe introduce a v2 on the same server. Seems like a lot of premature optimization for not a whole lot of gain.

I get this on one level but wow can you tell the difference between when a team uses the API they offer users and when they don’t, and the “single API” approach gets you there so well.<p>I have basically never seen a nice user-facing API when it’s been split out. Sometimes that’s fine, but at least for enterprise use cases having a “real” API just feels like table stakes in so many domains for getting bigger clients onboard.

Tangential, but the lack of a &quot;these types are available&quot; as the counterpart to &quot;Accept&quot; has always been so obviously missing. What&#x27;s the point of Accept if you don&#x27;t know what to ask for?<p>Maybe in the trivial case saying &quot;I&#x27;d prefer a JPG to a PNG&quot; can be an assymetrical choice. But in all the interesting use cases I can think of, e.g. where there are competing representation formats, you&#x27;d want the server to be able to respond to a HEAD with the choices.<p>That&#x27;s the kind of thing you can put in Swagger, but that might lead to hoisting the client&#x27;s choice into the API, away from Content Negotiation.

&gt; Data APIs should be rate limited<p>Hypermedia APIs should be rate limited as well, because otherwise people will just go and screen scrape (like many HN apps do, because HN doesn&#x27;t offer an API). All a &quot;data&quot; API does is make the scraper&#x27;s job easier.<p>&gt; Data APIs typically use some sort of token-based authentication &#x2F; Hypermedia APIs typically use some sort of session-cookie based authentication<p>So what. Any web framework worth its salt can support multiple authentication &#x2F; credential mechanisms - the only &quot;benefit&quot; I can see from limiting cookie authentication is to make life for bad actors with cookie-stealer malware harder (like GitLab does, IIRC).

Isn&#x27;t the name &quot;Data Application Programming Interface&quot; redundant ?<p>And &quot;Hypermedia Application Programming Interface&quot; wrong, because generally not for an application at all, but rather a (non-programming, as the author says, &quot;for humans&quot;) interface to display multimedia documents ? (I guess you get (inevitable, if not necessarily good) feature creep as soon as you start including something like forms - see also : forms in pdfs ?)

Split it out but keep using the same models. Wouldn’t want your entities and semantics to drift apart between your UI and API.<p>Personally I prefer sticking to the standards. At least that way when you move between projects you know what you’re getting into.<p>But everyone has their own conventions these days. It’s all fragmenting.

At a very basic level, isn&#x27;t it just confusing to have the same URL and method return two different completely different formats?<p>I feel like a good API is limited in what it accepts, and this alone is enough to say one should not do content negotiation unless forced to.

Carson Gross&#x27; main thesis is this: &quot;I am advocating tightly coupling your web application to your hypermedia API&quot;. In other words &quot;have an API that returns HTML snippets tightly coupled to your website.&quot;<p>It&#x27;s ironic he wrote an article about how the industry uses the term &quot;REST API&quot; incorrectly, because he himself keeps using the term &quot;API&quot; incorrectly. If an &quot;API&quot; is tightly coupled to a single application, it&#x27;s not an &quot;Application Programming Interface&quot;... it&#x27;s just a part of your application.<p>An API is supposed to be an interface on top of which <i>multiple</i> applications may rest. Particularly without a specific frontend in mind - so web, desktop app, mobile app, as a component of other services and so on. Obviously if it serves site-specific HTML snippets, that&#x27;s not the case. The only reason he advocates this whole thing is because without it HTMX won&#x27;t work, and in this way I find it quite myopic as a position. But if I was pushing HTMX I&#x27;d also be compelled to figure out reasons to make it sound good.<p>So from that PoV, talking about &quot;Content Negotiation in HTML APIs&quot; loses meaning, as what he has is not an API in the first place, it&#x27;s just his HTML website, but with some partial requests in the mix. And <i>of course</i> you wouldn&#x27;t mix your API and your site. But this <i>does not</i> imply you can&#x27;t and shouldn&#x27;t use Content Negotiation either on your site, or in your API. You simply shouldn&#x27;t use them to mix two things that never made sense to mix.<p>A lot of his blog posts would become completely unnecessary if he just says &quot;don&#x27;t mix your website and your API, and the HTMX partial requests are part of your website, not your API&quot;. Alas he&#x27;s stuck on this odd formulation of &quot;hypermedia API&quot; and constantly having to clarify himself and making things as clear as mud.

&gt;Not being content with alienating only the general purpose JSON API enthusiasts, let me now proceed to also alienate my erstwhile hypermedia enthusiast allies<p>Nah I think he&#x27;s right and it&#x27;s coherent to avoid HTTP subtleties in that web architecture.