HTTP requests in different languages

Posted on September 23, 2023
Tags: howto, web, emacs, python, julia, haskell, bash

This is a simple overview over how to issue GET and POST (and maybe some other) HTTP request with different tools and in different programming languages.

Python

In python the requests package is sort of the modern standard.

Quickstart — Requests documentation

GET

import requests as req
from pprint import pprint

url = "http://httpbin.org/get"

response = req.get(url)

print(f"status code: {response.status_code}")
pprint(response.json())

POST

import requests as req
from pprint import pprint

url = "http://httpbin.org/post"

data = {"key": "value"}

response = req.post(url, data)

print(f"status code: {response.status_code}")
pprint(response.json())

Curl

Curl is a common commandline tool for web requests.

GET

curl "http://www.example.com/when/junk.cgi?birthyear=1905&press=OK"
curl http://url1.example.com
curl "http://www.example.com/when/junk.cgi?birthyear=1905&press=OK"
curl --user name:password http://www.example.com

curl -H "Authorization: OAuth <ACCESS_TOKEN>" http://www.example.com
curl -H "Authorization: Bearer <ACCESS_TOKEN>" http://www.example.com

POST

curl -d 'name=admin&shoesize=12' http://example.com/
curl -T localfile http://example.com/new/resource/file
curl -d "data to PUT" -X PUT http://example.com/new/resource/file

Others

GET is default, using -d or -F makes it a POST, -I generates a HEAD and -T sends a PUT.

man curl

-X, –request <command> (HTTP) Specifies a custom request method to use when communicating with the HTTP server. The specified request method will be used instead of the method otherwise used (which defaults to GET). Read the HTTP 1.1 specification for details and explanations. Common additional HTTP requests include PUT and DELETE, but related technologies like WebDAV offers PROPFIND, COPY, MOVE and more.

https://everything.curl.dev/

Matlab

Matlab has simple functions wrapping http requests, webread and webwrite. Both can be passed a weboptions object (Specify parameters for RESTful web service - MATLAB weboptions).

GET

Read content from RESTful web service - MATLAB webread

url = "http://requestserver.mathworks.com/employee"

opts = weboptions("ContentType", "text")

response = webread(url, "firstName", "Sarah", opts);

POST

Write data to RESTful web service - MATLAB webwrite - MathWorks Deutschland

url = 'https://requestserver.mathworks.com';

data(1).Name = 'Jon';
data(1).Occupation = 'Doctor';
data(2).Name = 'Sarah';
data(2).Occupation = 'Engineer';

opts = weboptions('MediaType', 'application/json');

response = webwrite(url, data, opts)

Emacs Lisp

In Emacs Lisp http requests were a bit more tricky to learn as parameter passing is different compared to the packages from other languages. The standard package is url.el which one loads with:

(require 'url)

The main functions are

(url-retrieve URL CALLBACK &optional CBARGS SILENT INHIBIT-COOKIES)

for asynchronous requests, where you need to parse a callback function, and

(url-retrieve-synchronously URL &optional SILENT INHIBIT-COOKIES TIMEOUT)

which runs synchronously, so emacs halts and one needs to wait for the response.

Both functions take same getting used to as they use external variables and juggle response buffers around: url-retrieve-synchronously returns a buffer with the response data (of nil if there is no data).

EmacsWiki: Url Package

GET

(let* ((url "http://httpbin.org/get")
       (url-request-method "GET")
       (url-request-extra-headers `(("Content-Type" . "application/json")))
       (buffer (url-retrieve-synchronously url)))
  (switch-to-buffer buffer))

POST

For a POST one also sets the variable url-request-data, which is just a string.

(let* ((url "http://httpbin.org/post")
       (url-request-method "POST")
       (url-request-extra-headers `(("Content-Type" . "application/json")))
       (url-request-data "key=value")
       (buffer (url-retrieve-synchronously url)))
  (switch-to-buffer buffer))

(buffer-substring url-http-end-of-headers (point-max)))

Other

When writing a emacs-lisp package to interact with some rest apis, I stumbled upon a weird behaviour / bug, which was resolved by utf-8 encoding of url-request-method and url-request-data like in the following function:

(defun mnd/http-request (method url auth &optional data)
  "Make a HTTP request and return json string."
  (let* ((url-request-method (encode-coding-string method 'utf-8))
         (url-request-extra-headers
          `(("Authorization" . auth)
            ("Content-Type" . "application/json")))
         (url-request-data (when data (encode-coding-string data 'utf-8)))
         (buffer (url-retrieve-synchronously url)))
    (unwind-protect
        (with-current-buffer buffer
          (message "%s" url-request-data)
          (message "%s" (buffer-substring (point-min) url-http-end-of-headers))
          (buffer-substring url-http-end-of-headers (point-max)))
      (kill-buffer buffer))))

It also illustrates how the response buffer of url-retrieve-synchronously can be used further by first extracting the response data from it via

(buffer-substring url-http-end-of-headers (point-max))

Here are some links I collected that described the same problem, when multibyte strings are the url-request-data:

Emacs restclient

There is also a complete restclient written in emacs-lisp, GitHub - pashky/restclient.el: HTTP REST client tool for emacs, which is awesome and there is even an org-babel mode for it, GitHub - alf/ob-restclient.el: An org-mode extension to restclient.el. With these packages running http requests in a literate org document looks like this:

GET

GET http://httpbin.org/get

POST

POST http://httpbin.org/post
{"key": "value"}

Julia

Julia being the most modern language in this list, the API for http requests feels very nice, like it combines the benefits of all other packages. The package for http is HTTP.jl and using it look like this:

GET

import HTTP
import JSON

url = "http://httpbin.org/get"

query = ["key" => "value"]

response = HTTP.get(url, query=query)

print(JSON.parse(String(response.body)))

POST

import HTTP

url = "http://httpbin.org/post"

data = Dict("key" => "value")

response = HTTP.post(url; body=data)
print(response)

Haskell

I think in haskell a good goto package for web requests is wreq.

GET

#!/usr/bin/env stack
{- stack
script
--resolver lts-18.19
--package wreq
--package lens
--package aeson
--
+RTS -s -RTS
-}
{-# LANGUAGE OverloadedStrings #-}

import Network.Wreq
import Control.Lens

main = do
        r <- get "http://httpbin.org/get"
        print $ r ^. responseStatus . statusCode
        print $ r ^. responseBody

POST

#!/usr/bin/env stack
{- stack
script
--resolver lts-18.19
--package wreq
--package lens
--package aeson
--
+RTS -s -RTS
-}
{-# LANGUAGE OverloadedStrings #-}

import Network.Wreq
import Control.Lens
import Data.Aeson

main = do
        r <- post "http://httpbin.org/post" ["num" := (3 :: Int), "str" := ("wat" :: String)]
        print $ r ^. responseStatus
        print $ r ^. responseBody