HTTP

Introduction

The Hypertext Transfer Protocol (HTTP) is the foundation for the World Wide Web.

Unless you've been living under a rock for the past 30 years, you have probably been using HTTP pretty much every day (maybe without knowing). If you are reading this through a web browser for example, you are using HTTP.

It is an application communication protocol allowing a client to present a 'request' to a server, which then sends back a 'response'.

HTTP Request/Response cycle

The point of these requests and responses at the end of the day is to enable systems to easily query resources on remote systems.

Methods

HTTP defines a bunch of standard 'methods' (sometimes also called 'verbs') that can be requested against resources:

GET
PUT
POST
PATCH
DELETE
and many more...

The first few are really the heart of the modern web. Viewing a page like the one you are reading right now involves issuing a 'GET' request to the corresponding webserver for the specific resource path (or locator).

Other methods serve different purposes: 'POST' is typically used to request the creation of a new resource, while 'DELETE', as the name implies, is mostly used to request the deletion of a given resource.

Headers

Just as we've seen above, HTTP allows us to access resources across the internet both for reading and writing. As such, we need a way to enforce some kind of authentication. We want to be able to verify that the originator of any request is in fact allowed to carry out the operation.

In addition to this problem, we will also want to add some meta-information about our request to give the server more information about exactly how to process it. Examples of this include the desired response format, management of the underlying connection, or data caching parameters.

To address all of these issues, HTTP was designed so that both requests and responses can (and usually do) contain 'headers' - additional information in the form of key-value pairs joined to the message.

One rather common header that you probably have already heard about is the 'Cookie' header.

URL's

Part of HTTP's power is the ability to access resources through a text-based 'locator' - a kind of address which basically tells us not only where the resource is (the 'locator' part, after the "://"), but also how to access it (the 'protocol' part, before the "://").

For example, let's consider the following URL, which points to the current page on the Dooba website:

https://dooba.io/pages/http

We can break it down into smaller parts:

'https' - Protocol to use ('how' to access the resource)
'://' - Delimiter between protocol and location parts
'dooba.io' - Host (and port, implicit in this case: 443 for 'https' protocol) on which the resource is to be found
'/pages/http' - Path to the actual resource on the host

When we combine all of this, we get this simple string of text which allows anyone to easily access the same resource.

Structure of an HTTP request

No time for pointless debates

Over the past years, one of the top technical debates among IT engineers has been the distinction between the concepts of "URL", "URI" and "URN". When it comes to Dooba, let us put an end to this nonsense right now. Addresses in the form 'protocol://resource-location' will be referred to as URLs, that's it. The original definitions from the RFCs are ambiguous and this is why we have been debating their meaning ever since.

Response codes

The key part of the response message is the HTTP response code (sometimes referred to as 'status code'), which indicates to the originator of a request whether it was successful.

Structure of an HTTP response

Some commonly known HTTP response codes are:

200 OK
301 Moved Permanently
302 Found (Moved Temporarily)
401 Unauthorized
404 Not Found
500 Server Error

The http library

Enough with the theory.

How can we use HTTP to access the web with Dooba? The answer lies in the http library.

This library allows us to create both clients and servers that communicate over HTTP. This works by using network sockets - go have a quick look if you're not yet familiar with the concept.

Because it relies on the generic socket system of Dooba, HTTP can be used seamlessly over any network device, provided it integrates with Dooba Sockets (like the ESP8266 WiFi modules for example).

Basic usage

As always, let's start by adding the http library to our dependencies in dfe.conf:

# ...
deps:
  - http

Once this is done, we can start using things from the '<http/client.h>' and '<http/server.h>' header files.

Both client and server features will require us to create a structure to hold information about our client / server. After initializing the structure through calls to http_client_* / http_server_ * functions, we can basically wait for something to happen (either actively or in the background).

The examples below present how to use both the client and server features. To keep things simple, they will assume a socket interface is available as 'wl0' (this should be provided by the substrate).

HTTP client example

 1 #include <http/client.h>
 2 
 3 #include "substrate.h"
 4 
 5 // Our HTTP Client
 6 struct http_client hc;
 7 
 8 // Main Initialization
 9 void init()
10 {
11 	struct http_arg *a;
12 	uint8_t i;
13 
14 	// Init Substrate
15 	substrate_init();
16 
17 	// POST some form data to 'http://eresse.net/example.php'
18 	if(http_post(&hc, wl0, "http://eresse.net/example.php?param1=%s", "test-value",
19 
20 		// Pass in some headers
21 		HTTP_CLIENT_HEADER, "User-Agent", "Example HTTP Client V.%i", 1,
22 		HTTP_CLIENT_HEADER, "X-Example-Header", "Some %s: %i", "value", 42,
23 
24 		// Add some form fields
25 		HTTP_CLIENT_FORMDATA, "name", "User #%i", 12,
26 		HTTP_CLIENT_FORMDATA, "likes", "raccoons",
27 
28 		// Always end with 0
29 		0)) { scli_printf("ERROR! Failed to send HTTP request :(\n"); return; }
30 
31 	// Actively wait for request to complete & Check Failed
32 	http_client_wait(&hc);
33 	if(http_client_failed(&hc)) { scli_printf("ERROR! Failed to complete HTTP request :(\n"); return; }
34 
35 	// Display Response Info
36 	scli_printf("Response code: %i (%t)\n", http_client_res_status(&hc), http_client_res_status_txt(&hc), http_client_res_status_txt_l(&hc));
37 	scli_printf("Headers (%i):\n", http_client_res_head_count(&hc));
38 	for(i = 0; i < http_client_res_head_count(&hc); i = i + 1)
39 	{
40 		a = http_client_res_head_i(&hc, i);
41 		scli_printf("   * [%t] -> [%t]\n", http_arg_name(&hc, a), http_arg_name_l(&hc, a), http_arg_val(&hc, a), http_arg_val_l(&hc, a));
42 	}
43 	scli_printf("Body: %i bytes\n", http_client_res_body_l(&hc));
44 	scli_printf(" ------------\n%t\n------------\n", http_client_res_body(&hc), http_client_res_body_l(&hc));
45 }
46 
47 // Main Loop
48 void loop()
49 {
50 	// Update Substrate
51 	substrate_loop();
52 }

Daemonizing (running in background)

There will be cases when we don't want to waste time actively waiting for the response, but rather continue our program's execution flow while letting the request complete in the background. Thanks to [Discover_Dooba#eloop|eloop] this is trivial. We can even specify a callback function to be executed whenever the request completes.

 1 #include <http/client.h>
 2 
 3 #include "substrate.h"
 4 
 5 // Our HTTP Client
 6 struct http_client hc;
 7 
 8 // On Request Complete (callback)
 9 void on_req_complete(void *user, struct http_client *c)
10 {
11 	struct http_arg *a;
12 	uint8_t i;
13 
14 	// Check Failed
15 	if(http_client_failed(c)) { scli_printf("ERROR! Failed to complete HTTP request :(\n"); return; }
16 
17 	// Display Response Info
18 	scli_printf("Response code: %i (%t)\n", http_client_res_status(c), http_client_res_status_txt(c), http_client_res_status_txt_l(c));
19 	scli_printf("Headers (%i):\n", http_client_res_head_count(c));
20 	for(i = 0; i < http_client_res_head_count(c); i = i + 1)
21 	{
22 		a = http_client_res_head_i(c, i);
23 		scli_printf("   * [%t] -> [%t]\n", http_arg_name(c, a), http_arg_name_l(c, a), http_arg_val(c, a), http_arg_val_l(c, a));
24 	}
25 	scli_printf("Body: %i bytes\n ------------\n%t\n ------------\n", http_client_res_body_l(c), http_client_res_body(c), http_client_res_body_l(c));
26 }
27 
28 // Main Initialization
29 void init()
30 {
31 	// Init Substrate
32 	substrate_init();
33 
34 	// POST some form data to 'http://eresse.net/example.php'
35 	if(http_post(&hc, wl0, "http://eresse.net/example.php?param1=%s", "test-value",
36 
37 		// Pass in some headers
38 		HTTP_CLIENT_HEADER, "User-Agent", "Example HTTP Client V.%i", 1,
39 		HTTP_CLIENT_HEADER, "X-Example-Header", "Some %s: %i", "value", 42,
40 
41 		// Add some form fields
42 		HTTP_CLIENT_FORMDATA, "name", "User #%i", 12,
43 		HTTP_CLIENT_FORMDATA, "likes", "raccoons",
44 
45 		// Always end with 0
46 		0)) { scli_printf("ERROR! Failed to send HTTP request :(\n"); return; }
47 
48 	// Daemonize (complete request in the background)
49 	http_client_daemonize(&hc, on_req_complete, 0);
50 }
51 
52 // Main Loop
53 void loop()
54 {
55 	// Update Substrate
56 	substrate_loop();
57 }

! WARNING ! When daemonizing a request, we must make sure the client structure (struct http_client hc; in our case) is a global variable. If we use a local variable (stored on the stack, in the frame of the executing function), it will get trashed as soon as we return from the function, and the background processing of the request will fail.

HTTP server example

 1 #include <http/server.h>
 2 
 3 #include "substrate.h"
 4 
 5 // Our HTTP Server
 6 struct http_server hs;
 7 
 8 // HTTP Request Handler
 9 void req_handler(void *user, struct http_server_client *c)
10 {
11 	uint8_t i;
12 	char ip[NET_IP_ADDR_LEN_TXT];
13 	uint16_t port;
14 	struct http_arg *a;
15 
16 	// Display Request Info
17 	scli_printf("Got HTTP Request! Details below:\n");
18 
19 	// Get Client Info
20 	if(socket_get_peer(c->conn.sck, ip, &port)) { scli_printf("ERROR: Failed to get peer info :(\n"); }
21 	else { scli_printf(" * Client: %s:%i\n", ip, port); }
22 
23 	// Display Request Details (Method + URL)
24 	scli_printf(" * Method: %s (%t)\n", http_code2meth(http_server_req_meth_code(c)), http_server_req_meth(c), http_server_req_meth_l(c));
25 	scli_printf(" * URL: %t\n", http_server_req_url(c), http_server_req_url_l(c));
26 	scli_printf("    - Path: %t\n", http_server_req_url_path(c), http_server_req_url_path_l(c));
27 	if(http_server_req_url_arg_count(c)) { scli_printf("    - Args: %t\n", http_server_req_url_args(c), http_server_req_url_args_l(c)); }
28 	for(i = 0; i < http_server_req_url_arg_count(c); i = i + 1)
29 	{
30 		a = http_server_req_url_arg_i(c, i);
31 		scli_printf("        %t -> %t\n", http_arg_name(c, a), http_arg_name_l(c, a), http_arg_val(c, a), http_arg_val_l(c, a));
32 	}
33 
34 	// Display Request Headers
35 	scli_printf(" * Headers:\n");
36 	for(i = 0; i < http_server_req_head_count(c); i = i + 1)
37 	{
38 		a = http_server_req_head_i(c, i);
39 		scli_printf("    - %t -> %t\n", http_arg_name(c, a), http_arg_name_l(c, a), http_arg_val(c, a), http_arg_val_l(c, a));
40 	}
41 
42 	// Display Form Fields
43 	if(http_server_req_field_count(c)) { scli_printf(" * Form Fields:\n"); }
44 	for(i = 0; i < http_server_req_field_count(c); i = i + 1)
45 	{
46 		a = http_server_req_field_i(c, i);
47 		scli_printf("    - %t -> %t\n", http_arg_name(c, a), http_arg_name_l(c, a), http_arg_val(c, a), http_arg_val_l(c, a));
48 	}
49 
50 	// Display Body
51 	if(http_server_req_body_l(c)) { scli_printf(" * Raw body: %i bytes\n ------------\n%t\n ------------\n", http_server_req_body_l(c), http_server_req_body(c), http_server_req_body_l(c)); }
52 
53 	// Respond with 200 - OK
54 	http_server_respond(c, HTTP_RES_OK);
55 
56 	// Set a 'Server' header
57 	http_server_header(c, HTTP_HEADER_SERVER, "Example HTTP Server v.%i.%i", 1, 3);
58 
59 	// Add a body - say hello!
60 	http_server_resp_body(c, "Hello world!");
61 }
62 
63 // Main Initialization
64 void init()
65 {
66 	// Init Substrate
67 	substrate_init();
68 
69 	// Create HTTP Server on Port 80 & Daemonize (run it in background)
70 	if(http_server_start(&hs, 80, wl0, req_handler, 0)) { scli_printf("ERROR! Failed to create HTTP Server :(\n"); return; }
71 	http_server_daemonize(&hs);
72 }
73 
74 // Main Loop
75 void loop()
76 {
77 	// Update Substrate
78 	substrate_loop();
79 }