# httpclient

- Manual
- Standard library
- Index

Search:
Group by:

This module implements a simple HTTP client that can be used to retrieve webpages and other data.

# Retrieving a website

This example uses HTTP GET to retrieve `http://google.com`:

    import httpClient
    var client = newHttpClient()
    echo client.getContent("http://google.com")

The same action can also be performed asynchronously, simply use the `AsyncHttpClient`:

    import httpClient
    var client = newAsyncHttpClient()
    echo await client.getContent("http://google.com")

The functionality implemented by `HttpClient` and `AsyncHttpClient` is the same, so you can use whichever one suits you best in the examples shown here.

Note: You will need to run asynchronous examples in an async proc otherwise you will get an `Undeclared identifier: 'await'` error.

# Using HTTP POST

This example demonstrates the usage of the W3 HTML Validator, it uses `multipart/form-data` as the `Content-Type` to send the HTML to be validated to the server.

    var client = newHttpClient()
    var data = newMultipartData()
    data["output"] = "soap12"
    data["uploaded_file"] = ("test.html", "text/html",
      "<html><head></head><body><p>test</p></body></html>")

    echo client.postContent("http://validator.w3.org/check", multipart=data)

You can also make post requests with custom headers. This example sets `Content-Type` to `application/json` and uses a json object for the body

    import httpclient, json

    let client = newHttpClient()
    client.headers = newHttpHeaders({ "Content-Type": "application/json" })
    let body = %*{
        "data": "some text"
    }
    let response = client.request("http://some.api", httpMethod = HttpPost, body = $body)
    echo response.status

# Progress reporting

You may specify a callback procedure to be called during an HTTP request. This callback will be executed every second with information about the progress of the HTTP request.

    import asyncdispatch, httpclient

    proc onProgressChanged(total, progress, speed: BiggestInt) {.async.} =
      echo("Downloaded ", progress, " of ", total)
      echo("Current rate: ", speed div 1000, "kb/s")

    proc asyncProc() {.async.} =
      var client = newAsyncHttpClient()
      client.onProgressChanged = onProgressChanged
      discard await client.getContent("http://speedtest-ams2.digitalocean.com/100mb.test")

    waitFor asyncProc()

If you would like to remove the callback simply set it to `nil`.

    client.onProgressChanged = nil

Warning: The `total` reported by httpclient may be 0 in some cases.

# SSL/TLS support

This requires the OpenSSL library, fortunately it's widely used and installed on many operating systems. httpclient will use SSL automatically if you give any of the functions a url with the `https` schema, for example: `https://github.com/`.

You will also have to compile with `ssl` defined like so: `nim c -d:ssl ...`.

# Timeouts

Currently only the synchronous functions support a timeout. The timeout is measured in milliseconds, once it is set any call on a socket which may block will be susceptible to this timeout.

It may be surprising but the function as a whole can take longer than the specified timeout, only individual internal calls on the socket are affected. In practice this means that as long as the server is sending data an exception will not be raised, if however data does not reach the client within the specified timeout a `TimeoutError` exception will be raised.

Here is how to set a timeout when creating an `HttpClient` instance:

    import httpclient

    let client = newHttpClient(timeout = 42)

# Proxy

A proxy can be specified as a param to any of the procedures defined in this module. To do this, use the `newProxy` constructor. Unfortunately, only basic authentication is supported at the moment.

Some examples on how to configure a Proxy for `HttpClient`:

    import httpclient

    let myProxy = newProxy("http://myproxy.network")
    let client = newHttpClient(proxy = myProxy)

Get Proxy URL from environment variables:

    import httpclient

    var url = ""
    try:
      if existsEnv("http_proxy"):
        url = getEnv("http_proxy")
      elif existsEnv("https_proxy"):
        url = getEnv("https_proxy")
    except ValueError:
      echo "Unable to parse proxy from environment variables."

    let myProxy = newProxy(url = url)
    let client = newHttpClient(proxy = myProxy)

# Redirects

The maximum redirects can be set with the `maxRedirects` of `int` type, it specifies the maximum amount of redirects to follow, it defaults to `5`, you can set it to `0` to disable redirects.

Here you can see an example about how to set the `maxRedirects` of `HttpClient`:

    import httpclient

    let client = newHttpClient(maxRedirects = 0)

# Imports

net, strutils, uri, parseutils, base64, os, mimetypes, math, random, httpcore, times, tables, streams, monotimes, asyncnet, asyncdispatch, asyncfile, nativesockets

# Types

    Response = ref object
      version*: string
      status*: string
      headers*: HttpHeaders
      body: string
      bodyStream*: Stream

  Source Edit

    AsyncResponse = ref object
      version*: string
      status*: string
      headers*: HttpHeaders
      body: string
      bodyStream*: FutureStream[string]

  Source Edit

    Proxy = ref object
      url*: Uri
      auth*: string

  Source Edit

    MultipartEntries = openArray[tuple[name, content: string]]

  Source Edit

    MultipartData = ref object
      content: seq[string]

  Source Edit

    ProtocolError = object of IOError

  exception that is raised when server does not conform to the implemented protocol Source Edit

    HttpRequestError = object of IOError

  Thrown in the `getContent` proc and `postContent` proc, when the server returns an error Source Edit

    ProgressChangedProc[ReturnType] = proc (total, progress, speed: BiggestInt): ReturnType {...}{.
        closure, gcsafe.}

  Source Edit

    HttpClientBase[SocketType] = ref object
      socket: SocketType
      connected: bool
      currentURL: Uri              ## Where we are currently connected.
      headers*: HttpHeaders        ## Headers to send in requests.
      maxRedirects: int
      userAgent: string
      timeout*: int                ## Only used for blocking HttpClient for now.
      proxy: Proxy                 ## ``nil`` or the callback to call when request progress changes.
      when SocketType is Socket:
        onProgressChanged
      else:
        onProgressChanged
      when false:
        sslContext
      contentTotal: BiggestInt
      contentProgress: BiggestInt
      oneSecondProgress: BiggestInt
      lastProgressReport: MonoTime
      when SocketType is AsyncSocket:
          bodyStream
          parseBodyFut

      else:
        bodyStream
      getBody: bool                ## When `false`, the body is never read in requestAux.

  Source Edit

    HttpClient = HttpClientBase[Socket]

  Source Edit

    AsyncHttpClient = HttpClientBase[AsyncSocket]

  Source Edit

# Consts

    defUserAgent = "Nim httpclient/1.0.6"

  Source Edit

# Procs

    proc code(response: Response | AsyncResponse): HttpCode {...}{.
        raises: [ValueError, OverflowError].}

  Retrieves the specified response's `HttpCode`.

  Raises a `ValueError` if the response's `status` does not have a corresponding `HttpCode`.

  Source Edit

    proc contentType(response: Response | AsyncResponse): string

  Retrieves the specified response's content type.

  This is effectively the value of the "Content-Type" header.

  Source Edit

    proc contentLength(response: Response | AsyncResponse): int

  Retrieves the specified response's content length.

  This is effectively the value of the "Content-Length" header.

  A `ValueError` exception will be raised if the value is not an integer.

  Source Edit

    proc lastModified(response: Response | AsyncResponse): DateTime

  Retrieves the specified response's last modified time.

  This is effectively the value of the "Last-Modified" header.

  Raises a `ValueError` if the parsing fails or the value is not a correctly formatted time.

  Source Edit

    proc body(response: Response): string {...}{.raises: [Defect, IOError, OSError],
                                        tags: [ReadIOEffect].}

  Retrieves the specified response's body.

  The response's body stream is read synchronously.

  Source Edit

    proc body(response: AsyncResponse): Future[string] {...}{.
        raises: [Exception, ValueError, FutureError], tags: [RootEffect].}

  Reads the response's body and caches it. The read is performed only once. Source Edit

    proc newProxy(url: string; auth = ""): Proxy {...}{.raises: [], tags: [].}

  Constructs a new `TProxy` object. Source Edit

    proc newMultipartData(): MultipartData {...}{.raises: [], tags: [].}

  Constructs a new `MultipartData` object. Source Edit

    proc add(p: var MultipartData; name, content: string; filename: string = "";
            contentType: string = "") {...}{.raises: [ValueError], tags: [].}

  Add a value to the multipart data. Raises a `ValueError` exception if `name`, `filename` or `contentType` contain newline characters. Source Edit

    proc add(p: var MultipartData; xs: MultipartEntries): MultipartData {...}{.discardable,
        raises: [ValueError], tags: [].}

  Add a list of multipart entries to the multipart data `p`. All values are added without a filename and without a content type.

      data.add({"action": "login", "format": "json"})

  Source Edit

    proc newMultipartData(xs: MultipartEntries): MultipartData {...}{.raises: [ValueError],
        tags: [].}

  Create a new multipart data object and fill it with the entries `xs` directly.

      var data = newMultipartData({"action": "login", "format": "json"})

  Source Edit

    proc addFiles(p: var MultipartData; xs: openArray[tuple[name, file: string]]): MultipartData {...}{.
        discardable, raises: [ValueError, IOError], tags: [ReadIOEffect].}

  Add files to a multipart data object. The file will be opened from your disk, read and sent with the automatically determined MIME type. Raises an `IOError` if the file cannot be opened or reading fails. To manually specify file content, filename and MIME type, use `[]=` instead.

      data.addFiles({"uploaded_file": "public/test.html"})

  Source Edit

    proc `[]=`(p: var MultipartData; name, content: string) {...}{.raises: [ValueError], tags: [].}

  Add a multipart entry to the multipart data `p`. The value is added without a filename and without a content type.

      data["username"] = "NimUser"

  Source Edit

    proc `[]=`(p: var MultipartData; name: string;
              file: tuple[name, contentType, content: string]) {...}{.raises: [ValueError],
        tags: [].}

  Add a file to the multipart data `p`, specifying filename, contentType and content manually.

      data["uploaded_file"] = ("test.html", "text/html",
        "<html><head></head><body><p>test</p></body></html>")

  Source Edit

    proc newHttpClient(userAgent = defUserAgent; maxRedirects = 5;
                      sslContext = getDefaultSSL(); proxy: Proxy = nil; timeout = -1): HttpClient {...}{.
        raises: [], tags: [].}

  Creates a new HttpClient instance.

  `userAgent` specifies the user agent that will be used when making requests.

  `maxRedirects` specifies the maximum amount of redirects to follow, default is 5.

  `sslContext` specifies the SSL context to use for HTTPS requests.

  `proxy` specifies an HTTP proxy to use for this HTTP client's connections.

  `timeout` specifies the number of milliseconds to allow before a `TimeoutError` is raised.

  Source Edit

    proc newAsyncHttpClient(userAgent = defUserAgent; maxRedirects = 5;
                           sslContext = getDefaultSSL(); proxy: Proxy = nil): AsyncHttpClient {...}{.
        raises: [], tags: [].}

  Creates a new AsyncHttpClient instance.

  `userAgent` specifies the user agent that will be used when making requests.

  `maxRedirects` specifies the maximum amount of redirects to follow, default is 5.

  `sslContext` specifies the SSL context to use for HTTPS requests.

  `proxy` specifies an HTTP proxy to use for this HTTP client's connections.

  Source Edit

    proc close(client: HttpClient | AsyncHttpClient)

  Closes any connections held by the HTTP client. Source Edit

    proc getSocket(client: HttpClient): Socket {...}{.raises: [], tags: [].}

  Get network socket, useful if you want to find out more details about the connection

  this example shows info about local and remote endpoints

      if client.connected:
        echo client.getSocket.getLocalAddr
        echo client.getSocket.getPeerAddr

  Source Edit

    proc getSocket(client: AsyncHttpClient): AsyncSocket {...}{.raises: [], tags: [].}

  Source Edit

    proc request(client: AsyncHttpClient; url: string; httpMethod: string; body = "";
                headers: HttpHeaders = nil): Future[AsyncResponse] {...}{.
        raises: [Exception, ValueError, FutureError], tags: [RootEffect, TimeEffect].}

  Connects to the hostname specified by the URL and performs a request using the custom method string specified by `httpMethod`.

  Connection will be kept alive. Further requests on the same `client` to the same hostname will not require a new connection to be made. The connection can be closed by using the `close` procedure.

  This procedure will follow redirects up to a maximum number of redirects specified in `client.maxRedirects`.

  Source Edit

    proc request(client: HttpClient; url: string; httpMethod: string; body = "";
                headers: HttpHeaders = nil): Response {...}{.raises: [ValueError,
        HttpRequestError, SslError, OSError, IOError, TimeoutError, ProtocolError,
        KeyError, Defect], tags: [ReadIOEffect, WriteIOEffect, TimeEffect].}

  Connects to the hostname specified by the URL and performs a request using the custom method string specified by `httpMethod`.

  Connection will be kept alive. Further requests on the same `client` to the same hostname will not require a new connection to be made. The connection can be closed by using the `close` procedure.

  This procedure will follow redirects up to a maximum number of redirects specified in `client.maxRedirects`.

  Source Edit

    proc request(client: AsyncHttpClient; url: string; httpMethod = HttpGet; body = "";
                headers: HttpHeaders = nil): Future[AsyncResponse] {...}{.
        raises: [Exception, ValueError, FutureError], tags: [RootEffect, TimeEffect].}

  Connects to the hostname specified by the URL and performs a request using the method specified.

  Connection will be kept alive. Further requests on the same `client` to the same hostname will not require a new connection to be made. The connection can be closed by using the `close` procedure.

  When a request is made to a different hostname, the current connection will be closed.

  Source Edit

    proc request(client: HttpClient; url: string; httpMethod = HttpGet; body = "";
                headers: HttpHeaders = nil): Response {...}{.raises: [ValueError,
        HttpRequestError, SslError, OSError, IOError, TimeoutError, ProtocolError,
        KeyError, Defect], tags: [ReadIOEffect, WriteIOEffect, TimeEffect].}

  Connects to the hostname specified by the URL and performs a request using the method specified.

  Connection will be kept alive. Further requests on the same `client` to the same hostname will not require a new connection to be made. The connection can be closed by using the `close` procedure.

  When a request is made to a different hostname, the current connection will be closed.

  Source Edit

    proc head(client: AsyncHttpClient; url: string): Future[AsyncResponse] {...}{.
        raises: [Exception, ValueError, FutureError], tags: [RootEffect, TimeEffect].}

  Connects to the hostname specified by the URL and performs a HEAD request.

  This procedure uses httpClient values such as `client.maxRedirects`.

  Source Edit

    proc head(client: HttpClient; url: string): Response {...}{.raises: [ValueError,
        HttpRequestError, SslError, OSError, IOError, TimeoutError, ProtocolError,
        KeyError, Defect], tags: [ReadIOEffect, WriteIOEffect, TimeEffect].}

  Connects to the hostname specified by the URL and performs a HEAD request.

  This procedure uses httpClient values such as `client.maxRedirects`.

  Source Edit

    proc get(client: AsyncHttpClient; url: string): Future[AsyncResponse] {...}{.
        raises: [Exception, ValueError, FutureError], tags: [RootEffect, TimeEffect].}

  Connects to the hostname specified by the URL and performs a GET request.

  This procedure uses httpClient values such as `client.maxRedirects`.

  Source Edit

    proc get(client: HttpClient; url: string): Response {...}{.raises: [ValueError,
        HttpRequestError, SslError, OSError, IOError, TimeoutError, ProtocolError,
        KeyError, Defect], tags: [ReadIOEffect, WriteIOEffect, TimeEffect].}

  Connects to the hostname specified by the URL and performs a GET request.

  This procedure uses httpClient values such as `client.maxRedirects`.

  Source Edit

    proc getContent(client: AsyncHttpClient; url: string): Future[string] {...}{.
        raises: [Exception, ValueError, FutureError], tags: [RootEffect, TimeEffect].}

  Connects to the hostname specified by the URL and returns the content of a GET request. Source Edit

    proc getContent(client: HttpClient; url: string): string {...}{.raises: [ValueError,
        HttpRequestError, SslError, OSError, IOError, TimeoutError, ProtocolError,
        KeyError, Defect, OverflowError],
        tags: [ReadIOEffect, WriteIOEffect, TimeEffect].}

  Connects to the hostname specified by the URL and returns the content of a GET request. Source Edit

    proc delete(client: AsyncHttpClient; url: string): Future[AsyncResponse] {...}{.
        raises: [Exception, ValueError, FutureError], tags: [RootEffect, TimeEffect].}

  Connects to the hostname specified by the URL and performs a DELETE request. This procedure uses httpClient values such as `client.maxRedirects`. Source Edit

    proc delete(client: HttpClient; url: string): Response {...}{.raises: [ValueError,
        HttpRequestError, SslError, OSError, IOError, TimeoutError, ProtocolError,
        KeyError, Defect], tags: [ReadIOEffect, WriteIOEffect, TimeEffect].}

  Connects to the hostname specified by the URL and performs a DELETE request. This procedure uses httpClient values such as `client.maxRedirects`. Source Edit

    proc deleteContent(client: AsyncHttpClient; url: string): Future[string] {...}{.
        raises: [Exception, ValueError, FutureError], tags: [RootEffect, TimeEffect].}

  Connects to the hostname specified by the URL and returns the content of a DELETE request. Source Edit

    proc deleteContent(client: HttpClient; url: string): string {...}{.raises: [ValueError,
        HttpRequestError, SslError, OSError, IOError, TimeoutError, ProtocolError,
        KeyError, Defect, OverflowError],
        tags: [ReadIOEffect, WriteIOEffect, TimeEffect].}

  Connects to the hostname specified by the URL and returns the content of a DELETE request. Source Edit

    proc post(client: AsyncHttpClient; url: string; body = "";
             multipart: MultipartData = nil): Future[AsyncResponse] {...}{.
        raises: [Exception, ValueError, FutureError], tags: [RootEffect, TimeEffect].}

  Connects to the hostname specified by the URL and performs a POST request. This procedure uses httpClient values such as `client.maxRedirects`. Source Edit

    proc post(client: HttpClient; url: string; body = ""; multipart: MultipartData = nil): Response {...}{.raises: [
        ValueError, HttpRequestError, SslError, OSError, IOError, TimeoutError,
        ProtocolError, KeyError, Defect],
        tags: [ReadIOEffect, WriteIOEffect, TimeEffect].}

  Connects to the hostname specified by the URL and performs a POST request. This procedure uses httpClient values such as `client.maxRedirects`. Source Edit

    proc postContent(client: AsyncHttpClient; url: string; body = "";
                    multipart: MultipartData = nil): Future[string] {...}{.
        raises: [Exception, ValueError, FutureError], tags: [RootEffect, TimeEffect].}

  Connects to the hostname specified by the URL and returns the content of a POST request. Source Edit

    proc postContent(client: HttpClient; url: string; body = "";
                    multipart: MultipartData = nil): string {...}{.raises: [ValueError,
        HttpRequestError, SslError, OSError, IOError, TimeoutError, ProtocolError,
        KeyError, Defect, OverflowError],
        tags: [ReadIOEffect, WriteIOEffect, TimeEffect].}

  Connects to the hostname specified by the URL and returns the content of a POST request. Source Edit

    proc put(client: AsyncHttpClient; url: string; body = ""; multipart: MultipartData = nil): Future[
        AsyncResponse] {...}{.raises: [Exception, ValueError, FutureError],
                        tags: [RootEffect, TimeEffect].}

  Connects to the hostname specified by the URL and performs a PUT request. This procedure uses httpClient values such as `client.maxRedirects`. Source Edit

    proc put(client: HttpClient; url: string; body = ""; multipart: MultipartData = nil): Response {...}{.raises: [
        ValueError, HttpRequestError, SslError, OSError, IOError, TimeoutError,
        ProtocolError, KeyError, Defect],
        tags: [ReadIOEffect, WriteIOEffect, TimeEffect].}

  Connects to the hostname specified by the URL and performs a PUT request. This procedure uses httpClient values such as `client.maxRedirects`. Source Edit

    proc putContent(client: AsyncHttpClient; url: string; body = "";
                   multipart: MultipartData = nil): Future[string] {...}{.
        raises: [Exception, ValueError, FutureError], tags: [RootEffect, TimeEffect].}

  Connects to the hostname specified by the URL andreturns the content of a PUT request. Source Edit

    proc putContent(client: HttpClient; url: string; body = "";
                   multipart: MultipartData = nil): string {...}{.raises: [ValueError,
        HttpRequestError, SslError, OSError, IOError, TimeoutError, ProtocolError,
        KeyError, Defect, OverflowError],
        tags: [ReadIOEffect, WriteIOEffect, TimeEffect].}

  Connects to the hostname specified by the URL andreturns the content of a PUT request. Source Edit

    proc patch(client: AsyncHttpClient; url: string; body = "";
              multipart: MultipartData = nil): Future[AsyncResponse] {...}{.
        raises: [Exception, ValueError, FutureError], tags: [RootEffect, TimeEffect].}

  Connects to the hostname specified by the URL and performs a PATCH request. This procedure uses httpClient values such as `client.maxRedirects`. Source Edit

    proc patch(client: HttpClient; url: string; body = ""; multipart: MultipartData = nil): Response {...}{.raises: [
        ValueError, HttpRequestError, SslError, OSError, IOError, TimeoutError,
        ProtocolError, KeyError, Defect],
        tags: [ReadIOEffect, WriteIOEffect, TimeEffect].}

  Connects to the hostname specified by the URL and performs a PATCH request. This procedure uses httpClient values such as `client.maxRedirects`. Source Edit

    proc patchContent(client: AsyncHttpClient; url: string; body = "";
                     multipart: MultipartData = nil): Future[string] {...}{.
        raises: [Exception, ValueError, FutureError], tags: [RootEffect, TimeEffect].}

  Connects to the hostname specified by the URL and returns the content of a PATCH request. Source Edit

    proc patchContent(client: HttpClient; url: string; body = "";
                     multipart: MultipartData = nil): string {...}{.raises: [ValueError,
        HttpRequestError, SslError, OSError, IOError, TimeoutError, ProtocolError,
        KeyError, Defect, OverflowError],
        tags: [ReadIOEffect, WriteIOEffect, TimeEffect].}

  Connects to the hostname specified by the URL and returns the content of a PATCH request. Source Edit

    proc downloadFile(client: HttpClient; url: string; filename: string) {...}{.raises: [
        ValueError, HttpRequestError, SslError, OSError, IOError, TimeoutError,
        ProtocolError, KeyError, Defect, Exception, OverflowError],
        tags: [ReadIOEffect, WriteIOEffect, TimeEffect].}

  Downloads `url` and saves it to `filename`. Source Edit

    proc downloadFile(client: AsyncHttpClient; url: string; filename: string): Future[void] {...}{.
        raises: [FutureError, Exception], tags: [RootEffect, TimeEffect].}

  Source Edit

# Exports

Http417, Http503, Http431, ==, contains, Http304, Http406, ==, $, clear, Http408, $, Http411, is3xx, Http418, Http206, HttpPost, HttpMethod, Http101, Http505, Http413, newHttpHeaders, HttpVer11, HttpTrace, Http200, []=, Http414, add, Http401, Http205, ==, Http407, Http500, Http404, Http416, Http302, HttpHeaders, Http300, Http428, Http410, is2xx, Http202, Http502, headerLimit, HttpHeaderValues, contains, newHttpHeaders, $, [], HttpHead, Http305, Http451, Http409, Http504, Http426, HttpConnect, hasKey, del, HttpPut, pairs, Http429, HttpVersion, HttpVer10, []=, Http421, HttpOptions, Http307, Http301, HttpPatch, is4xx, Http203, getOrDefault, Http100, Http501, HttpDelete, len, Http400, Http403, HttpGet, is5xx, Http415, toString, Http412, Http405, Http303, Http204, Http201, HttpCode, httpcore, Http422, []