Webhooks

  • Updated: Oct 17, 2016
  • Starting Guide
  • Webhooks
  • Serverless

Adding server-side business logic to your data

Realtime Webhooks are HTTP callbacks triggered by a Realtime event, such as a new user connection or a message published in a pub/sub channel.

In simple terms, a given endpoint in your server backend will be invoked whenever a predefined event occurs in the Realtime network.

The Realtime message causing the webhook trigger is passed in the request payload, so your logic can act based on the message content, including changing it before it's delivered to the channel subscribers!

This means you'll be able to execute business logic on your data as it streams across the Realtime network, like translating a chat message, saving messages to a database or send them as custom push notifications for disconnected users.

One of the best news is that since webhooks are triggered server-side you can start using them right away, even in apps already deployed to production, with no changes on the client code!

Some use cases

There's an infinitude of use cases but here's a short list to get you started:

  • Save messages in your own database
    Simply code your webhook to save the published messages in your own database (or a Cloud Storage table), for an easy implementation of a "history" feature;
  • Validate the message format
    Add to your webhook the logic to validate the message format and content before the message is sent to other subscribers. If the message isn't formatted according to your app requirements you can simply return a 400 status code in your webhook response and the Realtime server will not send the message to the channel subscribers;
  • Translate chat messages
    Use an external translation API (e.g. systran.io) in your webhook to automatically translate your chat messages. How? Invoke the translation API with the message text and add the translations as new properties to the original message. The subscribers can simply pick the right language property to show the message.
  • Send custom push notifications for disconnected users
    Sometimes your mobile chat users are not connected to your app when someone sends them a message. Implement a webhook in conjunction with the Realtime presence service, and you'll be able to know if the message recipients are disconnected and send them a customized push notification with the message content through the Realtime Custom Push API.

As you're probably aware by now the only limit to the possible use cases it's your own imagination!

The pricing?

The good news keep coming! A webhook invokation will only add one message to your Realtime Messaging monthly message count.

Activating the webhooks feature

Before you can use Realtime Webhooks you need to activate the feature in your Messaging subscription.

Goto your subscription list, expand the desired subscription and select the Edit option as shown below:

Scroll down and turn on the Webhooks option:

You're now a few steps away from creating your first Realtime webhook.

Configuring a webhook

After you have activated webhooks, go back to your subscription list and select the "Webhooks" option in the subscription panel:

Now select the Add new webhook button to create one. You'll get a form like this:

Here's the definition of each input field (you can get these definitions directly in the form by clicking the little ? marks next to each field):

  • Event:
    The Realtime event that will trigger the webhook. For the 'message published' event you'll need to enter the desired channel.

    The other events will automatically fill in the channel name.
  • Channel:
    The channel to bind your webhook.

    You can use a fully qualified channel name to bind exclusively to that channel, the * wildcard to bind to all channels and [main-channel]:* to bind to all [main-channel] sub-channels.

    The Realtime server will first look for the fully qualified channel name, like chat:123. If no webhook is defined for the channel then it will look for the sub-channel wildcard (for the previous example it would be chat:*). If no sub-channel wildcard is found it will look for the * wildcard (also known as the default webhook).

    The first webhook found in this waterfall sequence will be triggered with the event.
  • Active:
    Defines the status of the webhook. Inactive webhooks won't be triggered when the configured event occurs;
  • URL:
    If you are not using the Realtime Code Hosting service, you'll need to enter the public URL of the server endpoint that will be invoked by the webhook - nothing stops you from using an AWS Lambda function in conjunction with the AWS API Gateway, or any other hosting solution to get the URL's of your webhooks.

    The endpoints used must accept HTTP POST requests;
  • Execute:
    Here you'll define when your webhook URL should be invoked: before or after the message is delivered to subscribers.

    Using before will allow you to change or discard the message depending on the URL response. This is the option you should use if you want to translate a chat message, since your code needs to change the message before it's delivered to the channel subscribers.

    Using after will not allow you to change or discard the message since the Realtime server won't wait for the webhook response and will deliver the message to the channel subscribers while your webhook is executing. This is the option you should use if you simply need to save the message in a database or any other "fire-and-forget" type of operation that doesn't change the delivered message;
  • Rule:
    Defining a rule is optional and when defined only the messages that validate the rule condition(s) will trigger the webhook. This is useful when you want to filter the messages you send to your webhooks based on their content.

    Rules are very similar to SQL WHERE conditions and you can access the content of each message and use the values in your conditions.

    Later in this guide there's a special section about the Webhook Rules format.
  • API Key Header and API Key Value:
    You'll use this options when your URL requires an API key header and its value to validate the URL access rights. When they are set the Realtime server invoking the webhook will simply add this header (and its value) to the request.

To add or edit the webhook simply select the SAVE button on the end of the form. If the Active property was set your webhook is immediately ready to use and will be triggered as soon as the configured event occurs.

The webhook request payload

The Realtime server will invoke your webhook URL using a HTTP POST request with a predefined payload. Here's an example (definitions below):

{ 
	"headers": { 
     	"x-realtime-signature": "29b5ea52f46cd901e638ae38898227", 
   	},
  	"body": { 
  		"triggers": [ 
  			{ 
  				"id": "f23972ef15993974",
       			"timestamp": 1474299311424,
       			"channel": "myChannel",
       			"message": "{ \"sender\": \"John Smith\", \"timestamp\": 1474299311424, \"text\": \"Hi there! A great day for a chat!\" }",
       			"client":  { 
       				"ip": "96.94.78.248", 
       				"metadata": "console client" 
       			} 
       		} 
       	]  
    } 
}
        

Headers

The x-realtime-signature header is used for security.

It's a HMAC hex digest of the request body using with your Realtime subscription private key as the secret key. You should compute and validate this header to be sure the webhook is being invoked by a Realtime server. We will look into this with more detail later, on the Securing Webhooks section.

Body

The request body is where the fun begins!

It will be composed by an array of triggers, where each element corresponds to one message that has triggered the webhook.

Most times you'll have a single trigger in the array, but if the channel message frequency is high, the Realtime server will aggregate several triggers and send them together in a single batch. This will save bandwith and reduce the load on your server.

You should always iterate through the triggers array to process each one, otherwise, if you only process the first element of the array you'll risk missing messages.

The definition of each trigger element in the triggers array is explained below.

Trigger

As mentioned before a trigger object represents one message that caused the webhook invocation. These are the trigger properties:

  • id
    The unique trigger id.
    It's used by the Realtime servers to identity the triggers requests and their responses.
  • timestamp
    The timestamp of the event in UTC
  • channel
    The channel where the message triggering the webhook was published
  • message
    The message causing the webhook trigger.

    If the original message published to the channel is in JSON format you'll need to parse it to get its properties.
  • client
    The IP address and connection metadata of the client sending the message.
    This will allow you to get details about the message publisher and maybe add properties to the delivered message or make decisions based on the user profile.

Now that you know what to expect in a webhook request let's have a look at the response the Realtime server expects from your webhooks.

The webhook response payload

Since the request payload contains an array of triggers, the webhook response must be also an array of responses, each one corresponding to one processed trigger.

The response payload for the request example above should look like this:

[
  {
    "id": "f23972ef15993974",
    "statusCode": 200,
    "message": "{ \"sender\": \"John Smith\", \"timestamp\": 1474299311424, \"text\": \"Hi there! A great day for a chat!\" }"
  }
]
        

In this example we have an array with a single response since the request also had only a single trigger. You should have as many trigger responses in the array as the number of triggers you have received.

Here's the explanation for each response property:

  • id
    The trigger id as referred to in the request payload. This will be used by the Realtime server to identify the message causing the trigger.
  • statusCode
    The HTTP status code of the response. All trigger responses with a status code different than 200 (OK) will cause the discarding of the associated message and an exception to the publisher client (the message property will contain the exception description in this case).
  • message
    The message to be delivered to the channel subscribers.

    If the status code is 200 the message property must contain the message to be delivered to the channel subscribers (if the webhook is configured with the "before" message delivery option). This is how your webhook can change the message content. Please note that even if you don't want to change the original message you still need to send it back in the webhook response.

    If the status code is not 200 the message property must contain the exception description to be sent to the publisher.

Now that we know the request and response formats of the Realtime webhooks let's look at a simple example.

An example: validating a message format

Let's say you are implementing a chat app and you would like to allow third-parties, like a chat bot, to receive and send messages to your chat rooms. Since you don't control the "sender" code it could happen that some of these third-parties are not following your chat message format causing all sort of havoc in the client apps.

A simple webhook can solve your validation problem while allowing third-parties to simply use any Realtime Messaging SDK to send messages to your chats.

First let's assume that your message format is JSON and has the following properties:

{
	"sender": "John Smith",
	"timestamp": 1474298409178,
	"text": "Hi there! What a great day for a chat!"
}
        

You want to enforce that only messages with those properties are sent to the chat subscribers, so you need to write a validation function and expose it in a public server accepting HTTP POST - or simply deploy that function as a Realtime Code Hosting function (more about this later in this guide).

Writing the webhook function

In this example we'll be using the Express framework with Node.js to code our webhook endpoint but you can use any language and stack that you see fit.

For simplicity we'll not validate the x-realtime-signature header (we'll look into this under the "Securing Webhooks" section) but we strongly advise you to validate it in production deployments for security reasons.

First let's look at the validation function:

function validate(req, res) {
	// req: request payload
	// res: response

	// get the webhook triggers array from the request
	var triggers = req.body.triggers;
    
    // iterate through all the triggers
    for(var t = 0; t < triggers.length; t++) {
		var isValid = false;

		// check if the trigger message has all the required fields
		try {
			// our message format is JSON so we need to parse it
			var message = JSON.parse(triggers[t].message);

			// perform the validation
			if((message.sender && message.timestamp > 0 && message.text)) {
			    isValid = true;
			}
		}
		catch(e) {
		    // message is not valid
		}
		finally {
		    if(isValid) {
		    	// the message is valid
		    	// add the OK status code property to the trigger
		        triggers[t].statusCode = 200;
		    } else {
		    	// the message is invalid
		    	// add the the bad request status code to the trigger
		        triggers[t].statusCode = 400;

		        // define the exception description
		        triggers[t].message = "Message is not valid"
		    }
		}
	}

	// send the response
	return res.status(200).send(triggers);
}
        

It's pretty straightforward, right? Create a folder named RealtimeWebhook and simply save the code above in a file named app.js inside that folder.

The next step is to add the Express framework so we can expose our function as a HTTP POST.

Add the following code to your app.js file:

var express = require("express");
var bodyParser = require("body-parser");
var app = express();

// we'll use the body parser to parse our requests
app.use(bodyParser.json());

// add the validate route as a POST
app.post("/validate", function(req, res) {
	// call our validation function
	// and return the result
	return validate(req, res);
});

// listen to requests in port 3000
var server = app.listen(3000, function () {
    console.log("Listening on port %s...", server.address().port);
});
        

The following code shows the same logic performed in PHP:

<?php

$entityBody = file_get_contents('php://input');
$json_o = json_decode($entityBody, true);

for($i = 0; $i < count($json_o["triggers"]); $i++)
{
	$isValid = false;
	if (isset($json_o["triggers"][$i]["message"])) {
		$message = json_decode($json_o["triggers"][$i]["message"]);
		if (isset($message->sender) && isset($message->timestamp) && $message->timestamp > 0 && isset($message->text)) {
			$json_o["triggers"][$i]['statusCode'] = 200;
			$isValid = true;
		}
	}
	if($isValid == false){
		$json_o["triggers"][$i]['statusCode'] = 400;
		$json_o["triggers"][$i]['message'] = "Message is not valid";
	}
}

$json_string = json_encode($json_o, JSON_PRETTY_PRINT);
echo $json_string;

?>

Couldn't be simpler.

Now we need to publish our code in a public server so the Realtime servers can access it.

Deploying the webhook in a public server

Enter zeit.co now, a cloud service that allows you to run your node.js code in the cloud ... for free!

We are using the now plaftorm just as an example, you can use any hosting platform. In the end of this guide you'll find several links to guides for other hosting platforms, like AWS Lambda and our own Realtime Code Hosting.
You're spoiled for choice!

Now, back to our validation function. Before we can upload our code to the now platform we need to define the package.json file, declaring our API name and its dependencies.

Simply create a package.json file with the following content in your RealtimeWebhook folder:

{
  "name": "Realtime-Webhook-Example",
  "version": "1.0.0",
  "description": "A simple example to demo the Realtime Webhooks",
  "main": "app.js",
  "scripts": {
   	"start": "node app.js"
  },
  "dependencies": {
  	"express": "4.14.0",
  	"body-parser": "1.15.2"
  },
  "author": "Realtime",
  "license": "MIT"
}

        

Now we are ready to register at now and deploy our code. Enter the following commands in your terminal to install now and start the registration process (you need to have node.js installed):

$ npm install -g now
$ cd RealtimeWebhook
$ now
        

If you're using now for the first time it'll send you an email to validate your registration. Simply click the validation link in the email and you're set to go. If everything went right you'll get an output like this:

$ now
> Deploying ~/RealtimeWebhook
> Using Node.js 6.6.0 (default)
> Ready! https://realtimewebhookexample-owufhulksh.now.sh [1s]
> Upload [====================] 100% 0.0s
> Sync complete (2.2kB) [1s]
> Initializing…
> Building
> ▲ npm install
> Installing package express@4.14.0
> Installing package body-parser@1.15.2
> ▲ npm start
> Listening on port 3000...
> Deployment complete!
        

See the URL on the fourth line? That's your deployment public URL, ready to use.

You just need to add the endpoint /validate to access your webhook (as we have defined in our app.js file).

Using the deployment above the validate webhook URL would be the following:

https://realtimewebhookexample-owufhulksh.now.sh/validate

This is the URL we'll register in Realtime for our validation webhook. Feel free to use the URL above for the rest of this guide if you don't want to deploy your own.

Testing the webhook

To test your webhook deployment (or our deployment above) you just need a HTTP client, like cURL or Postman.

Paste the following cURL command in your terminal to send a POST to our webhook deployment with a valid message:

curl -X POST -H "Content-Type: application/json" -d '{
	"triggers": [
		{
			"id": "f3f31943c03a0f04",
			"timestamp": 1474370421868,
			"channel":"validate",
		    "message": "{\"sender\":\"John Smith\",\"timestamp\":1474370421868, \"text\":\"Hi there! What a great day for a chat!\"}",
		    "client":{
		        "ip":"46.189.228.182",
		        "metadata":"client data"
		    }
		}
	]
}
' "https://realtimewebhookexample-owufhulksh.now.sh/validate"

Since the message is valid you should have the following response from the webhook, a trigger with a 200 (OK) status code and the exact same message you have sent in the POST request:

[
  {
    "id": "f3f31943c03a0f04",
    "timestamp": 1474370421868,
    "channel": "validate",
    "message": "{\"sender\":\"John Smith\",\"timestamp\":1474370421868, \"text\":\"Hi there! What a great day for a chat!\"}",
    "client": {
      "ip": "46.189.228.182",
      "metadata": "client data"
    },
    "statusCode": 200
  }
]

Now let's try with an invalid message (without the sender property). Paste the following cURL command into your terminal window:

curl -X POST -H "Content-Type: application/json" -d '{
	"triggers": [
		{
			"id": "f3f31943c03a0f04",
			"timestamp": 1474370421868,
			"channel":"validate",
		    "message": "{\"timestamp\":1474370421868, \"text\":\"Hi there! What a great day for a chat!\"}",
		    "client":{
		        "ip":"46.189.228.182",
		        "metadata":"client data"
		    }
		}
	]
}
' "https://realtimewebhookexample-owufhulksh.now.sh/validate"

Since the message is considered invalid you should get a trigger response with a 400 status code and an exception description in the message property (instead of the message sent in the request). It should look like this:

[
  {
    "id": "f3f31943c03a0f04",
    "timestamp": 1474370421868,
    "channel": "validate",
    "message": "Message is not valid",
    "client": {
      "ip": "46.189.228.182",
      "metadata": "client data"
    },
    "statusCode": 400
  }
]

Now that you have deployed and tested your webhook URL let's configure it in your Realtime subscription.

Configuring the webhook

Login to the Realtime Account Management website and open your subscription list. Expand the Realtime subscription where you wish to configure the webhook and select the Webhooks option.

Click the Add new webhook button and enter the appropriate data into the form fields as explained below:

  • Event
    Select A message was published
  • Channel
    Enter the channel name to bind the webhook. Let's use chat:*
  • URL
    Select Use an external URL and enter your webhook URL.

    To use our deployment URL enter https://realtimewebhookexample-owufhulksh.now.sh/validate
  • Execute
    Keep the Before message is delivered to subscribers option selected

And that's it for now. Click the SAVE button to start validating the messages sent to all sub-channels of the chat channel.

Using the webhook

If you completed all the previous actions successfully, everytime a message is published in one of the sub-channels of the chat channel, your webhook URL will be invoked. If the message is valid it will be delivered to the channel subscribers, if not you'll see an exception and the channel subscribers will not receive the original message.

Let's try it.

Open two browser tabs with the Realtime Advanced Console.

For our tests one tab will have the role of the publisher (sending messages) and the other tabe the role of the subscribers (open as many subscriber tabs as you want).

Now enter your Realtime application key in both tabs and click the connect button.

In the "subscriber" tab subscribe channel chat:demo.

We're now ready to publish some messages.

In the "publisher" tab send the following message to channel chat:demo:

{ "sender": "John Smith", "timestamp": 1474298409178, "text": "Hi there! What a great day for a chat!" }

Since the message is valid you will receive it in the "subscriber" tab.

Now send the following message from the "publisher" tab to the chat:demo channel:

{ "timestamp": 1474298409178, "text": "Hi there! What a great day for a chat!" }

Since this message doesn't have a "sender" it's invalid and you should see an exception in the messages log. Check that in the "subscriber" tab you haven't receive this invalid message.

This means your validation webhook did its job at blocking the invalid message. Yay!

You're welcome to try publishing invalid messages using other Realtime SDKs. All invalid messages will be blocked since the webhook is agnostic to the client SDK used to publish the message.

Realtime Webhooks are simple and powerful!

The next sections of this guide will cover more advanced features like applying Rules, validating security and using other hosting platforms.

Using rules

In some situations you may need to execute your webhooks only for some messages, those that follow some pre-defined pattern. Webhook rules will allow you to do that.

In the Rule parameter of your webhook configuration you can enter a set of conditions to define the messages that should trigger the webhook.

The webhooks rule implements a sub-set of the SQL WHERE format with the following specification:

Operators

  • =
  • >=
  • >
  • <=
  • <>
  • <
  • like

Conditionals

  • and
  • or

Math

  • +
  • -
  • *
  • /

Note: The conditionals and like operator are case-sensitive, you must use them in lower-case. You must also use ' as the text delimeter.

Data scope

The message published will be available to your webhook rule in the message namespace.

If your message is in JSON format you can access your JSON object properties using the . notation, incluing nested objects.

Imagine you're implementing a weather app and your temperature messages are like this:

{
	"city": "London",
	"temperature": 50.8,
	"timestamp": 1472464189645,
	"geo": {
		"latitude": 51.528308,
		"longitude": -0.3817802
	}
}
        

To execute a webhook only for temperature messages regarding locations north of London (> 51.5074°N) with temperatures above 77°F, you would set the following webhook rule:

message.geo.latitude > 51.5074 and message.temperature > 77
        

Notice the message.geo.latitude > 51.5074 condition as an example of using nested objects.

Non-JSON messages

If your messages are not in JSON format you'll be more limited in your conditional options but you'll still be able to use the webhooks rules.

In this case the message namespace will contain your message as a string and you could apply the LIKE operator as follows to execute the webhook only for messages containing foo:

message LIKE '%foo%'
        

Notice the % wildcard used in the like operator.

Securing webhooks

In the previous "Webhook request payload" section we have mentioned the x-realtime-signature header, a SHA-256 HMAC hex digest of the request body using with your Realtime subscription private key as the secret key.

This digest is computed by the Realtime servers for all webhook requests and you should compute it also in your webhook function to check if they coincide.

If they coincide it's safe to assume the requests comes from a Realtime server as it was signed with your private key. If they don't match you cannot trust the request and you should return a 403 status code.

Computing the x-realtime-signature header

We'll be using node.js and the Crypto module to compute the x-realtime-signature header, but you can use any language and stack you want as long as you have a way of computing HMAC hex digests, like the hash_hmac PHP method.

Here's how to validate the security signature in Node.js:

var crypto = require("crypto");

...

// the request body, assuming the request is in req object
var body = req.body;

// your Realtime private key
var privateKey = "9gppgOwLR5rK";

// compute the HMAC digest
var signature = crypto.createHmac("SHA256", privateKey).update(JSON.stringify(body)).digest('hex');

// compare the signatures
if(signature === req.get('x-realtime-signature')) {
	// request is valid, continue
} else {
	// request is not valid, abort request
}
        

Checkout this GitHub repository with the complete Node.js code for the previous validation example using security.

The following code shows the same process performed in PHP:

<?php

$entityBody = file_get_contents('php://input');
$json_o = json_decode($entityBody, true);
$headers = getallheaders();
$signature = $headers["x-realtime-signature"];
$KEY = "9gppgOwLR5rK";
$hash = hash_hmac("sha256", utf8_encode($entityBody), utf8_encode($KEY), false); 


for($i = 0; $i < count($json_o["triggers"]); $i++)
{
	$isValid = false;
	if (strcmp($hash, $signature) == 0) {

		if (isset($json_o["triggers"][$i]["message"])) {
			$message = json_decode($json_o["triggers"][$i]["message"]);
			if (isset($message->sender) && isset($message->timestamp) && $message->timestamp > 0 && isset($message->text)) {
				$json_o["triggers"][$i]['statusCode'] = 200;
				$isValid = true;
			}
		}
		if($isValid == false){
			$json_o["triggers"][$i]['statusCode'] = 400;
			$json_o["triggers"][$i]['message'] = "Message is not valid";
		}
	}else{
		$json_o["triggers"][$i]['statusCode'] = 403;
		$json_o["triggers"][$i]['message'] = "Message signature is not valid";
	}
}

$json_string = json_encode($json_o["triggers"], JSON_PRETTY_PRINT);
echo $json_string;

?>	

Limitations

  • Maximum execution time per request
    Your webhook URLs must reply in less than 10 seconds, otherwise a timeout exception will be thrown to the client sending the message.
  • Maximum number of pending requests
    If in a given moment you have more than 100 pending requests from the same Realtime server, a Too many pending webhook requests exception will be thrown to the client sending the message. In this situation you should retry sending the message later.
  • Maximum message length
    Messages larger than 16Kb (20 Realtime message parts) will not trigger the webhook and will generate a Messages with more than 20 parts are not allowed as webhook messages exception.

Using other hosting platforms

In this guide we used the now platform for simplicity, however you can use any other cloud platform to host your webhooks functions or even your own server.

Here are two serverless examples:

Back to Delivery Modes - Next: Presence

If you find this interesting please share: