Putting your Hooks to work with the Outbox

The Outbox is an under utilized feature of BipIO for crafting and sending messages to any of your web hooks at a moments notice. You can find it under Dashboard > Apps. It’s great for invoking or testing web services from one place without having to write a client yourself and it just received a great new Redactor overhaul. Keeping in the spirit of BipIO, messages you send with the Outbox are ephemeral in nature, with no history being tracked on the server side unless you explicitly need it. Otherwise, your messages are bound to your browsers local storage only and when they’re gone, they’re gone.

I’m going to show you how to put it to work for HTML, Plain Text, JSON and Markdown payloads with a short video. And yes, this post was built and published to Tumblr with the Outbox itself!

Incase you don’t know, ‘Web Hook’ Bip’s are a special kind of graph from which you can receive messages from a web browser or app, and perform some workflow based on a payload. Hooks are the workhorse of many of BipIO’s own website logic (so meta!) because they can serve or process data for any application. To come to grips with web hooks, I’m going to fall back to using the ‘my first bip’ hook which you may have already created through the website tour when you first signed up.



Didn’t quite follow? These are the basic rules -

  • HTML mode will send as HTML. ‘HTML Mode’ is when all formatting options are available.
  • Code mode will send un-formatted text. Toggle HTML/Code mode with the <> button
  • To translate to JSON, hit the ‘Parse JSON’ check box while in code mode
  • Unless you’re ‘parsing JSON’, the Outbox exports will be for ‘title’ and ‘body’
You may notice I took a couple of seconds to create a Markdown to HTML Channel to handle Markdown parsing, and dropped it onto ‘My First Bip’. Markdown is just plain text, and needs some special treatment. Otherwise HTML, Text and JSON parsing are already handled by the Outbox itself.

Have fun, with great power comes great responsibility! follow me on Twitter and fork me on GitHub to see whats happening in the here and now.

[edit] Exciting news! JSON, Markdown, HTML and plaintext Outbox modes are compatible with the Poetica Chrome plugin so you can collaborate socially on the full gamut of payload type with joyful, instant abandon!

Fun With HTTP Pods

A little while ago I rolled out the HTTP Pod for BipIO which provides a basic webhook channel (called ‘request’) that can drop onto any hub to do some work. Think fetching a file, or posting form data or making some ad-hoc API call as part of a pipeline. It’s the kind of irresponsibly powerful tool you might want to leverage for your app to fan out web requests or temporarily make available a hidden or authenticated resource to (un)trusted 3rd parties.

A more interesting feature of Bips (the graphs that do work) and in particular the HTTP flavor is that they support the concept of rendering dynamic content from any available channel. Bips themselves have the characteristics of ephemerality, authentication, name, transport encapuslation and graph processing. They’re public facing endpoints that anyone can call, with a graph behind them that potentially can process requests. Without a renderer, HTTP Bips will just sit there accepting data and all going well respond with a happy ‘200 OK’ message. With renderers, we add the ability to respond in a custom way beyond the generic ‘200 OK’, per endpoint. This makes them a very powerful tool for backing parts of a web application or adding ‘app-like’ characteristics to simple URL’s in something as simple as email.

That said, I want to demonstrate a few of the funky things I was able to get going in no time with HTTP Bips and some of the new renderers the HTTP Channel provides. Using both HTTP Bips and Channels is a good starting point because they share the same protocol, but keep in mind that any Pod/Channel which has a renderer can also serve content or have its protocol encapsulated via HTTP Bips. The demo’s were active workflows during the beta pre-launch period for user onboarding and segmentation which turned out really useful. I hope you find so, too!

Simple Proxying

When the launch key emails were being sent out, I wanted to get a rough guage of how many people were opening the message for their invite code, grouped per day. There’s a range of different techniques for surfacing this kind of simple metric but I went for simply tracking when an image was loaded - specifically the bipio logo embedded in the email’s HTML.

The endpoint is https://beta.bip.io/bip/http/logo.png and it looks and renders like just any other image. Because HTTP Bips don’t know anything about serving images by themselves it was the perfect time to try out the ‘proxy’ renderer supported by the new ‘Request’ Channel in the HTTP Pod. The logic simply being, ‘When the image is served, record a count grouped by day’.

To get started, the ‘Channels’ section of the  bipio dashboard is where the channel itself will get created. Clicking on the HTTP Request icon will start the channel creation process. Channels will generally just sit there doing nothing until added to a Bip’s graph, or enabled as a renderer. Lets have this ‘http.request’ channel GET the logo image file :

To create this with the API, just POST a channel structure like so :

POST /rest/channel
{
  "name": "BipIO logo",
  "action": "http.request",
  "config": {
    "url": "https://bip.io/static/img/logo.png"
  }
}

Building the web hook which will serve this channel’s content publicly is then pretty simple. Under ‘Bips’, click ‘Create Web Hook’ and make sure :

- It has a name that looks like an image file,

- Has authentication disabled

- A renderer set as a the ‘HTTP Proxy’ renderer of the logo image channel which was created earlier.  You can find renderer setup in its own tab during Bip configuration :

So just enable the ‘HTTP Proxy’ renderer for your new channel.

Via the API, it’s something like :

POST /rest/bip
{
    "name": "logo.png",
    "type": "http",
    "config": {
      "auth": "none",
      "renderer": { // renderer object
        "channel_id": "01ded262-4150-4041-bcea-6727bd46960e",
        "renderer": "proxy"
      }
    },
    "hub": {
      "source": {
        "edges": []
      }
    }
}

And that’s it! The named endpoint which is created

(https://beta.bip.io/bip/http/logo.png) will proxy any requests it receives via the renderer and serve up the file like magic. To handle hitcounts on the endpoint and build a basic report, its just a matter of filling out the hub with two extra channels for time formatting (time.format) and count (flow.counter). I’ve split it out here so you can wrap your head around the structure but it can replace the empty ‘hub’ in the previous POST, also :

"hub": {
    "76ba8c52-7161-4e80-aa06-146b45da75b9": {
        "transforms": {
            "c9a7c8ad-bd43-447a-9eea-fac54a9e1ebc": {
                "_note" : flow.counter channel",
                "increment_by": "1",
                "group_by": ""
            }
        },
        "edges": [
            "c9a7c8ad-bd43-447a-9eea-fac54a9e1ebc"
        ]
    },
    "source": {
        "transforms": {
            "76ba8c52-7161-4e80-aa06-146b45da75b9": {
                "_note" : "time.format channel",
                "format": "MMDDYYYY",
                "time": "1402247179"
            }
        },
        "edges": [
            "76ba8c52-7161-4e80-aa06-146b45da75b9"
        ]
    }
}

The ‘flow.counter’ channel has a renderer itself which dumps out all the data it has collected. I could either call that renderer directly to get at the data, or encapsulate it in a bip like the previous example to run a report etc. Pretty neat!

Request Redirection

To redirect a user directly to a target resource rather than proxy it, its just one small change to the renderer structure above - just set ‘renderer’ from ‘proxy’ to ‘redirect’ in the config section.

PATCH /rest/bip/{bip-id}

{
    "config": {
        "auth": "none",
        "renderer": {
            "channel_id": "01ded262-4150-4041-bcea-6727bd46960e",
            "renderer": "redirect"
        }
    }
}

A couple of cases for using the HTTP Pods redirect renderer might be as a link shortener, or to segment users in MailChimp by who clicks through via the generated link!

SSL Encapsulation

A common problem for people running content sites or apps with SSL is that any non-encrypted content being rendered into a browser raises a bunch of security warnings as it undermines the integrity of the content. On the flipside there is significant infrastructure overhead in downloading every piece of content onto your server or cdn for serving over SSL, simply to make browsers happy. By using the ‘http.request’ proxy renderer its possible to encapsulate insecure content in SSL instead.

Here’s a simple SSL bridge which you can use via the BipIO website (which forces SSL connections). Creating a bridge is very similar to the ‘http.request’ channel defined earlier which served a logo, however a ‘url’ config parameter is not defined - its injected by the Bip, instead :

POST /rest/channel

{
    "name": "SSL Bridge",
    "action": "http.request",
    "config": {
    }
}
POST /rest/bip

{
    "name": "anonymous_bridge",
    "type": "http",
    "config": {
        "auth": "token",
        "renderer": {
            "channel_id": "91ded262-4150-4041-acea-6727bd46960e",
            "renderer": "proxy"
        }
    },
    "hub": {
        "source": {
            "edges": [
            ]
        }
    }
}

To test it out just call the endpoint like so, authenticating with your username/API key :

https://{username}.bip.io/bip/http/anonymous_bridge?url=http://example.org

Voila!

A quick note on security …

Given the ability to proxy, redirect and encapsulate web requests with abandon, its generally a bad idea to accept and process any URL a web client throws at you. Be sure to always authenticate clients using the authentication config attributes in HTTP bips! Additionally, if you’re running the server on your own infrastructure, you may notice ‘blacklist’ errors returning to your connecting clients. This is because by default all local network interfaces are blacklisted by the HTTP Pod out of the box. To whitelist local network interfaces, add their IP or Hostname to the HTTP Pods ‘whitelist’ section in your server config.

As a quick follow up to the earlier API post, syncing SoundCloud favorites to Dropbox, with a few extras such as posting a Dropbox mirror link to everyones favorite social networks. Ideal of course for getting around Soundcloud download limits.

Here’s how its done via the website. Some of the transforms are pre-filled by our intelligent (learning) data mapper on the back end, hence some edges don’t need any configuration. Drag’n’Drop. Shazam.

Integration 101: SoundCloud favorites straight to Dropbox

Integration 101 is an ongoing series where I’ll describe some of the useful workflows that can be created with the Bipio API in a few easy steps. A basic understanding of RESTful API’s and SaaS is assumed, and if you have experience with graphs and graph traversals that’s helpful too but not necessary. If there’s anything you’d like to see, let me know. Otherwise it’s onward and upward with some crazy experiments!

I’m a lazy (busy?) SoundCloud music consumer, why click favorite > download > have a cup of tea > find in downloads > drag to Dropbox folder when I can just click ‘like’ instead and let a robot do the legwork?

Here’s a little assembly for doing just that with Bipio. It takes 2 Channels (soundcloud.get_favorites and dropbox.save_file) and 1 trigger bip to fire a message into the hub. Less clicks, more consumption.

SoundCloud

POST /rest/channel

{
  "action" : "soundcloud.get_favorites",
  "name" : "Get All The Favorites!"
}

Response

201 Created
{
  "id": "23bb1de5-c950-c448-fbbf-000056c1eed6",
  ... more attributes here
}


Dropbox

POST /rest/channel

{
  "action" : "dropbox.save_file",
  "name" : "Save to SoundCloud Folder",
  "config" : {
     "base_dir" : "/soundcloud"
  }
}

Response

201 Created
{
  "id": "5d2de02d-a70c-4ffd-ac15-9a97f6cb3d0f",
  ... more attributes here 
}


So armed with those two id’s, creating the bip is easy, ‘get_favorites’ is an emitter, meaning it generates its own content periodically (in this case, mp3 files and metadata). We can capture these files with a ‘trigger bip’, and process the files with an edge pointing to the Dropbox Channel.

{  
  "config": {
    "channel_id": "23bb1de5-c950-c448-fbbf-000056c1eed6"
  },  
  "end_life": {
    "time": 0,
    "imp": 0
  },
  "hub": {
    "source": {
      "annotation": "SoundCloud > Dropbox",
      "edges": [
        "5d2de02d-a70c-4ffd-ac15-9a97f6cb3d0f"
      ]
    }
  },
  "name": "SoundCloud to Dropbox",
  "type": "trigger"
}

Synced, just like that. There’s no need to transform any content unless its really necessary. Files that appear to a bip from an external service are simply carried across the hub incase channels need them. So, for this basic sync, we know that a file is appearing from SoundCloud, and whatever that file is, gets saved to DropBox.

Extra Credit
ok ok, so that assembly just ends up in big amorphous glob of music in the /soundcloud folder, so of course we should probably template the destination path a little better. Lets re-work the hub in that bip and add a transform to file it by Genre and Artist, which are two of the exports the soundcloud.get_favorites action already provides.

"hub": {
    "source": {
      "annotation": "SoundCloud > Dropbox",
      "edges": [
        "5d2de02d-a70c-4ffd-ac15-9a97f6cb3d0f"
      ],
      "transforms" : {
      	"5d2de02d-a70c-4ffd-ac15-9a97f6cb3d0f" : {
      		"/soundcloud/[%genre%]/[%artist%]" : "base_dir"
      	}
      }
    }
  }
Now isn’t that better?

Candy Strips and Graph Traversals

Bipio lets you create cheap dynamic endpoints quickly and have them perform some complex task, so when our LaunchRock page failed after a recent account migration (they were super helpful in debugging, no love lost) it seemed like a perfect opportunity to build a similar email capture widget using the toolset already baked into our API. Time to Eat Our Own Dogfood, as it were, as a simple exercise I could fit into a saturday morning between breakfast coffee and brunch coffee. Great!

The requirements for the widget were, at their core, very simple - for every user that arrived at the home page, I wanted to create a temporary, single use endpoint which took an email address, passed it to a data store and sent a thankyou message. Maybe later I can render this syndication into excel files containing batches of 100 leads for onboarding (because we can), but for now, just keep it simple. Don’t want to be late for brunch. Enter the Candy Strip. It’s a little dynamic call to action widget that you can find on our home page, and it took all of 30 minutes to set up. Fun Fact : Although the website itself is a blocking architecture PHP/Zend stack, creating these endpoints takes ~10ms across our DMZ to the API, a barely noticeable hit to application performance and we get a bunch of functionality that can be easily extended at no cost to application complexity.

So, we can create these endpoints right out of the box with HTTP Bips, with all the normal characteristics of a Bip including temporary lifespan, authentication scheme, dynamic naming and a predictable data export. To have that bip send an email and store some data, I just need to create a simple Hub (graph) with two channels (vertices) which are to send a transactional Email (email._transaction), and list email addresses to a List Syndication (syndication.list).

To create the syndication, just POST /rest/channel a little packet pointing at the Syndication Pods ‘list’ action. The list action is perfect because it gives me the option to render everything we’ve received to a CSV, TSV or Excel file down the track. Channels are re-usable across Hubs, so this Channel could now act as a datasource for a different hub sitting outside this email capture workflow if I don’t want to add more actions to the pipeline.


{
   "config": {
     "write_mode": "append" 
  },
   "action": "syndication.list",
   "name": "Signup Reservation List"
}
Done! The data store gets created, and every line item we get in this list is appended to the store

Just black holing content into a syndication isn’t very useful, so I wanted to send an email to the early adopter thanking them for their interest. I used the email template we already had for LaunchRock so didn’t have to don my Marketing Hat before lunchtime. Bonus, only 10 minutes in so far!


{
   "action": "email._transaction",
   "config": {
     "template": "beta_registered",
     "from": "Bipio Signups ",
     "subject": "A quick update about your beta invite" 
  }
   "name": "Email Capture Receipt"
}
email._transaction is a special Email Pod action we have for sending pre-baked transactional messages, you won’t find them in the API just yet, sorry!

We now have two Channels for processing the capture, so in the website controller code, I’ll just make a call out to the API when the page renders and assemble these Channels onto a Hub with a POST /rest/bip. The name of the bip matches the session id in our application, they’re single use or naturally expire after a day. Together between the size of our capture syndication, and the number of created http bips, I can also figure out a rough conversion rate. The ‘_repr’ attribute in the POST /rest/bip response payload tells the widget which endpoint URI to send the email address to in our eventual AJAX request. _repr means the string representation of the object, all REST resources in Bipio have this decorator, harking back to the systems Tornado roots.



If I hit the website, I see a new HTTP bip in the ‘beta.bip.io’ account dashboard whose name matches my session id, so its working. The graph for my temporary bip renders like so :

image


Now to expose the widget in the view and slap it some lipstick courtesy of cameo & ycc2106 @ colourlovers :



image


image


And we’re done :)

AES-256 encryption w/PHP + Node.js

Having trouble making AES encrypted payloads transportable to Node.js? Us too. Since decoupling the website from the API to implement on a more suitable stack, having mcrypt/openssl PHP AES implementations playing nicely with node crypto has been a point of pain, mostly because end-to-end openssl_encrypt problems often go uresolved for this undocumented method.

Here’s how its done on the PHP side, for encrypting and decrypting based on a configured key version, which is embedded the payload. PHP’s openssl_encrypt is lenient with the key length and format, which can be a subtle bug if you’re not looking for an errant character breaking on crypto.deCipherIv().

Most solutions involve shelling an openssl command. Not necessary.

.. and from Node, its quite easy. Just ensure you have the same key in the versioned keychain, and use v0.8+

Mastering API Glue so you don’t have to

Like many projects, bipio emerged from the fog of creativity and necessity to solve a problem and scratch an itch.  For us, it was mostly about solving some tedious problems around implementing (and refactoring) complex message delivery stacks, and also more mundanely trying to manage and control the volume, quality and re-usability of messages that flowed through the system, including what appeared in our inboxes and from whom.  

After many years of feeling the pain and inefficiency of shoe-horning TCP transports, API’s or other services into playing nicely together, the eventual takeaway was very clear :

API Glue Abstraction is hard, solving the abstraction problem efficiently and intuitively in a way that maximizes creativity and responsiveness is very valuable

.. so here we are.  We’re building an API to programatically  create dynamic endpoints  that can accept or generate content (these are called ‘bips’) and push that content, transformed or filtered, out to other services via re-usable components (‘channels’) with a strong API developer focus.

We’ve started with simple email -> smtp/rss pipelining, sign up and stay tuned :)

About me

is a thin filtering, distribution and moderation layer for email aliases, social networks and other popular web services

Ask me anything