As a quick follow up to the earlier API post, syncing SoundCloud favorites to Dropbox, with a few extras such as posting a Dropbox mirror link to everyones favorite social networks. Ideal of course for getting around Soundcloud download limits.

Here’s how its done via the website. Some of the transforms are pre-filled by our intelligent (learning) data mapper on the back end, hence some edges don’t need any configuration. Drag’n’Drop. Shazam.

Integration 101: SoundCloud favorites straight to Dropbox

Integration 101 is an ongoing series where I’ll describe some of the useful workflows that can be created with the Bipio API in a few easy steps. A basic understanding of RESTful API’s and SaaS is assumed, and if you have experience with graphs and graph traversals that’s helpful too but not necessary. If there’s anything you’d like to see, let me know. Otherwise it’s onward and upward with some crazy experiments!

I’m a lazy (busy?) SoundCloud music consumer, why click favorite > download > have a cup of tea > find in downloads > drag to Dropbox folder when I can just click ‘like’ instead and let a robot do the legwork?

Here’s a little assembly for doing just that with Bipio. It takes 2 Channels (soundcloud.get_favorites and dropbox.save_file) and 1 trigger bip to fire a message into the hub. Less clicks, more consumption.

SoundCloud

POST /rest/channel

{
  "action" : "soundcloud.get_favorites",
  "name" : "Get All The Favorites!"
}

Response

201 Created
{
  "id": "23bb1de5-c950-c448-fbbf-000056c1eed6",
  ... more attributes here
}


Dropbox

POST /rest/channel

{
  "action" : "dropbox.save_file",
  "name" : "Save to SoundCloud Folder",
  "config" : {
     "base_dir" : "/soundcloud"
  }
}

Response

201 Created
{
  "id": "5d2de02d-a70c-4ffd-ac15-9a97f6cb3d0f",
  ... more attributes here 
}


So armed with those two id’s, creating the bip is easy, ‘get_favorites’ is an emitter, meaning it generates its own content periodically (in this case, mp3 files and metadata). We can capture these files with a ‘trigger bip’, and process the files with an edge pointing to the Dropbox Channel.

{  
  "config": {
    "channel_id": "23bb1de5-c950-c448-fbbf-000056c1eed6"
  },  
  "end_life": {
    "time": 0,
    "imp": 0
  },
  "hub": {
    "source": {
      "annotation": "SoundCloud > Dropbox",
      "edges": [
        "5d2de02d-a70c-4ffd-ac15-9a97f6cb3d0f"
      ]
    }
  },
  "name": "SoundCloud to Dropbox",
  "type": "trigger"
}

Synced, just like that. There’s no need to transform any content unless its really necessary. Files that appear to a bip from an external service are simply carried across the hub incase channels need them. So, for this basic sync, we know that a file is appearing from SoundCloud, and whatever that file is, gets saved to DropBox.

Extra Credit
ok ok, so that assembly just ends up in big amorphous glob of music in the /soundcloud folder, so of course we should probably template the destination path a little better. Lets re-work the hub in that bip and add a transform to file it by Genre and Artist, which are two of the exports the soundcloud.get_favorites action already provides.

"hub": {
    "source": {
      "annotation": "SoundCloud > Dropbox",
      "edges": [
        "5d2de02d-a70c-4ffd-ac15-9a97f6cb3d0f"
      ],
      "transforms" : {
      	"5d2de02d-a70c-4ffd-ac15-9a97f6cb3d0f" : {
      		"/soundcloud/[%genre%]/[%artist%]" : "base_dir"
      	}
      }
    }
  }
Now isn’t that better?

Candy Strips and Graph Traversals

Bipio lets you create cheap dynamic endpoints quickly and have them perform some complex task, so when our LaunchRock page failed after a recent account migration (they were super helpful in debugging, no love lost) it seemed like a perfect opportunity to build a similar email capture widget using the toolset already baked into our API. Time to Eat Our Own Dogfood, as it were, as a simple exercise I could fit into a saturday morning between breakfast coffee and brunch coffee. Great!

The requirements for the widget were, at their core, very simple - for every user that arrived at the home page, I wanted to create a temporary, single use endpoint which took an email address, passed it to a data store and sent a thankyou message. Maybe later I can render this syndication into excel files containing batches of 100 leads for onboarding (because we can), but for now, just keep it simple. Don’t want to be late for brunch. Enter the Candy Strip. It’s a little dynamic call to action widget that you can find on our home page, and it took all of 30 minutes to set up. Fun Fact : Although the website itself is a blocking architecture PHP/Zend stack, creating these endpoints takes ~10ms across our DMZ to the API, a barely noticeable hit to application performance and we get a bunch of functionality that can be easily extended at no cost to application complexity.

So, we can create these endpoints right out of the box with HTTP Bips, with all the normal characteristics of a Bip including temporary lifespan, authentication scheme, dynamic naming and a predictable data export. To have that bip send an email and store some data, I just need to create a simple Hub (graph) with two channels (vertices) which are to send a transactional Email (email._transaction), and list email addresses to a List Syndication (syndication.list).

To create the syndication, just POST /rest/channel a little packet pointing at the Syndication Pods ‘list’ action. The list action is perfect because it gives me the option to render everything we’ve received to a CSV, TSV or Excel file down the track. Channels are re-usable across Hubs, so this Channel could now act as a datasource for a different hub sitting outside this email capture workflow if I don’t want to add more actions to the pipeline.


{
   "config": {
     "write_mode": "append" 
  },
   "action": "syndication.list",
   "name": "Signup Reservation List"
}
Done! The data store gets created, and every line item we get in this list is appended to the store

Just black holing content into a syndication isn’t very useful, so I wanted to send an email to the early adopter thanking them for their interest. I used the email template we already had for LaunchRock so didn’t have to don my Marketing Hat before lunchtime. Bonus, only 10 minutes in so far!


{
   "action": "email._transaction",
   "config": {
     "template": "beta_registered",
     "from": "Bipio Signups ",
     "subject": "A quick update about your beta invite" 
  }
   "name": "Email Capture Receipt"
}
email._transaction is a special Email Pod action we have for sending pre-baked transactional messages, you won’t find them in the API just yet, sorry!

We now have two Channels for processing the capture, so in the website controller code, I’ll just make a call out to the API when the page renders and assemble these Channels onto a Hub with a POST /rest/bip. The name of the bip matches the session id in our application, they’re single use or naturally expire after a day. Together between the size of our capture syndication, and the number of created http bips, I can also figure out a rough conversion rate. The ‘_repr’ attribute in the POST /rest/bip response payload tells the widget which endpoint URI to send the email address to in our eventual AJAX request. _repr means the string representation of the object, all REST resources in Bipio have this decorator, harking back to the systems Tornado roots.



If I hit the website, I see a new HTTP bip in the ‘beta.bip.io’ account dashboard whose name matches my session id, so its working. The graph for my temporary bip renders like so :

image


Now to expose the widget in the view and slap it some lipstick courtesy of cameo & ycc2106 @ colourlovers :



image


image


And we’re done :)

AES-256 encryption w/PHP + Node.js

Having trouble making AES encrypted payloads transportable to Node.js? Us too. Since decoupling the website from the API to implement on a more suitable stack, having mcrypt/openssl PHP AES implementations playing nicely with node crypto has been a point of pain, mostly because end-to-end openssl_encrypt problems often go uresolved for this undocumented method.

Here’s how its done on the PHP side, for encrypting and decrypting based on a configured key version, which is embedded the payload. PHP’s openssl_encrypt is lenient with the key length and format, which can be a subtle bug if you’re not looking for an errant character breaking on crypto.deCipherIv().

Most solutions involve shelling an openssl command. Not necessary.

.. and from Node, its quite easy. Just ensure you have the same key in the versioned keychain, and use v0.8+

Mastering API Glue so you don’t have to

Like many projects, bipio emerged from the fog of creativity and necessity to solve a problem and scratch an itch.  For us, it was mostly about solving some tedious problems around implementing (and refactoring) complex message delivery stacks, and also more mundanely trying to manage and control the volume, quality and re-usability of messages that flowed through the system, including what appeared in our inboxes and from whom.  

After many years of feeling the pain and inefficiency of shoe-horning TCP transports, API’s or other services into playing nicely together, the eventual takeaway was very clear :

API Glue Abstraction is hard, solving the abstraction problem efficiently and intuitively in a way that maximizes creativity and responsiveness is very valuable

.. so here we are.  We’re building an API to programatically  create dynamic endpoints  that can accept or generate content (these are called ‘bips’) and push that content, transformed or filtered, out to other services via re-usable components (‘channels’) with a strong API developer focus.

We’ve started with simple email -> smtp/rss pipelining, sign up and stay tuned :)

About me

is a thin filtering, distribution and moderation layer for email aliases, social networks and other popular web services

Ask me anything