• JSON is for machines…

    Here’s an involved (and well written) tutorial on RAML, which goes over concepts like schemas, resources, resource types, includes, Parameter Transformers(?), and traits. As much as RAML simplifies the API definitions, it is still no easy spec to learn. Seriously, good luck figuring out how to singularize properly if you don’t know RAML already. Given REST(…)

  • Swagger vs RAML vs API Blueprint

    I’m doing some evaluation of the different API spec formats. My quick thought is that the syntax difference between the 3 are irrelevant. On API Blueprint’s web site, we see “JSON is a machine format. Leave it to machines. Who will edit your documentation next year? Who will proofread it? With API Blueprint everybody can!(…)

  • Problems with Mocking Frameworks

    Just want to jot down some quick thoughts on my issues with mocking frameworks. In some cases, creating and configuring a mock is more verbose and harder to understand than defining a mocked implementation. Counter argument is that using mocks everywhere is more consistent and allows for invocation verification when needed. But at what cost(…)

  • Guava’s Futures.successfulAsList

    I really appreciate Google’s Guava library. I think it is a great example of a set of common and minimal APIs with a lot of reusability. Today I want to rehash some of my recent experiences with the Futures API. Assume you have the following list of Future instances: Then you call successfulAsList(futuresList).get(100, MILLISECONDS), what(…)

  • Father’s Day

    Today I am reminded how luck I am to have my two wonderful daughters. Also, what great responsibility it is to be a father. In life, They can have anything they want through perseverance and hard work. But only one person can give them a father they can be proud of. I hope I don’t(…)

  • Deep Dives into Ning’s Async Http Client

    Few thoughts during a deep dive of NettyResponseFuture * Calls to .get() destroys the async nature of the API. This is obvious in retrospect. .get() will block on the calling thread. * asyncHandler.onThrowable() is called not just for connection timeouts. It is also called if the onCompleted() method throws an Exception. * I’m not sure(…)

  • Inactivity

    It is hard to believe it has been a year since I last posted. I suspect it has more to do with enthusiasm about my work at The Rubicon Project than anything else. However, some recent technical arguments with my coworkers, as well as thoughts about reorganizing my online accounts, have me back to blogging.(…)

JSON is for machines…

Here’s an involved (and well written) tutorial on RAML, which goes over concepts like schemas, resources, resource types, includes, Parameter Transformers(?), and traits. As much as RAML simplifies the API definitions, it is still no easy spec to learn. Seriously, good luck figuring out how to singularize properly if you don’t know RAML already.

Given REST API specification is such a specialized niche, I really don’t this is something a typical technical writer will learn. IMO it falls into the realm of UML – great if you have the right tools for it.

Swagger vs RAML vs API Blueprint

I’m doing some evaluation of the different API spec formats. My quick thought is that the syntax difference between the 3 are irrelevant.

On API Blueprint’s web site, we see

“JSON is a machine format. Leave it to machines.

Who will edit your documentation next year? Who will proofread it? With API Blueprint everybody can! Your designer and copywriter too.”

The truth is, your designer and copywriter will not be able to maintain a document like this. Even a programmer will bulk at learning API Blueprint’s syntax.

My first use case for a API documentation tool is design – I would like a command line tool that can generate API documentation from spec, so that I can quickly mock up some API documentation during the design phase of a project and review with relevant parties.

Swagger – No great command line tools. You could, however, write simple Java classes and use Maven to generate the Swagger spec, then use Swagger UI to render it. This is not practical for non-Java developers. There is a Swagger editor, which may be a better alternative.

API Blueprint – Has a command line tool called “apiary”. This is a Ruby gem that you need to install and I’ve already ran into a bug when generating documentation.

RAML – Has a nodejs command line client to generate documentation. I still have to spend a bit more time evaluating this.

When it comes to being able to prototype a design quickly and review it with API users, none of the 3 tools are great at what they do.

My second use case is generating documentation from code –

Swagger – Very good Java support. But this may be useless for developers using other languages.

API Blueprint – More to come…

RAML – More to come…

 

 

 

 

 

 

Problems with Mocking Frameworks

Just want to jot down some quick thoughts on my issues with mocking frameworks.

In some cases, creating and configuring a mock is more verbose and harder to understand than defining a mocked implementation. Counter argument is that using mocks everywhere is more consistent and allows for invocation verification when needed. But at what cost of readability? At what point is it easier to write mocked implementations? On a related topic, I find Python’s Mock package to be more concise and readable than Java’s Mockito.

Mocking frameworks do not support object Factories easily. This is a situation where you call a method that creates objects, and you need to mock the behavior of those objects. For example, AsyncHttpClient’s execute(request, handler) method returns a ListenableFuture. In this case, you not only need to mock the execute() method to return a mocked Future, you also need to mock the returned Future to emulate desired behavior (specifically, handling callbacks). This can be done, but requires a lot of code.

Guava’s Futures.successfulAsList

I really appreciate Google’s Guava library. I think it is a great example of a set of common and minimal APIs with a lot of reusability. Today I want to rehash some of my recent experiences with the Futures API.

Assume you have the following list of Future instances:

future1 ... takes 5ms to complete
future2 ... takes 19ms to complete
future3 ... takes 510ms to complete
future4 ... takes 2ms to complete

futuresList = newArrayList(future1, future2, future3, future4);

Then you call successfulAsList(futuresList).get(100, MILLISECONDS), what do you expect to happen? Would you expect to get the successful results from futures 1, 2, and 4, but not future3? No. It doesn’t work that way. As it turns out, if any of the futures times out, then the entire combined future will time out. It is a bit non-intuitive, and certainly could be better documented.

Let’s take it a bit further, what if you wrap future3 in a wrapper that looks like this so that it doesn’t throw a TimeoutException?

  @Override
  public V get(long timeout, TimeUnit unit) { 
    try {
      delegate.get(timeout, unit);
    } catch (Throwable t) {
      // ignore
    }
  }

That won’t work either. Calling successfulAsList(futuresList).get(time, unit) will still throw a TimeoutException. It turns out the Futures....asList callbacks to collect successful responses, and the callbacks are not invoked by timeouts to the get() method.

One way to get with a timeout when using Futures....asList() is to use a scheduled task to ensure each underlying future reaches the ‘done’ state before your time limit. This technique is described in this stackoverflow post. If you are doing this a lot, you’ll be scheduling a lot of tasks, which can be expensive.

It seems like a lot of people are waiting on a good getSuccessfulListWithinDeadline(). I am too.

Father’s Day

Today I am reminded how luck I am to have my two wonderful daughters. Also, what great responsibility it is to be a father. In life, They can have anything they want through perseverance and hard work. But only one person can give them a father they can be proud of. I hope I don’t mess that up.

Deep Dives into Ning’s Async Http Client

Few thoughts during a deep dive of NettyResponseFuture

* Calls to .get() destroys the async nature of the API. This is obvious in retrospect. .get() will block on the calling thread.
* asyncHandler.onThrowable() is called not just for connection timeouts. It is also called if the onCompleted() method throws an Exception.
* I’m not sure if there is a good technique for expressing “Make a HTTP call to X and give me the full result before Y ms or a TimeoutException after Y ms” and be nearly 100% asynchronous
* Assuming the full HTTP response is available, calls to .get() does not block (again, cpt obvious here)

Misc
* For the Netty provider, it might be worth setting the REUSE_ADDRESS setting if you have lot of sockets in TIME_WAIT state and you know the risks.
* The ApacheResponseFuture.get() call will never throw a TimeoutException. Instead, it will be wrapped in an ExecutionException. WTF?
* The ApacheResponseFuture.get(timeout, unit) should not be called with a duration less than responseTimeoutInMS, it seems to do some weird things.
* Netty uses ThreadRenamingRunnable to rename threads in your thread pool. Threads used by Netty will always be named “New I/O boss” or “New I/O worker”
* Netty allows you to configure its HashedWheelTimer. The default tolerance is 100ms.

Inactivity

It is hard to believe it has been a year since I last posted. I suspect it has more to do with enthusiasm about my work at The Rubicon Project than anything else. However, some recent technical arguments with my coworkers, as well as thoughts about reorganizing my online accounts, have me back to blogging.

Regarding the technical arguments, I had a very educational argument with my coworkers about designing APIs, REST vs RPC, and encapsulation. It forced me to read more about the semantics of REST vs RPC and makes me question why REST is popular. Some things about REST that felt icky to me are: Underlying data model can be represented as different resources – for example, an advertising campaign may be represented as a large object graph, or you might represent parts of it, such as the delivery status, as its own resource. Some resources within the campaign may overlap. For different level of granularity, you may need to represent parts of the same object differently. In a way, we’re using purely nouns to describe operations. I’m not sure if that’s completely intuitive. I’ll have to spend more thought digesting it. But some parts of my mind is seeing REST semantics as taking OOP and decoupling data and methods. There’s something there that bothers me and I can’t quite articulate it. There are also some arguments on how to use Immutable object builders in Java effectively. I’m not sure what’s the best practice for it, but I’m almost certain you should never write Builders by hand.

Lastly I’m beginning to rethink my migration away from blogger. As much as I enjoy wordpress and the community around it, the task of maintaining security and keeping WP and PHP up-to-date is becoming frustrating to me, to the point where I am certain this frustration has a lot to do with my inactivity. But I’m back and I’ll try to force myself to write more. It’s been too long, and it feels great to write again.

Scaling Your Documentation

As good programmers, we all want to create reusable components. We carefully learn techniques like OOP, FP, and patterns to make our code clean and reusable. We use many tools to help create good documentation. The defining character of a good programmer, I believe, is writing good documentation. Recently, however, I started to observe that documenting the code is no longer sufficient, a great programmer must start writing good service documentation.

10 years ago, I’d hear things like “We’re a Java shop/We’re a Ruby shop”. The implication is that in that company, they’d use a specific programming language as the standard, and share reusable libraries within that language. Today, if your company is a __whatever__ shop, you’re doomed. Today, polyglot programmers are the norm and it is now impossible to staff a house full of good programmers if you’re constrained to a single language. One consequence of this evolution is that teams are no longer sharing “libraries” with each other – now we share functionality via web services. This service-centric approach to organization growth also means programmers need to start think bigger when it comes to documentation . Writing great libraries with great documentation is no longer enough – most people in your organization are not going to using your library directly. More and more people are depending on you creating easy to use service APIs and your service level documentation. If you have great code documentation, but your services are not documented, who is going to use your code? What value is it going to provide?

I think it is ironic that as computers have gotten more powerful and programming languages have gotten easier, the programmer’s job has only gotten harder. It makes sense in retrospect – more productivity and more power equates more responsibility. The defining character of a good programmer, maybe, is not so much about writing good documentation, but knowing that documentation what allows your programs to make people’s job easier. And that makes you scalable as a person and a leader in your organization, not just an ordinary programmer.

More Stupid Emacs Tricks

This time, from the “Getting latest version of tramp working OS X” edition.

This is the version of Emacs I use on my mac, from http://emacsformacosx.com/. It comes with tramp 2.2.6. To upgrade to the latest version of tramp, download the tarball from the GNU FTP site. Extract, and use the following configure command:

./configure \
   --with-emacs=/Applications/Emacs.app/Contents/MacOS/Emacs \
   --with-lispdir=/Applications/Emacs.app/Contents/Resources/lisp \
   --with-infodir=/Applications/Emacs.app/Contents/Resources/info

I also add in a sprinkle of this in my .emacs file to customize my ssh control path (do this at your own risk, says the FAQ).

   (require 'tramp)
   (setq tramp-ssh-controlmaster-options
         "-o ControlPath=~/.ssh/connections/%%r_%%h_%%p -o ControlMaster=auto"))

Farewell XML

Changes in my emacs config.

This

;; Executes 'Executes 'js-beautify -i -s4  2>/dev/null' on the current region.
(defun jc-beautify ()
  "Executes 'Executes 'js-beautify -i -s4  2>/dev/null' on the current region."
  (interactive)
  (shell-command-on-region (region-beginning) (region-end)
                           "/usr/local/share/python/js-beautify -i -s4 2>/dev/null"))

Replaced

;; Executes 'tidy -asxml -xml -indent -wrap 2000 2>/dev/null' on the current region.
(defun jc-tidy ()
  "Executes 'tidy -asxml -xml -indent -wrap 2000 2>/dev/null' on the current region."
  (interactive)
  (shell-command-on-region (region-beginning) (region-end)
                           "tidy -asxml -xml -indent -wrap 2000 2>/dev/null"))

It’s end of an era.