RESTful Interfaces in the Small and in the Large
June, 17th 2015
This post outlines an architecture strategy to design-in-the-small standardized components - just following the Don't Repeat Yourself (DRY) principle - and to scale them for whole application landscapes.
Introduction
Problem
At the beginning, we often implement a few interfaces. The number of interfaces grows then over time significantly. Interface implementations are often representing a significant percentage of the code base and hence costs.
What architectures strategies for implementing interfaces provide economies of scale and facilitate the transition from the programming-in-the-small to the programming-in-the-large?
Solution
The solution, which I present here, is based on the following principles
Adopt early on design patterns to standardize the design;
Use the programming-language's interface to decouple the pattern elements; and
Generalize pattern elements to re-use them across the board.
This is nothing new per se. But, the interesting twist is how to start programming- in-the-small and evolve the design to one that fits for programming-in-the-large. This requirement rules out strategies requiring a large initial investment.
As a case study, I show how to apply these principles to the implementation of RESTful interfaces in the programming language Clojure. Clojure offers offers an interface notion, called protocols, and a macro mechanism that facilitates the generalization and re-use. As a design pattern, we use the Model-View-Controller (MVC) design pattern []. For a brief summary of concepts, see Appendix.
The Solution
Here, I use a simplified example for explanatory purposes. In particular, the functional scope is limited and data storage, security features and so on are omitted. The evolutionary design is outlined by a series of mini-tutorials for this example, which are available on here.
An Interface Component provides in our case just
A set of RESTful interfaces with their resources and methods. For instance, a resource currency
accountproviding aGETmethod that accepts an account identifier and returns , if the caller is authorized, the account data in JSON.A mocked data set that the RESTful interfaces are delivering
Implementing a simple interface
Hello World Example
In Clojure, the library ring and compojure enable to define interfaces and end-points in terms of routes and their run-time context.
As an example, I use these libraries to implement an interface that delivers a static HTML page. This is our "hello world" example to get started.
(defn home
"Function returning a static html page"
[req]
(render (io/resource "index.html") req))
(defroutes app-routes
"Defined routes of the application"
(GET "/" [] home) ;; home-page
(route/resources "/" ) ;; resources required
(route/not-found "Not found")) ;; exception case
(def handler
"Handler chain invoked on a request by the server"
(-> app-routes
wrap-params))
The handler is passed to the Jetty web-server at start-up time (the details are found in the project.clj configuration file under :ring). The example is available here.
A First Interface getAccounts
First, we define example data. For this purpose, we simply declare an immutable map of accounts, where each account is described by a map too. Here, just a single account with three credit and debit bookings.
(def account-test-data
{:101 {:account-id 101 :currency "CHF"
:bookings[ {:amount 100 :value-date "2014-01-02" :ccy "CHF" :xref "A1"}
{:amount -100 :value-date "2014-01-02" :ccy "CHF" :xref "A2"}
{:amount 100 :value-date "2014-01-02" :ccy "CHF" :xref "A3"}]})For updating the data, we assign the defined map to an atom , i.e. a named reference holding our immutable value and providing functions to swap one value with another one.
(def accounts-data (atom account-test-data))Second, we need a handler for our interface method. The library Liberator implements the REST protocol as per RFC. Here, we use this library and add the necessary logic for our particular handler.
(defresource accounts-r [id]
:available-media-types ["application/json"]
:allowed-methods [:get]
:exists? (fn [_]
(if-let [acc-j (get-in @account [(keyword id)])]
[true {:account acc-j}]
[false {:message (format "Account %s not found" id)}]))
:handle-ok (fn [{acc-j :account}](j/write-str acc-j)))This resource definition includes a check whether the requested account exists or not and a handler that returns the retrieved account.
Now, we can run on the command line the server and use curl to invoke the service (as soon as the hello world we-page is showing up). Alternatively, we can also type the URL into the browser.
$ curl -i http://localhost:8000/accounts/101 returns the account object
$ curl -i http://localhost:8000/accounts/1012 returns a 404
At this point, we have a working RESTful interface. But, the developer of the next interface has to understand the Liberator library. Furthermore, the developer might make different design choices and hence each interface implementation might be somewhat different. Could we not standardize this resource definition and provide it as a library?
A Standardized Re-usable Controller
For building a reusable resource definition, I have to understand first how the MVC design pattern relates to the above code. Second, where to use protocols to decouple the controller.
The MVC pattern resource model, view and controller correspond in above code to the atom with the functions on this atom, the reader/writer for JSON , and the defresource form that defines the controller.
A GET controller has always the same structure, so that I rewrite the above example as the following Clojure macro (assuming that this is a good enough one).
(defmacro my-r-macro
[r-name r-id & {:keys [lookup-fn]}]
`(defresource ~r-name [~r-id]
:allowed-methods [:get ]
:known-content-type? #(check-content-type % ["application/json"])
:exists?
(fn [_#]
(if-let [res# (~lookup-fn (keyword ~r-id))]
true
[false {:message (format "~r-name %s not found" ~r-id)}]))
:handle-ok (fn [_#] (j/write-str (~lookup-fn (keyword ~r-id))))))
With this macro, the actual controller definition is expressed in terms of the macro, that is:
(my-r-macro accounts-r id
:lookup-fn lookup-accounts)This simple macro, requires that we declare the functions used to access the model. This results in unnecessary boiler-plate code. More importantly, the requirements on these functions are not clearly specified.
To address these shortcomings, I introduce a protocol defining the signature to access the resource models.
(defprotocol` RM-Accessor
(get-item [this id])
(duplicate-item? [this id item])
(valid-item? [this id item])
(add-item [this id item]))This protocol supports also the update of a resource using a POST. Before adding an item, you can determine whether the item is a duplicate and whether it is valid.
I rewrite next the controller macro using the protocol just defined. In above macro example, we replace the ~lookup-fn by the protocol function get-item. The latter requires as a first argument a type instance implementing the protocol, wich I will call r-m - an input argument of the macro.
(defmacro defresource-macro [r-name r-id r-m & {:keys [malformed-fn]}]
`(defresource ~r-name [~r-id]
:available-media-types ["application/json"]
:allowed-methods [:get :post]
:known-content-type?
#(check-content-type % ["application/json"])
:malformed?
(fn [{{method# :request-method} :request :as ctx#}]
(if (= :post method#)
(try
(if-let [body# (body-as-string ctx#)]
(let [record# (j/read-str body# :key-fn keyword)]
(if (~malformed-fn record#)
[true {:message "booking incomplete."}]
(if (not (valid-item? ~r-m ~r-id record#))
[true {:message "invalid entry" }]
[false {:record record#}])))
[true {:message "No body"}])
(catch Exception e#
[true {:message (format "exception: %s" (.getMessage e#))}]))))
:exists?
(fn [_#]
(if-let [res# (get-item ~r-m ~r-id)]
[true res#]
[false {:message (format "%s %s not found..." (get-name ~r-m) ~r-id)}]))
:can-post-to-missing?
(fn [_#] [false {:message (format "%s %s not found!" (get-name ~r-m) ~r-id)}])
:post!
(fn [{record# :record}]
(if (duplicate-item? ~r-m ~r-id record#)
[false {:message (format "account booking %s already exists" ~r-id)}]
[true {:result (add-item ~r-m ~r-id record#)}]))
:location #(build-entry-url (get % :request) )
:handle-ok (fn [_#] (j/write-str (get-item ~r-m ~r-id)))))
You might want to compare the rewrite of the functions that define :exists? and :handle-ok. The additional functions are necessary for the POST method main methods are:
:malformed?checks whether the request itself is valid. I my implementation, I distinguish between the protocol- and model-specific validation. The former and the latter are carried out by the functions~malformed-fnand the protocol functionvalidate-item.:post!updates the accountsatomunless the booking item is a duplicate.
Before I can create an interface using the re-written macro, I have to implement the protocol. Clojure provides deftype and defrecord for implementing types in the implementation and application domain, respectively.
The kind of resource model is defined by a type. As an example, I define DependentResource, which is suitable to handle bookings given an account identifier.
(deftype DependentResourceModel [rm-name data schema validation-fn dr-key xref-key]
RM-Accessor
(get-item [this id]
(get-in @data [(keyword id) dr-key]))
(duplicate-item? [this id item]
(if (empty? (xref-key item))
false
(let [journal (get-in @data [(keyword id) dr-key])
item-keys (vec
(clojure.set/difference
(set (keys item)) (set (list xref-key))))]
(clojure.set/subset?
(clojure.set/project (set (list item)) item-keys )
(clojure.set/project (set journal) item-keys)))))
(valid-item? [this id item]
(let [obj (get-in @data [(keyword id)])
v1 (valid-schema? schema item)
v2 (validation-fn obj item)]
(and (first v1)(first v2))))
(add-item [this id item]
(let [j-item (conj {:time-stamp (l/format-local-time
(l/local-now) :date-time)} item)
items (get-in @data [(keyword id) dr-key])]
(swap! data assoc-in [(keyword id) dr-key]
(vec (conj items j-item ))))))This type implements
get-itemto retrieve the account bookings (as per parameterdr-key) given the account identifierid; andadd-itemto add an account bookingitemto a particular accountid.
The function valid-item? checks valid-schema? and validation-fn. The latter checks in our example that the currency of the booking item corresponds to the account currency. The former checks that a booking item complies with the data shape defined for it.
For defining data shapes and validation them, I use the schema library.
(s/defschema account-booking-s
{:value-date (s/pred value-date? 'value-date?)
:amount s/Num
:ccy currency-code-s
(s/optional-key :xref)s/Str
(s/optional-key :ts) s/Str})Noteworthy is the definition of a predicate value-date? that checks that the value is a string in the format "YYYY-MM-DD" that represents a valid date. Furthermore, I apply first validations only where necessary - here the POSTrequest body. Second, the schemata are not imposing type constraints on the implementations. This makes it possible to aim for generic implementations, but constrain their inputs as necessary from a business perspective.
At this point, we can define our RESTful interface by simply declaring a suitable resource type instance that takes as arguments
- an atom
accounts-dataand schema `accounts-booking-s - a
booking-validation-fn?; and - the keywords used by the type-specific protocol implementation.
(def accounts-bookings-r-m
(DependentResourceModel. "account-bookings"
accounts-data account-booking-s
booking-validation-fn?
:bookings :xref))
(defresource-macro accounts-bookings-r id
accounts-bookings-r-m
:malformed-fn malformed-request?) This definition is sufficient to implement RESTful interfaces for dependent resources using JSON as media type. Under the hood of this macro, we could replace the REST library or extend the macro to support multiple media types as well as to provide end-points using additional transport protools, such SOAP/HTTPs.
A solution architect designing an application need only to follow "Don't Repeat Yourself" (DRY) to obtain the here outlined solution. For institutionalizing such a solution across a domain or enterprise, one could adopt a community approach or implement integration libraries centrally.
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.