Thursday, May 7, 2015

Offline data access and synchronization in a mobile application with Couchbase Lite

This is the translation of my article that first appeared on Habrahabr.

Couchbase and Couchbase Lite

When developing data-driven mobile applications, we often encounter the customer's wish to fully access all of the app’s features, including changing the data, when the device is offline. The changes made to the data also have to sync with the backend when the device goes online. The backend is also concurrently accessed by desktop and web frontend applications which may also modify the data.

Public cloud synchronization is not always viable, especially when security concerns are in place, and customer wishes to keep all of their data on their private servers. In this article I’ll describe my experience of solving this task by using Couchbase database on the server and Couchbase Lite database in the mobile application with two-way replication between them.

The Couchbase database is a document-oriented distributed NoSQL database that ensures high performance by writing data into memory first, eventually persisting it onto the disk. Couchbase enables strong consistency between the nodes in a clustered environment by making them independent and equal, while each document being bound to a certain node. Couchbase is queried with indexed views that implement the MapReduce pattern.

Couchbase Lite is a lightweight version of Couchbase that is intended for desktop and mobile applications and is able to replicate with Couchbase server. Couchbase Lite is implemented on iOS, Android, Java and .NET platforms, so it can be used not only in mobile but also in desktop applications. It’s worth mentioning that the iOS version of Couchbase Lite currently has several advantages against other platforms. For instance, there is full-text search, and also automatic mapping of documents to Objective C and Swift objects.

For synchronization of Couchbase and Couchbase Lite, a CouchDB-almost-compatible replication protocol is used. Almost — because the authors don’t guarantee complete compatibility due to obscure documentation of CouchDB protocol which they even had to partly reverse. This protocol is implemented in Sync Gateway — a REST-based replication service. All clients that wish to sync data should connect to the central database using this service.

Couchbase Server installation and setup

Couchbase installation

The installation process of Couchbase differs between platforms and is described in the documentation. Let’s assume the database is already installed on localhost. The default location of admin console is http://localhost:8091/. Let’s go there and create a bucket named "demo" which we’ll use for storing our documents. To do that, open Data Buckets tab and click Create New Data Bucket button.

Enter the bucket name "demo" and limit it’s memory quota to 100 MB.

When all is done, a new bucket named demo will appear in the list of buckets, with a green circle beside it that indicates its normal activity.

Click the Documents button and observe that the newly created bucket is empty.

Sync Gateway setup

Sync Gateway installation and setup are described in the documentation. Here I’ll provide a sync-gateway-config.json file that will allow you to run the sample application that we’ll develop in this article:

     "log": ["CRUD+", "REST+", "Changes+", "Attach+"],
               "users": {
                    "GUEST": {"disabled": false, "admin_channels": ["*"]}
               "sync":`function(doc) {channel(doc.channels);}`

After running the Sync Gateway with this config file, you should observe the following log showing that the demo bucket is ready for acting as our central data synchronization storage:

23:27:02.411961 Enabling logging: [CRUD+ REST+ Changes+ Attach+]
23:27:02.412547 ==== Couchbase Sync Gateway/1.0.3(81;fa9a6e7) ====
23:27:02.412559 Configured Go to use all 8 CPUs; setenv GOMAXPROCS to override this
23:27:02.412604 Opening db /demo as bucket "demo", pool "default", server 
23:27:02.413160 Opening Couchbase database demo on 
23:27:02.601456 Reset guest user to config
23:27:02.601467 Starting admin server on
23:27:02.603461 Changes+: Notifying that "demo" changed (keys="{_sync:user:}") count=2
23:27:02.604248 Starting server on :4984 ...

Refresh the page with the bucket document list, and you should see some internal Sync Gateway documents there which IDs start with _sync:

Console application

The code of the console application is available on GitHub together with the mobile application. It is mainly intended for demonstrating and testing the interaction of mobile and desktop databases and is comprised of a simple Java application that connects to an embedded Couchbase Lite database, which is also implemented in Java. The application is able to create a local document with an image attachment and a timestamp_added attribute. It also initiates replication of local changes to Couchbase Server.

Mobile application

The mobile application will show thumbnails of pictures that were added in a console application, persisted to the local database and replicated to the mobile database via server database. The process of creating this mobile application is described here in full. I chose the iOS platform for the mobile application as it is has better support for the Couchbase Lite API. The language used here is Swift.

Creating a project and adding dependencies

First let’s create a simple Single View Application:

To attach the couchbase-lite-ios library to the project, let's use the CocoaPods dependency manager. The CocoaPods installation is described in its documentation. Let’s initialize CocoaPods in the project directory:

pod init

Add the couchbase-lite-ios dependency to Podfile:

target 'CouchbaseSyncDemo' do
     pod 'couchbase-lite-ios', '~> 1.0'

Install the specified library into the project:

pod install

Now you should reopen the project as a workspace (CouchbaseSyncDemo.xcworkspace). Now add a bridging header file so you can use the CocoaPods-installed Objective C libraries in your Swift classes. To do that, add to the project the following header file, naming it CouchbaseSyncDemo-Bridging-Header.h:

#ifndef CouchbaseSyncDemo_CouchbaseSyncDemo_Bridging_Header_h
#define CouchbaseSyncDemo_CouchbaseSyncDemo_Bridging_Header_h
#import "CouchbaseLite/CouchbaseLite.h"

Specify this file in your Build Settings:

UI stub

Inherit the automatically generated ViewController class from the UICollectionViewController:

class ViewController: UICollectionViewController {

Open Main.storyboard and switch the default ViewController to a Collection View Controller, dragging it from the Object Library and redirecting the Storyboard Entry Point to it. In the Custom Class section of the Identity Inspector specify the generated ViewController. Also select the Collection View Cell and in its Attribute Inspector specify "cell" as its Reuse Identifier. The result is shown on the following screenshot:

Initializing and starting the replication

Create a class CouchbaseService that will incapsulate the database-related functionality and implement it as a singleton:

private let CouchbaseServiceInstance = CouchbaseService()

class CouchbaseService {

     class var instance: CouchbaseService {
          return CouchbaseServiceInstance


Now open the demo database in the constructor of this class and start continuous pull replication. If the application is run inside the emulator, and Couchbase Server is running on the same machine, then we can use localhost as the address for replication. The continuous flag ensures that the replication runs continuously via long polling mechanism. You should also create the "images" view for extracting the list of all images:

private let pull: CBLReplication
private let database: CBLDatabase

private init() {

     // create or open the database
     database = CBLManager.sharedInstance().databaseNamed("demo", error: nil)

     // initiate pull replication
     let syncGatewayUrl = NSURL(string: "http://localhost:4984/demo/")
     pull = database.createPullReplication(syncGatewayUrl)
     pull.continuous = true;

     // create a view of all documents in the database
     database.viewNamed("images").setMapBlock({(doc: [NSObject : AnyObject]!, emit: CBLMapEmitBlock!) -> Void in
          emit(doc["timestamp_added"], nil)
     }, version: "1")

Couchbase Lite views

Couchbase view is an indexed and automatically refreshed result of execution of a pair of functions — map and (optionally) reduce — on all of the documents in the bucket. Here the view is specified only by its map function that for each document returns its creation timestamp as the key. The key in views is also used to sort the view’s results, so the images will always be sorted by the time they were added. The version parameter specifies the view’s version and has to be changed each time we change the view’s code. The change in the version is a signal for Couchbase to rebuild the view using the new version of the code.

Views in Couchbase can be queried. A specific type of queries is a live query, which results in an automatically updated array of documents. Thanks to Objective C and Swift’s KVO feature, we can observe this array’s changes and update the interface of our application when new data arrives via replication.

As a matter of fact, this way of tracking the changes may only signal the fact that the query results changed, but not the concrete added or deleted records. Such information would allow us to minimize the updates to interface — and gladly, Couchbase Lite provides it via the kCBLDatabaseChangeNotification event. This event signals of all new revisions that are added to the database. But in this example I decided to use the more simple live query mechanism.

Dealing with the data

Let’s add to CouchbaseService class a function for executing live query to our images view:

func getImagesLiveQuery() -> CBLLiveQuery {
     return database.viewNamed("images").createQuery().asLiveQuery()

The iOS implementation of Couchbase Lite stands out from other platforms by its automatic bi-directional mapping of documents to object models. This mapping leverages dynamic features of Objective C. A Swift implementation of this mapping is as follows:

class ImageModel: CBLModel {

     @NSManaged var timestamp_added: NSString

     var imageInternal: UIImage?

     var image: UIImage? {
          if (imageInternal == nil) {
               imageInternal = UIImage(data: self.attachmentNamed("image").content)
          return imageInternal


The timestamp_added attribute is dynamically linked to the corresponding field in the document, and the attachmentNamed: function allows us to receive binary data attached to the document. To convert the document to its object model, we can use the ImageModel constructor.

Binding interface and data

All that’s left to do is to subscribe ViewController to live query refresh and process this refresh by reloading the collection view. The images attribute keeps the list of documents converted to object models.

private var images: [ImageModel] = []

private var query: CBLLiveQuery?

override func viewDidAppear(animated: Bool) {
     query = CouchbaseService.instance.getImagesLiveQuery()
     query!.addObserver(self, forKeyPath: "rows", options: nil, context: nil)

override func observeValueForKeyPath(keyPath: String, ofObject object: AnyObject, change: [NSObject : AnyObject], context: UnsafeMutablePointer) {
     if object as? NSObject == query {
          var rows = query!.rows
          while let row = rows.nextRow() {
               images.append(ImageModel(forDocument: row.document))

The UICollectionViewDataSource protocol methods are quite typical and self-explanatory, except that we use the "cell" reuse identifier that we specified for the collection view cell in the storyboard earlier.

override func collectionView(collectionView: UICollectionView, numberOfItemsInSection section: Int) -> Int {
     return images.count

override func collectionView(collectionView: UICollectionView, cellForItemAtIndexPath indexPath: NSIndexPath) -> UICollectionViewCell {
     let cell = collectionView.dequeueReusableCellWithReuseIdentifier("cell", forIndexPath: indexPath) as! UICollectionViewCell
     cell.backgroundView = UIImageView(image:images[indexPath.item].image)
     return cell

Running the application

Now let’s see what we’ve achieved. Let’s run the console application. By issuing the start command inside the console application, we’re starting the replication; with the attach command we can create several documents with images.


CBL started
апр 15, 2015 11:41:14 PM com.github.oxo42.stateless4j.StateMachine publicFire
push event: PUSH replication event. Source: com.couchbase.lite.replicator.Replication@144c1e50 Transition: INITIAL -> RUNNING Total changes: 0 Completed changes: 0
апр 15, 2015 11:41:15 PM com.github.oxo42.stateless4j.StateMachine publicFire
push event: PUSH replication event. Source: com.couchbase.lite.replicator.Replication@144c1e50 Transition: RUNNING -> IDLE Total changes: 0 Completed changes: 0


Saved image with id = 8e357b3c-1c7f-4432-b91d-321dc1c9fd9d
push event: PUSH replication event. Source: com.couchbase.lite.replicator.Replication@144c1e50 Total changes: 1 Completed changes: 0
push event: PUSH replication event. Source: com.couchbase.lite.replicator.Replication@144c1e50 Total changes: 1 Completed changes: 1

The data is replicated to the mobile device and gets displayed right away:


In this article I demonstrated synchronization of data between server side and mobile application by means of Couchbase and Couchbase Lite. This allows us to create a mobile application that can be fully functional while the device is offline. In my future articles I’ll explore document revisions and replication protocol of Couchbase Lite more closely and test it for bad connectivity, sudden backgrounding of application and other perils of mobile app development.


Sources of sample applications on GitHub
Couchbase Lite
Sync Gateway
MapReduce computation model
Couchbase installation
Installing and running Sync Gateway
Installing CocoaPods

Friday, February 6, 2015

Solving issues with installing the "Real World OCaml" prerequisite libraries under Mac OS X

    I’ve run into some troubles installing the «Real World Ocaml» prerequisites under Mac OS X Yosemite. The installation process is described on the books’ wiki here. I've found little help by googling the error messages, which possibly means they could be very specific to my installation. So I've decided to share my experience in case someone runs into similar issues.
    At one point, the instruction advises you to install some libraries that are used throughout the book, by issuing the following command:
opam install \
   async yojson core_extended core_bench \
   cohttp async_graphics cryptokit menhir

    At first I’ve failed to install the cohttp and async_graphics packages. Possibly that was because I already had objective-caml installed, and my process of installation deviated a bit from the one prescribed by the instruction.
    The cohttp package depends on ctypes package, which was failing with the following error:
# fatal error: 'ffi.h' file not found
# #include <ffi.h>

    To solve it, just install libffi and add it to LDFLAGS environment variable for the time of build:
brew install libffi
export LDFLAGS=-L/usr/local/opt/libffi/lib
opam install cohttp

    If during the installation of async_graphics module you receive an error:
# Error: Unbound module Graphics

    That probably means you’ve installed objective-caml without the Graphics module, which may be verified by the listing:
ls /usr/local/opt/objective-caml/lib/ocaml/graphics*

    If the listing is empty, then you should reinstall objective-caml with graphics:
brew uninstall objective-caml
brew install objective-caml --with-x11

Now the process of installing the async_graphics package should work just fine:
opam install async_graphics

Installing Ocsigen web framework under Mac OS X and CentOS and creating a simple web application

        I recently wanted to play around with OCaml and create a web application. It appears that the Ocsigen framework is the only (or the most popular) choice for building web applications with OCaml, so here’s how to install it on Mac OS X and CentOS 6 and create a simple web application.

Installing on Mac OS X

        This process was tested on Yosemite. First, install brew, if you haven't got it already:
ruby -e "$(curl -fsSL"

        Now install ocaml, opam package manager and all the prerequisites for the Eliom (web application framework) and Macaque (database framework). Macaque is not really needed to run a simple example, but you're going to need it soon enough if you're about to develop a database-backed web application.
brew install ocaml opam libev gdbm pcre openssl pkg-config sqlite3

        Create a symbolic link for pkgconfig to the sqlite package. Use your current installed version of sqlite instead. Not a very clean solution, as it will break if the sqlite package is updated via brew. If you know how to reference the latest sqlite version, please let me know.
ln -s /usr/local/Cellar/sqlite/ /usr/local/lib/pkgconfig/sqlite3.pc

        Now initialize the opam package manager. It will create the ~/.opam directory, where it keeps all of its data, including installed packages.
opam init

        Now edit the ~/.profile file and add this line:
eval `opam config env`

        Restart the terminal shell to pick up the environment variables, and then check that the scaffolding tool is available:

Installing on CentOS 6

        First add the OCaml repository to yum:
cd /etc/yum.repos.d/

        Now install OCaml, opam and all the prerequisites:
yum install ocaml opam ocaml-camlp4 ocaml-camlp4-devel ocaml-ocamldoc openssl-devel pcre-devel sqlite-devel

        Initialize the opam repository and install the eliom web framework and macaque database framework:
opam init
opam install eliom macaque

        If for some reason you encounter an error during installation:
# ocamlfind: Package `camlp4' not found

        Then try to reinstall the ocamlfind package and run the installation again:
opam reinstall ocamlfind
opam install eliom macaque

Creating and running your first Ocsigen web application

        Create a barebones application using the generator:
eliom-distillery -name mysite -template basic -target-directory mysite

        Run it:
cd mysite
make test.byte

        Open http://localhost:8080/ in your browser. You should see the «Welcome from Eliom’s distillery!» greeting message.