Serialization using Google protobuf

Protocol buffers (that's what protobuf means :)) are a means to serialize structured data. Now you may be wondering what this 'serialize' means.

To be brief : Serialization is a process of packing the data structures or objects into sequence of bits so that they can be stored in a file or a buffer and transmitted across the network and then can be retrieved on the same or other system environment. Examples include XML, JSON, ASN.1, YAML, ProtoBuf etc.

Advantages :
1. Persistent object creation
2. RPC enabling
3. Are simple
4. Save space on disk
5. Very fast
6. Platform and language independent

ProtoBuf is available freely from Google (http://code.google.com/p/protobuf/). Procedure of installation and compilation is present there.

Everything that needs to be serialized is put in a protocol buffer message (a .proto file). This file tells the compiler how the data is to be serialized.

For each field there are setter and getter functions fieldname() is a getter and set_fieldname() is a setter. Remember to use these functions as the exact way it is told here.
The tutorials for the different languages can be found here

I present a small simple example to use the protobuf library and there is much more than that one can do with the protobuf :)

To make and run the example (on linux) follow the steps :
1. Download the tar.gz package from here
2. Login as root
3. Untar it in your home directory (say user1)
4. Go inside the protobuf directory and run ./configure
5. Add the path /usr/local/lib to the environment variable LD_LIBRARY_PATH
6. Then run :
make
make install
7. You are now ready to use the library.
8. To use protobuf you will need following files (I am explaining for c++ only):
.cc file
.proto file
Makefile
Rest of the files are generated on the fly.

Here is an example of a Vehicle parts which contains three fields , two mandatory and one optional.
-Part ID (mandatory)
-Part Name (mandatory)
-Vendor (optional)

We write this in .proto file

//partbook.proto
package tutorial;

message Part {
  required string name = 1;
  required int32 partid = 2;        // Unique ID number for this part
  optional string vendor = 3;

}

message PartDetails {
  repeated Part part = 1;
}

Then we write a c++ program to use the library and .proto to add serialized data to the data file.

/*add_part.cc*/
#include <iostream>
#include <fstream>
#include <string>
#include "partbook.pb.h"
using namespace std;

/* This function fills in a Part message based on user input.*/
void PromptForPart(tutorial::Part* part) {
  cout << "Enter part ID number: ";
  int partid;
  cin >> partid;
  part->set_partid(partid);
  cin.ignore(256, '\n');

  cout << "Enter part name: ";
  getline(cin, *part->mutable_name());

  cout << "Enter vendor name (blank for none): ";
  string vendor;
  getline(cin, vendor);
  if (!vendor.empty()) {
    part->set_vendor(vendor);
  }

}

/* Main function:  Reads the entire part list from a file,
 * adds one part based on user input, then writes it back out to the same
 *   file.
 */
int main(int argc, char* argv[]) {
  /* Verify that the version of the library that we linked against is
   * compatible with the version of the headers we compiled against.
   */
  GOOGLE_PROTOBUF_VERIFY_VERSION;

  if (argc != 2) {
    cerr << "Usage:  " << argv[0] << " PART_BOOK_FILE" << endl;
    return -1;
  }

  tutorial::PartDetails part_book;

  {
    // Read the existing part book.
    fstream input(argv[1], ios::in | ios::binary);
    if (!input) {
      cout << argv[1] << ": File not found.  Creating a new file." << endl;
    } else if (!part_book.ParseFromIstream(&input)) {
      cerr << "Failed to parse part book." << endl;
      return -1;
    }
  }

  // Add a new part.
  PromptForPart(part_book.add_part());

  {
    // Write the new part book back to disk.
    fstream output(argv[1], ios::out | ios::trunc | ios::binary);
    if (!part_book.SerializeToOstream(&output)) {
      cerr << "Failed to write part book." << endl;
      return -1;
    }
  }

  // Optional:  Delete all global objects allocated by libprotobuf.
  google::protobuf::ShutdownProtobufLibrary();

  return 0;
}

Then we write a c++ program to use the library and .proto to serialize the data.

/*list_parts.cc*/
#include <iostream>
#include <fstream>
#include <string>
#include "partbook.pb.h"
using namespace std;

// Iterates though all parts in the PartDetails and prints info about them.
void ListParts(const tutorial::PartDetails& part_book) {
  for (int i = 0; i < part_book.part_size(); i++) {
    const tutorial::Part& part = part_book.part(i);

    cout << "Part ID: " << part.partid() << endl;
    cout << "  Name: " << part.name() << endl;
    if (part.has_vendor()) {
      cout << "  Vendor: " << part.vendor() << endl;
    }

  }
}

// Main function:  Reads the entire part book from a file and prints all
//   the information inside.
int main(int argc, char* argv[]) {
  // Verify that the version of the library that we linked against is
  // compatible with the version of the headers we compiled against.
  GOOGLE_PROTOBUF_VERIFY_VERSION;

  if (argc != 2) {
    cerr << "Usage:  " << argv[0] << " PART_BOOK_FILE" << endl;
    return -1;
  }

  tutorial::PartDetails part_book;

  {
    // Read the existing data file.
    fstream input(argv[1], ios::in | ios::binary);
    if (!part_book.ParseFromIstream(&input)) {
      cerr << "Failed to parse part book." << endl;
      return -1;
    }
  }

  ListParts(part_book);

  // Optional:  Delete all global objects allocated by libprotobuf.
  google::protobuf::ShutdownProtobufLibrary();

  return 0;
}

#Makefile
#This creates executables for python cpp and java , to create only one language use #'make cpp' or 'make python' or 'make java'

.PHONY: all cpp java python clean

all: cpp java python

cpp:    add_part_cpp    list_parts_cpp
java:   add_part_java   list_parts_java
python: add_part_python list_parts_python

clean:
 rm -f add_part_cpp list_parts_cpp add_part_java list_parts_java add_part_python list_parts_python
 rm -f javac_middleman AddPart*.class ListParts*.class com/example/tutorial/*.class
 rm -f protoc_middleman partbook.pb.cc partbook.pb.h partbook.pb2.py com/example/tutorial/PartBookProtos.java
 rm -f *.pyc
 rmdir com/example/tutorial 2>/dev/null || true
 rmdir com/example 2>/dev/null || true
 rmdir com 2>/dev/null || true

protoc_middleman: partbook.proto
 protoc --cpp_out=. --java_out=. --python_out=. partbook.proto
 @touch protoc_middleman

add_part_cpp: add_part.cc protoc_middleman
 pkg-config --cflags protobuf  # fails if protobuf is not installed
 c++ add_part.cc partbook.pb.cc -o add_part_cpp `pkg-config --cflags --libs protobuf`

list_parts_cpp: list_parts.cc protoc_middleman
 pkg-config --cflags protobuf  # fails if protobuf is not installed
 c++ list_parts.cc partbook.pb.cc -o list_parts_cpp `pkg-config --cflags --libs protobuf`

javac_middleman: AddPart.java ListParts.java protoc_middleman
 javac AddPart.java ListParts.java com/example/tutorial/PartBookProtos.java
 @touch javac_middleman

add_part_java: javac_middleman
 @echo "Writing shortcut script add_part_java..."
 @echo '#! /bin/sh' > add_part_java
 @echo 'java -classpath .:$$CLASSPATH AddPart "$$@"' >> add_part_java
 @chmod +x add_part_java

list_parts_java: javac_middleman
 @echo "Writing shortcut script list_parts_java..."
 @echo '#! /bin/sh' > list_parts_java
 @echo 'java -classpath .:$$CLASSPATH ListParts "$$@"' >> list_parts_java
 @chmod +x list_parts_java

add_part_python: add_part.py protoc_middleman
 @echo "Writing shortcut script add_part_python..."
 @echo '#! /bin/sh' > add_part_python
 @echo './add_part.py "$$@"' >> add_part_python
 @chmod +x add_part_python

list_parts_python: list_parts.py protoc_middleman
 @echo "Writing shortcut script list_parts_python..."
 @echo '#! /bin/sh' > list_parts_python
 @echo './list_parts.py "$$@"' >> list_parts_python
 @chmod +x list_parts_python

Save all these files under <proto-installation-path>/examples/ and run make

To run the example
./add_part_cpp <data-filename>
./list_parts_cpp <data-filename>