Big Data: Working with Oracle NoSQL (KVLite)

KVLite is a single-node, single Replication Group store. It usually runs in a single process and is used to develop and test client applications. KVLite is installed when you install Oracle NoSQL Database.

Environment
Windows 7 64bit, Oracle NoSQL is certified for Linux and Solaris but can be tested on Windows too.
Prerequisite
- download JDK 7u11 from here and install
- download and unzip KVLite (kv-2.0.26) from here at your desired location
- familiarity with Java (JDK)
Output
Here I created command line Java application (NoSQL Client) which does the following
- Takes three arguments on command line and puts them in NoSQL DB
- If one argument is provided (as key) , its related values are displayed


1-Starting KVLite (with default options)

KVLite is started by using the kvlite utility (run by java interpreter) , which can be found in KVHOME/lib/kvstore.jar. I unzipped it in C:\temp

C:\temp\kv-2.0.26>java -jar lib/kvstore.jar kvlite
Created new kvlite store with args:
-root ./kvroot -store kvstore -host Inam-pc -port 5000 -admin 5001

When KVLite has started successfully, it issues one of two statements to stdout, depending on whether it created a new store or is opening an existing store.

c:\temp\kv-2.0.26>java -jar lib/kvstore.jar kvlite
Opened existing kvlite store with config:
-root ./kvroot -store kvstore -host Inam-pc -port 5000 -admin 5001

where
store kvstore =>  name of the store
host Inam-pc =>  name of the local host
/kvroot => directory where Oracle NoSQL Database data is placed (known as KVROOT)

It takes about 10 - 60 seconds before this message is issued, depending on the speed of your machine.

2- Verifying the Installation, you can use several ways to verify the installation
i) Start another shell and run
c:\temp\kv-2.0.26>set path=C:\glassfish3\jdk7\bin;%path%
c:\temp\kv-2.0.26>jps -m
8160 Jps -m
6672 -- process information unavailable

ii) Run the kvclient test application

c:\temp\kv-2.0.26>java -jar lib/kvclient.jar
11gR2.2.0.26

iii) Compile and run the example program, there should not be any error

C:\temp\kv-2.0.26>javac -g -cp lib/kvclient.jar;:examples examples/hello/HelloBigDataWorld.java

Run the example using all default parameters:
C:\temp\kv-2.0.26>java -cp lib/kvclient.jar;examples hello.HelloBigDataWorld
Hello Big Data World!

3- Record Design Considerations

Oracle NoSQL Database provides the KVStore, which offers storage of key-value pairs. Each such pair can be thought of as a single record in a database, where the key is used to locate the value. Both the key and the value are application-defined.

The value portion of the record can be simply a byte array, but in most cases it should use
Avro to identify its schema.

Avro Schemas

Avro (an API) is used to define the data schema for a record's value. This schema describes the fields allowed in the value, along with their data types. You apply a schema to the value portion of an Oracle NoSQL Database record using Avro bindings. These bindings are used to serialize values before writing them, and to deserialize values after reading them. The usage of these bindings requires your applications to use the Avro data format, which means that each stored value is associated with a schema. 

The Avro API is the result of an open source project provided by the Apache Software
Foundation. In addition, Avro makes use of the Jackson APIs for parsing JSON.

Creating Avro Schemas
An Avro schema is created using JSON format. JSON is short for JavaScript Object Notation,
and it is a lightweight, text-based data interchange format that is intended to be easy for
humans to read and write.


To describe an Avro schema, you create a JSON record which identifies the schema
{
    "type": "record",
    "name": "MemInfo",
    "namespace": "avro",
    "fields": [
        {"name": "ID", "type": "int", "default":0},
        {"name": "NAME", "type": "string", "default":""},
        {"name": "ADDRESS", "type": "string", "default":""}
    ]
}

To use the schema, you must define it in a flat text file, and then add the schema to your
store using the appropriate command line call. You must also somehow provide it to your
code. The schema that your code is using must correspond to the schema that has been added to your store.

4- Developing for Oracle NoSQL Database
a) Create schema and then add it to kvstore
Create a text file (MemInfo.avsc) and paste the code below. Save the file in KVHOME/examples/avro/

MemInfo.avsc
{
    "type": "record",
    "name": "MemInfo",
    "namespace": "avro",
    "fields": [
        {"name": "ID", "type": "int", "default":0},
        {"name": "NAME", "type": "string", "default":""},
        {"name": "ADDRESS", "type": "string", "default":""}
    ]
}

b) add schema to store by using runadmin utility, so run it
C:\temp\kv-2.0.26>java -jar lib/kvstore.jar runadmin -port 5000 -host Inam-PC
You will get the kv prompt where you will add the schema
kv-> ddl add-schema -file C:/temp/kv-2.0.26/examples/avro/MemInfo.avsc
Added schema: avro.MemInfo.2

You can see all schemas added to store using show schemas on kv prompt
kv-> show schemas
avro.MemInfo
  ID: 2  Modified: 2013-04-10 07:15:03 UTC, From: Inam-pc
avro.MyInfo
  ID: 1  Modified: 2013-04-06 10:59:48 UTC, From: Inam-pc

you can see the schema definition also

kv-> show schema -name avro.MemInfo
{
  "type" : "record",
  "name" : "MemInfo",
  "namespace" : "avro",
  "fields" : [ {
    "name" : "ID",
    "type" : "int",
    "default" : 0
  }, {
    "name" : "NAME",
    "type" : "string",
    "default" : ""
  }, {
    "name" : "ADDRESS",
    "type" : "string",
    "default" : ""
  } ]
}
kv->

c) Now write the client application that will read/write to the store (NoSQL database) using KVLite provided APIs. I've written a command line application but web version can be prepared using the same approach. Place the file in KVHOME/examples/avro

MemInfo.java
package avro;
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.util.Arrays;

package avro;
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.util.Arrays;

import org.apache.avro.Schema;
import org.apache.avro.generic.GenericData;
import org.apache.avro.generic.GenericRecord;

import org.codehaus.jackson.JsonNode;
import org.codehaus.jackson.map.ObjectMapper;
import org.codehaus.jackson.node.JsonNodeFactory;
import org.codehaus.jackson.node.ObjectNode;

import oracle.kv.KVStore;
import oracle.kv.KVStoreConfig;
import oracle.kv.KVStoreFactory;
import oracle.kv.Key;
import oracle.kv.ValueVersion;
import oracle.kv.avro.AvroCatalog;
import oracle.kv.avro.GenericAvroBinding;

public class MemInfo {

    private static final String SCHEMA_FILE_NAME = "examples/avro/MemInfo.avsc";
    private static final String MEM_INFO_SCHEMA_NAME = "avro.MemInfo";
   
    private final KVStore store;
    private final AvroCatalog catalog;
    private final Schema memInfoSchema;
    private final GenericAvroBinding binding;
   
    public static void main(String args[]) {
            try {
                if (args.length < 1) System.out.println("No valid arguments given...");
                MemInfo memInfo = new MemInfo();
                memInfo.runExample(args);
            } catch (Exception e) {
                    e.printStackTrace();
            }
        }//main
   
    MemInfo() throws IOException {

        String storeName = "kvstore";
        String hostName = "Inam-PC";
        String hostPort = "5000";
       
        /* Open the KVStore. */
        store = KVStoreFactory.getStore(new KVStoreConfig(storeName, hostName + ":" + hostPort));
       
        /* Parse the schemas file. */
        final Schema.Parser parser = new Schema.Parser();
        parser.parse(new File(SCHEMA_FILE_NAME));
       
        /* Get the schema from the parser's type map. */
        memInfoSchema =    parser.getTypes().get(MEM_INFO_SCHEMA_NAME);
       
        /* Create a Generic binding for the MemInfo schema. */
        catalog = store.getAvroCatalog();
        binding = catalog.getGenericBinding(memInfoSchema);
       
    }//Constructor
             
    /* Performs example operations and closes the KVStore. */
    void runExample(String args[]) {
         ValueVersion valueVersion=null;
         GenericRecord gr=null;;
        /* Use key "/mb/args[0]" to store the object. */
        final Key key = Key.createKey(Arrays.asList("mb", args[0]));
       
        /* Read the value we previous stored, if any. */
        if (args.length == 1)
        try{
            {
                valueVersion = store.get(key);
                gr = binding.toObject(valueVersion.getValue());
                /* Print object as a JSON string. */
                System.out.println("Key Found:\n" +gr.toString());
            }
        }catch(Exception ex){System.out.println("Key "+args[0]+" not available in KVStore"); }       
           
        /* Create a fresh object/record. */
        if (args.length == 3)
        {
            gr = createMem(Integer.parseInt(args[0]),args[1],args[2]);
            /* Serialize the GenericRecord and write it. */
            store.put(key, binding.toValue(gr));
            System.out.println("Values Stored...");
           
        }
       
        
        store.close();
    }//runExample
   
    /**
     * Uses the Avro API to create a GenericRecord that conforms to the
     * MemInfo schema.
     */        private GenericRecord createMem(int PID,String PNAME,String PADDRESS) {
        final GenericRecord gr = new GenericData.Record(memInfoSchema);
        gr.put("ID", PID);
        gr.put("NAME", PNAME);
        gr.put("ADDRESS", PADDRESS);
        return gr;
    } //createMember
}//class

5- Compile the application, set the path of jdk to compile correctly
PATH=C:\glassfish3\jdk7\bin;%path%
--compile the app
c:\temp\kv-2.0.26>javac -g -cp lib/kvclient.jar;examples examples/avro/MemInfo.java
 
6-  run/test the appa) adding values to kvstore
c:\temp\kv-2.0.26>java  -cp lib/kvclient.jar;examples avro.MemInfo 100 Inam Riyadh
c:\temp\kv-2.0.26>java  -cp lib/kvclient.jar;examples avro.MemInfo 200 Abuzar Lahore
Values Stored...

  
b) reading values from kvstore
c:\temp\kv-2.0.26>java  -cp lib/kvclient.jar;examples avro.MemInfo 100
Key Found:
{"ID": 100, "NAME": "Inam", "ADDRESS": "Riyadh"}

Related Posts:
Big Data: A Brief Intro
Oracle NoSQL Database - Intro