Recently I have moved from an application development team to
(shared) services team that deals with extremely large data sets (Big data) using process
of extracting data from a traditional database to no SQL database(Big Data
techniques) , cross-platform document-oriented database. Data processed across
the platforms / system that may be analyzed computationally to reveal patterns,
trends, and associations, especially relating to human behavior and
interactions and also the new process / UI systems (java script based) are
being built using the no SQL database.
I have been hearing about the recon /reconciliation process in
the team meetings for every run of the Big data process using ETL and this
being developed in java to generate adhoc reports to find the discrepancies
between the two systems and challenge there is the domain structure between traditional
database data and no SQL database e.g. place holders of data after reading the
data from a database will a POJO’s whereas data from No SQL data base is in a
document style in JSON format that makes to compare the data and find the
differences / recon the data meaning either the JSON object is converted into a
POJO to compare or POJO is converted into JSON and compare the JSON documents.
With previous experience of implementing / using the Spring data JPA in my previous projects developing restful service / API micro services with Spring Boot and Open shift I was reading about the Spring data Mongo dB from recon process prospective.
With previous experience of implementing / using the Spring data JPA in my previous projects developing restful service / API micro services with Spring Boot and Open shift I was reading about the Spring data Mongo dB from recon process prospective.
I have tried implementing POC using below Libraries / API’s
/ frameworks:
- Using flexjson libraries (flexjson.sourceforge.net/) – serialization and deserialization of JSON Object to / from Domain object.
- Using Spring data MongoDB libraries (https://projects.spring.io/spring-data-mongodb/) – project provides integration with the MongoDB document database. Key functional areas of Spring Data MongoDB are a POJO centric model for interacting with a MongoDB DBCollection and easily writing a Repository style data access layer.
Below is the implementation of Spring data Mongodb service
(Stand alone) (using my favorite tables from WestSide Auto project):
- Reading data from Oracle tables using JDBC Template.
- Inserting data into Mongo database.
- Reading data using Mongo Template and comparing the data.
Tables in Oracle XE:
Car.ddl
CREATE
TABLE "RAJESH"."CARS"
( "CARID" NUMBER(*,0),
"MAKE"
VARCHAR2(50 BYTE) NOT NULL ENABLE,
"MODEL"
VARCHAR2(50 BYTE) NOT NULL ENABLE,
"DESCRIPTION"
VARCHAR2(800 BYTE) NOT NULL ENABLE,
PRIMARY KEY ("CARID")
USING
INDEX PCTFREE 10 INITRANS 2 MAXTRANS 255 COMPUTE STATISTICS
STORAGE(INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645
PCTINCREASE
0 FREELISTS 1 FREELIST GROUPS 1 BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT
CELL_FLASH_CACHE DEFAULT)
TABLESPACE
"SYSTEM" ENABLE
) SEGMENT
CREATION IMMEDIATE
PCTFREE 10
PCTUSED 40 INITRANS 1 MAXTRANS 255 NOCOMPRESS LOGGING
STORAGE(INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645
PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1 BUFFER_POOL DEFAULT
FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
TABLESPACE
"SYSTEM" ;
Inventory.ddl
CREATE TABLE
"RAJESH"."INVENTORY"
( "SKU"
NUMBER(*,0),
"CARID"
NUMBER(*,0) NOT NULL ENABLE,
"PRICE"
NUMBER(*,0) NOT NULL ENABLE,
"STATUS"
VARCHAR2(50 BYTE),
PRIMARY KEY ("SKU")
USING
INDEX PCTFREE 10 INITRANS 2 MAXTRANS 255 COMPUTE STATISTICS
STORAGE(INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645
PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1 BUFFER_POOL DEFAULT
FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
TABLESPACE
"SYSTEM" ENABLE,
CONSTRAINT "FK_C007091" FOREIGN KEY
("CARID")
REFERENCES "RAJESH"."CARS"
("CARID") ENABLE
) SEGMENT
CREATION IMMEDIATE
PCTFREE 10
PCTUSED 40 INITRANS 1 MAXTRANS 255 NOCOMPRESS LOGGING
STORAGE(INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645
PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1 BUFFER_POOL DEFAULT
FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
TABLESPACE
"SYSTEM" ;
Java Application:
Domain Objects
/**
*
*/
package com.westside.mongo.domain;
import java.io.Serializable;
import java.math.BigDecimal;
import
org.apache.commons.lang3.builder.CompareToBuilder;
import
org.springframework.data.annotation.Id;
import
org.springframework.data.mongodb.core.mapping.Document;
/**
*
@author Pollepalli
*
*/
@Document(collection="westsidedbCol")
public class Car implements Serializable,
Comparable{
private
static final long serialVersionUID = 1L;
@Id
private
BigDecimal carId;
private
String make;
private
String model;
private
String description;
protected
Inventory inventory;
public
BigDecimal getCarId() {
return
carId;
}
public
void setCarId(BigDecimal carId) {
this.carId
= carId;
}
public
String getMake() {
return
make;
}
public
void setMake(String make) {
this.make
= make;
}
public
String getModel() {
return
model;
}
public
void setModel(String model) {
this.model
= model;
}
public
String getDescription() {
return
description;
}
public
void setDescription(String description) {
this.description
= description;
}
public
Inventory getInventory() {
return
inventory;
}
public
void setInventory(Inventory inventory) {
this.inventory
= inventory;
}
@Override
public
int compareTo(Car o) {
return
CompareToBuilder.reflectionCompare(this, o);
}
@Override
public
String toString() {
return
"Car [carId=" + carId + ", make=" + make + ",
model=" + model + ", description=" + description
+
", inventory=" + inventory + "]";
}
}
/**
*
*/
package com.westside.mongo.domain;
import java.io.Serializable;
import java.math.BigDecimal;
import
org.apache.commons.lang3.builder.CompareToBuilder;
import
org.springframework.data.mongodb.core.mapping.Document;
import
org.springframework.data.mongodb.core.query.SerializationUtils;
/**
*
@author Pollepalli
*
*/
public class Inventory implements
Serializable,Comparable{
/**
*
*/
private
static final long serialVersionUID = 1L;
private
BigDecimal sku;
private
BigDecimal carId;
private
BigDecimal price;
private
String status;
/**
*
*/
public
Inventory() {
//
TODO Auto-generated constructor stub
}
public
Inventory(BigDecimal sku, BigDecimal carId, BigDecimal price, String status) {
super();
this.sku
= sku;
this.carId
= carId;
this.price
= price;
this.status
= status;
}
public
BigDecimal getSku() {
return
sku;
}
public
void setSku(BigDecimal sku) {
this.sku
= sku;
}
public
BigDecimal getCarId() {
return
carId;
}
public
void setCarId(BigDecimal carId) {
this.carId
= carId;
}
@Override
public
String toString() {
return
"Inventory [sku=" + sku + ", carId=" + carId + ",
price=" + price + ", status=" + status + "]";
}
public
BigDecimal getPrice() {
return
price;
}
public
void setPrice(BigDecimal price) {
this.price
= price;
}
public
String getStatus() {
return
status;
}
public
void setStatus(String status) {
this.status
= status;
}
@Override
public
int compareTo(Inventory o) {
return CompareToBuilder.reflectionCompare(this,
o);
}
}
Service Layer:
/**
*
*/
package com.westside.mongo.service;
import java.util.List;
import com.westside.mongo.domain.Car;
/**
* @author Pollepalli
*
*/
public interface WestSideMongoService {
public void
insert(List cars);
public void insert(Car car);
public List
find(List keys, List values);
public Object find(String
key, String value);
public void
insertJsons(List jCars) ;
public void deteleAll();
}
/**
*
*/
package com.westside.mongo.service.impl;
import java.util.List;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.mongodb.core.MongoTemplate;
import org.springframework.data.mongodb.core.query.Criteria;
import org.springframework.data.mongodb.core.query.Query;
import org.springframework.stereotype.Service;
import com.westside.mongo.domain.Car;
import com.westside.mongo.service.WestSideMongoService;
/**
* @author Pollepalli
*
*/
@Service
public class WestSideMongoServiceImpl implements WestSideMongoService {
private MongoTemplate
mongoTemplate;
@Override
public void
insert(List cars) {
mongoTemplate.insertAll(cars);
}
@Override
public void
insertJsons(List jCars) {
mongoTemplate.insertAll(jCars);
}
@Override
public void insert(Car car)
{
mongoTemplate.insert(car);
}
@Override
public List
find(List keys, List values) {
return null;
}
@Override
public Car find(String key,
String value) {
Query
findQuery = new Query();
findQuery.addCriteria(Criteria.where(key).is(value));
return
mongoTemplate.findOne(findQuery, Car.class);
}
public MongoTemplate
getMongoTemplate() {
return
mongoTemplate;
}
public void
setMongoTemplate(MongoTemplate mongoTemplate) {
this.mongoTemplate
= mongoTemplate;
}
@Override
public void deteleAll() {
mongoTemplate.remove(new
Query(), "westsidedbCol");
}
}
/**
*
*/
package com.westside.oracle.service;
import java.util.List;
import
com.westside.mongo.domain.Car;
/**
* @author Pollepalli
*
*/
public interface
WestSideOracleService {
public List
findAll();
}
/**
*
*/
package com.westside.oracle.service.impl;
import java.math.BigDecimal;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Service;
import com.westside.mongo.domain.Car;
import com.westside.mongo.domain.Inventory;
import com.westside.oracle.service.WestSideOracleService;
/**
* @author Pollepalli
*
*/
@Service
public class WestSideOracleServiceImpl implements WestSideOracleService {
public JdbcTemplate jdbcTemplate;
public
List findAll() {
List
cars = new ArrayList();
List
for
(Map row : rows) {
Car
car = new Car();
car.setCarId((BigDecimal)(row.get("carId")));
car.setMake((String)row.get("make"));
car.setModel((String)row.get("model"));
car.setDescription((String)row.get("description"));
Inventory
inventory = new Inventory();
inventory.setCarId((BigDecimal)(row.get("carId")));
inventory.setPrice((BigDecimal)(row.get("price")));
inventory.setSku((BigDecimal)(row.get("sku")));
inventory.setStatus((String)row.get("status"));
car.setInventory(inventory);
cars.add(car);
}
return
cars;
}
public
JdbcTemplate getJdbcTemplate() {
return
jdbcTemplate;
}
public
void setJdbcTemplate(JdbcTemplate jdbcTemplate) {
this.jdbcTemplate
= jdbcTemplate;
}
}
Spring configuration:
/**
*
*/
package com.westside.mongo.spring.config;
import java.util.Arrays;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.convert.converter.Converter;
import org.springframework.data.mongodb.MongoDbFactory;
import org.springframework.data.mongodb.core.MongoTemplate;
import org.springframework.data.mongodb.core.SimpleMongoDbFactory;
import org.springframework.data.mongodb.core.convert.CustomConversions;
import
org.springframework.data.mongodb.core.convert.DefaultMongoTypeMapper;
import
org.springframework.data.mongodb.core.convert.MappingMongoConverter;
import org.springframework.data.mongodb.core.mapping.MongoMappingContext;
import com.mongodb.DBObject;
import com.mongodb.MongoClient;
import com.westside.mongo.service.WestSideMongoService;
import com.westside.mongo.service.impl.WestSideMongoServiceImpl;
/**
* @author Pollepalli
*
*/
@Configuration
public class WestSideMongoConfig {
@Bean
public MongoTemplate mongoTemplate() throws
Exception {
//remove
_class
MappingMongoConverter
converter =
new
MappingMongoConverter(mongoDbFactory(), new MongoMappingContext());
converter.setTypeMapper(new
DefaultMongoTypeMapper(null));
converter.setCustomConversions(new
CustomConversions(Arrays.asList(new DBObjectToStringConverter())));
converter.afterPropertiesSet();
MongoTemplate
mongoTemplate =
//new
MongoTemplate(new MongoClient("127.0.0.1"),"westsidedb");
new
MongoTemplate(mongoDbFactory(), converter);
return
mongoTemplate;
}
@Bean
public MongoDbFactory mongoDbFactory()
throws Exception {
MongoClient mongoClient = new
MongoClient("localhost", 27017);
return new
SimpleMongoDbFactory(mongoClient, "westsidedb");
}
// @Bean
// public MongoTemplate mongoTemplate() throws
Exception {
// MongoTemplate mongoTemplate = new
MongoTemplate(mongoDbFactory());
// return mongoTemplate;
// }
@Bean
public WestSideMongoService
westSideMongoService() throws Exception{
WestSideMongoServiceImpl svc= new
WestSideMongoServiceImpl();
svc.setMongoTemplate(mongoTemplate());
return svc;
}
class DBObjectToStringConverter implements
Converter {
public String convert(DBObject source) {
return source == null ? null :
source.toString();
}
}
}
/**
*
*/
package com.westside.mongo.spring.config;
import javax.sql.DataSource;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.datasource.DriverManagerDataSource;
import com.westside.oracle.service.impl.WestSideOracleServiceImpl;
/**
* @author Pollepalli
*
*/
@Configuration
public class WestSideOracleConfig {
@Bean
public DataSource dataSource()
{
DriverManagerDataSource
dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName("oracle.jdbc.driver.OracleDriver");
dataSource.setUrl("jdbc:oracle:thin:@localhost:1521:xe");
dataSource.setUsername("rajesh");
dataSource.setPassword("rajesh");
return dataSource;
}
@Bean
public JdbcTemplate jdbcTemplate() {
JdbcTemplate jdbcTemplate = new
JdbcTemplate(dataSource());
jdbcTemplate.setResultsMapCaseInsensitive(true);
return jdbcTemplate;
}
@Bean
public WestSideOracleService orclService(){
WestSideOracleServiceImpl svc= new WestSideOracleServiceImpl();
svc.setJdbcTemplate(jdbcTemplate());
return svc;
}
}
Driver to test:
Inserting data into MongoDB after reading
from Oracle:
/**
*
*/
package com.westside.test.driver;
import java.util.Iterator;
import java.util.List;
import org.bson.Document;
import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.AnnotationConfigApplicationContext;
import com.mongodb.BasicDBObject;
import com.mongodb.MongoClient;
import com.mongodb.client.FindIterable;
import com.mongodb.client.MongoCollection;
import com.mongodb.client.MongoDatabase;
import com.westside.mongo.domain.Car;
import com.westside.mongo.service.WestSideMongoService;
import com.westside.mongo.spring.config.WestSideMongoConfig;
import com.westside.mongo.spring.config.WestSideOracleConfig;
import com.westside.oracle.service.WestSideOracleService;
import flexjson.JSONSerializer;
/**
* @author Pollepalli
*
*/
public class WestSideMongoDriver {
/**
* @param args
*/
public static void
main(String[] args) {
ApplicationContext
orclCtx =
new
AnnotationConfigApplicationContext(WestSideOracleConfig.class);
ApplicationContext
mongoCtx =
new
AnnotationConfigApplicationContext(WestSideMongoConfig.class);
WestSideOracleService
westSideOracleService = (WestSideOracleService)
orclCtx.getBean("orclService");
WestSideMongoService
mongoOperation =
(WestSideMongoService)
mongoCtx.getBean("westSideMongoService");
//delete all
the data from collection
mongoOperation.deteleAll();
//read the
data from Oracle
List
cars= westSideOracleService.findAll();
//adding data
into Mongo
mongoOperation.insert(cars);
}
}
{
"_id" :
"1",
"make" :
"Volkswagen",
"model" : "New
Beetle",
"description" :
"The 2003 New Beetle is a 2-door, 4-passenger family coupe, or sports
coupe, \navailable in 8 trims, ranging from the GL 2.0L to the Turbo S
1.8L.",
"inventory" : {
"sku" :
"1",
"carId" :
"1",
"price" :
"18999"
}
}
{
"_id" :
"2",
"make" :
"Mini",
"model" :
"Cooper",
"description" :
"The new Mini Cooper is more fun than a carnival ride. Both models handle
\nlike sports cars and the Mini Cooper S can accelerate from 0 to 60 mph in
\nless than seven seconds.",
"inventory" : {
"sku" :
"2",
"carId" :
"2",
"price" :
"17597"
}
}
{
"_id" :
"3",
"make" :
"Mercedes-Benz",
"model" :
"SL-Class",
"description" :
"The 2003 SL-Class is a 2-door, 2-passenger luxury convertible, or
convertible \nsports car, available in two trims, the SL500 Roadster and the
SL55 \nAMG Roadster.",
"inventory" : {
"sku" :
"3",
"carId" :
"3",
"price" :
"92399"
}
}
{
"_id" :
"4",
"make" : "Land
Rover",
"model" : "Range
Rover",
"description" :
"The 2003 Range Rover is a 4-door, 5-passenger luxury sport-utility,
\navailable in one trim only, the HSE.",
"inventory" : {
"sku" :
"4",
"carId" :
"4",
"price" :
"71200"
}
}
{
"_id" :
"5",
"make" :
"Honda",
"model" : "Civic
Coupe",
"description" :
"The 2003 Civic Coupe is a 2-door, 5-passenger family coupe, available
\nin 16 trims, ranging from the DX 5-spd MT to the EX 4-spd AT w/ \nFront Side
Airbags.",
"inventory" : {
"sku" :
"5",
"carId" : "5",
"price" :
"12110"
}
}
Comparing the data between Oracle
and Mongo:
/**
*
*/
package com.westside.test.driver;
import java.util.List;
import org.springframework.context.ApplicationContext;
import
org.springframework.context.annotation.AnnotationConfigApplicationContext;
import com.westside.mongo.domain.Car;
import com.westside.mongo.service.WestSideMongoService;
import com.westside.mongo.spring.config.WestSideMongoConfig;
import com.westside.mongo.spring.config.WestSideOracleConfig;
import com.westside.oracle.service.WestSideOracleService;
/**
* @author Pollepalli
*
*/
public class WestSideMongoDriver {
/**
* @param args
*/
public static void
main(String[] args) {
ApplicationContext
orclCtx =
new
AnnotationConfigApplicationContext(WestSideOracleConfig.class);
ApplicationContext
mongoCtx =
new
AnnotationConfigApplicationContext(WestSideMongoConfig.class);
WestSideOracleService
westSideOracleService = (WestSideOracleService)
orclCtx.getBean("orclService");
WestSideMongoService
mongoOperation =
(WestSideMongoService)
mongoCtx.getBean("westSideMongoService");
//read the
data from Oracle
List
cars= westSideOracleService.findAll();
//compare
the data
compare(cars);
}
private static void
compare(List cars) {
ApplicationContext
mongoCtx =
new
AnnotationConfigApplicationContext(WestSideMongoConfig.class);
WestSideMongoService
mongoOperation =
(WestSideMongoService)
mongoCtx.getBean("westSideMongoService");
/**
* Compare the data with Mongo document
*/
for(Car
car:cars){
Car
mCar = (Car) mongoOperation.find("_id",
car.getCarId().toPlainString());
System.out.println("mCar
"+mCar);
System.out.println("Comparing
results :" + car.compareTo(mCar));
}
}
}
Comparing
Results from Logs:
Car from Oracle : Car [carId=1,
make=Volkswagen, model=New Beetle, description=The 2003 New Beetle is a 2-door,
4-passenger family coupe, or sports coupe,
available in 8 trims, ranging from
the GL 2.0L to the Turbo S 1.8L., inventory=Inventory [sku=1, carId=1,
price=18999, status=null]]
Car from MONGO : Car [carId=1,
make=Volkswagen, model=New Beetle, description=The 2003 New Beetle is a 2-door,
4-passenger family coupe, or sports coupe,
available in 8 trims, ranging from
the GL 2.0L to the Turbo S 1.8L., inventory=Inventory [sku=1, carId=1,
price=18999, status=null]]
Comparing results :0
Car from Oracle : Car [carId=2,
make=Mini, model=Cooper, description=The new Mini Cooper is more fun than a
carnival ride. Both models handle
like sports cars and the Mini
Cooper S can accelerate from 0 to 60 mph in
less than seven seconds.,
inventory=Inventory [sku=2, carId=2, price=17597, status=null]]
Car from MONGO : Car [carId=2,
make=Mini, model=Cooper, description=The new Mini Cooper is more fun than a
carnival ride. Both models handle
like sports cars and the Mini
Cooper S can accelerate from 0 to 60 mph in
less than seven seconds.,
inventory=Inventory [sku=2, carId=2, price=17597, status=null]]
Comparing results :0
Car from Oracle : Car [carId=3,
make=Mercedes-Benz, model=SL-Class, description=The 2003 SL-Class is a 2-door,
2-passenger luxury convertible, or convertible
sports car, available in two
trims, the SL500 Roadster and the SL55
AMG Roadster., inventory=Inventory
[sku=3, carId=3, price=92399, status=null]]
Car from MONGO : Car [carId=3,
make=Mercedes-Benz, model=SL-Class, description=The 2003 SL-Class is a 2-door,
2-passenger luxury convertible, or convertible
sports car, available in two
trims, the SL500 Roadster and the SL55
AMG Roadster., inventory=Inventory
[sku=3, carId=3, price=92399, status=null]]
Comparing results :0
Car from Oracle : Car [carId=4,
make=Land Rover, model=Range Rover, description=The 2003 Range Rover is a
4-door, 5-passenger luxury sport-utility,
available in one trim only, the
HSE., inventory=Inventory [sku=4, carId=4, price=71200, status=null]]
Car from MONGO : Car [carId=4,
make=Land Rover, model=Range Rover, description=The 2003 Range Rover is a
4-door, 5-passenger luxury sport-utility,
available in one trim only, the
HSE., inventory=Inventory [sku=4, carId=4, price=71200, status=null]]
Comparing results :0
Car from Oracle : Car [carId=5,
make=Honda, model=Civic Coupe, description=The 2003 Civic Coupe is a 2-door,
5-passenger family coupe, available
in 16 trims, ranging from the DX
5-spd MT to the EX 4-spd AT w/
Front Side Airbags.,
inventory=Inventory [sku=5, carId=5, price=12110, status=null]]
Car from MONGO : Car [carId=5,
make=Honda, model=Civic Coupe, description=The 2003 Civic Coupe is a 2-door,
5-passenger family coupe, available
in 16 trims, ranging from the DX
5-spd MT to the EX 4-spd AT w/
Front Side Airbags.,
inventory=Inventory [sku=5, carId=5, price=12110, status=null]]
Comparing results :0
Note : A special thanks to my manager Ramu Akula for trusting me and giving opportunity in the team.
Conclusion:
Spring data MongoDB provides Cross-store persistence - support for JPA
Entities with fields transparently persisted/retrieved using MongoDB that can be utilized effectively easily to compars the data between traditional database and No SQL document based database.
Note : A special thanks to my manager Ramu Akula for trusting me and giving opportunity in the team.
No comments:
Post a Comment