[If] we want to run Mysql Queries on the partitioned Mysql Database, does HiveDb expose any interface for that. I understand that it would not suppprt join operations on the partitioned database, but will it support simple, select and update query ?

The short answer is

Yes absolutely, anything you can do with MySQL you can do with HiveDB.

In essence HiveDB is just a coordinator that sits atop multiple MySQL databases and allows you to access them as a single data set. We think this is one of its great strengths. Anything you can do with MySQL you can do with HiveDB as long as you are aware of a few constraints. The first of which is that you can’t join across shards.

Another constraint is that keep the directory and data nodes must be kept in sync. HiveDB indexes certain values in a directory database so that it can use them to locate records. For example, say you are sharding your data by user id. When some one logs in you need to fetch their user data, but you may not know their user id. However you do know their email. HiveDB can keep an index of email -> user id -> shard so that you can locate the record. So, if the user’s email address changes you need to make sure that you update the directory entry.

HiveDB provides you with two ways to access you data. The first, is via the standard JDBC Connection. HiveDB can act as a connection provider to your sharded data. You say, “I need a read/write connection to joey@fakestreet.com’s data.” and HiveDB hands back an ordinary JDBC connect to the shard that your data resides on. If you it in this way you need to maintain directory synchronization yourself using the hive.directory().* methods. The second type of access it provides is via a Hibernate API. If you are using Hibernate ORM you can add some special annotations to your entity classes and swap in the HiveDB SessionFactory for Hibernate’s and use Hibernate just like you were before. Our Hibernate implementation will take care of all of the indexing behind the scenes.

Finally, if you need to do maintenance you can just connect directly to the shards or the directory with a MySQL client. Again you just have to be mindful of keeping the directory and data nodes in sync if you mutate any of the data.

We think one of the great advantages of HiveDB is that the guts of it are just MySQL. You can take advantage of all of the administration tools already available for MySQL, you can use MySQL native replication and there’s a large pool of expertise and wisdom on how to tune and configure MySQL. You can actually go out and hire some one to operate it.

We have changed our default connection pooling implementation from DBCP to C3P0. C3P0 has better failure characteristics and we think it will make HiveDB more stable and more fault tolerant. However, it also forces us to update HiveBasicDataSource in such a way that it breaks Java5 compatibility.

• Presentation Slides (online)
• Presentation Slides (PDF)

• Download the source tarball screencast.tar.gz
• Checkout the screencast branch from GitHub

  
  $ git clone git://github.com/britt/hivedb.git
  $ cd hivedb/
  $ git checkout --track -b screencast-example origin/screencast
  
  

We’ve put together a screencast that gives a quick introduction to working with HiveDB and Hibernate Shards. We’ve tried hard to make it as easy and as much like working with regular Hibernate as possible. With this tutorial you should be able to get a partitioned data store running and ready for development in just a few minutes.

Horizontal Data Partitioning with HiveDB & Hibernate Shards from Britt Crawford on Vimeo.

Pick your poison:

• Vimeo
• Blip.tv
• Quicktime

From now on HiveDB and HiveDB-Blobject will be located at:

• HiveDB (git://github.com/britt/hivedb.git)
• Blobject (git://github.com/britt/hivedb-blobject.git)

The old subversion repository will stay online but no new code will be added to it.

To get the source code you can run

git clone git://github.com/britt/hivedb.git

or you can download the tarball.

More info on git

• Why Distributed Version Control Matters to You, Today
• Everyday Git in 20 Commands or So
• Git Magic - a good git manual
• Installing Git on OSX
• … or there’s always git help <command> the man pages are a lot better than most.

P.S. This maybe the first time that we have mentioned the hivedb-blobject project on the blog. There is more information to come, but the short description is that Blobject is a library that plugs into Hibernate and lets you store the bulk of an object as a serialized blob. Why would you want to do this you might ask? Well, an alter statement on 10^9 rows doesn’t complete very quickly. Blobject lets you store the bulk of your objects as serialized blobs. This way you can store multiple versions of the blob schema in the database simultaneously and upgrade them on demand as they are retrieved.

• Presented by: Britt Crawford (Cafepress.com), Justin McCarthy (Cafepress.com)
• Time: 10:50am - 11:50am Thursday, 04/17/2008
• Tracks: Architecture and Technology, General, Java, LAMP, MySQL Cluster and High Availability, Replication and Scale-Out
• Location: Ballroom D

Massively scalable systems used to be the sole province of large corporations with acronyms for names and budgets the size of a small country’s GDP, but with the explosion of user-created content sites scaling is now everybody’s problem.

One of the main strategies for scaling databases is to partition the data across many servers. This is a simple enough idea but requires a good deal of expertise in systems design, programming, and MySQL administration to execute successfully. Many horizontal partitioning systems have been built and put into production in corporate environments, but so far none of them have:

• Built an easily extensible platform
• Released their system under an open source license

This session introduces HiveDB, an open source framework for partitioning MySQL that implements the core ideas behind horizontal partitioning in as clear and concise a way as possible.

We had some recent DNS difficulties that took us offline over weekend. The problem was compounded by the fact that I was out of town and not paying attention to email. Sorry about that,

List entityClasses = Lists.newList(Continent.class, WeatherReport.class, WeatherEvent.class);
ShardAccessStrategy strategy = new SequentialShardAccessStrategy();
HiveSessionFactory factoryBuilder = new HiveSessionFactoryBuilderImpl( hiveUri, entityClasses, strategy);
WeatherReport report = new WeatherReport();
...
Session session = factoryBuilder.openSession();
Transaction tx = null;
try {
	tx = session.beginTransaction();
	session.saveOrUpdate(report);
} catch( RuntimeException e ) {
	if(tx != null)
		tx.rollback();
} finally {
	session.close();
}

@Resource("WeatherReport")
public interface WeatherReport {
    @PartitionIndex
    String getContinent();

    @EntityId
    Integer getReportId();

    @Index
    Integer getRegionCode();

    @Index(type=IndexType.Data)
    Double getLatitude();

    @Index(type=IndexType.Data)
    Double getLongitude();

    int getTemperature();
}

ConfigurationReader reader  = new ConfigurationReader(WeatherReport.class);
reader.install(hiveUri);

There are a lot more new features including automated proxying for entity classes, simple data validation, serialized blob storage along with many updates and improvements to the core HiveDB code. We intend for this to be the last development release before HiveDB 1.0. We’ll be updating all of the documentation and posting more about the new features in the coming weeks.

Finally, come and see us at MySql Conference 2008.

<dependency>
   <groupId>org.hivedb</groupId>
   <artifactId>hivedb</artifactId>
   <version>0.9.2</version>
</dependency>