Sony Arouje Blog

a programmer's log

Posts Tagged ‘itraveller

Asynchronous Messaging in iTraveller

with 2 comments

Download iTraveller

I decided to use Lucene.Net to implement Search functionality in iTraveller. But the main design constrain was how to implement the indexing process without affecting the performance of any other functionality. Below are my use cases

  • Should start indexing asynchronously, no other process like Insert/Update should wait to complete indexing.
  • Other process like Insert/Update send a message to Indexing process and continue without waiting to finish the indexing process.
  • Lucene.Net will apply lock before start indexing. So only one process can access the Lucene.Net indexing API at a time.
  • Should have a single entry point for other process to invoke indexing.
  • Before indexing remove the record (if exist) from Index file.
  • Gather all the data related to photo like Photo Title, Description, Comments, etc. Index the data using Lucene.Net

In the above points, the first  one is the important thing I wanted to achieve to make iTraveller running smoothly. It’s obvious that I have to implement threading, and started my search for a thread safe queuing approach. At last I decided to use ‘Queue’ in System.Collections.Generic namespace. Let’s go through how I implemented message queue in iTraveller.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
using System.IO;
using ITraveller.Domain;
using ITraveller.Utils.Logging;
namespace ITraveller.Indexer
{
    public class IndexQueue
    {
        private static Queue<Photo> _queue;
        private static ManualResetEvent _enqueuedEvent;
        private static Thread _workerThread;
        private static Photo _currentPhoto;
        private static ILogger _logger;
        static IndexQueue()
        {
            _queue = new Queue<Photo>();
            _logger = Logger.DefaultLogger;
            _enqueuedEvent = new ManualResetEvent(false);
            _workerThread = new Thread(new ThreadStart(PerformIndex));
            _workerThread.Start();
        }
        public static void RestartThread()
        {
            if (_workerThread.ThreadState == ThreadState.Stopped)
            {
                _workerThread.Abort();
                _workerThread = new Thread(new ThreadStart(PerformIndex));
                _workerThread.Start();
            }
        }
        public static void Add(Photo photo)
        {
            try
            {
                if (_queue == null)
                    _queue = new Queue<Photo>();
                lock (_queue)
                {
                    _queue.Enqueue(photo);
                    _enqueuedEvent.Set();
                }
                RestartThread();
            }
            catch (Exception ex)
            {
                _logger.LogInfo("Error while Adding to index Queue");
                _logger.LogError(ex);
            }
        }
        public static void Add(List<Photo> photos)
        {
            if (_queue == null)
                _queue = new Queue<Photo>();
            for (int i = 0; i < photos.Count; i++)
            {
                try
                {
                    lock (_queue)
                    {
                        _queue.Enqueue(photos[i]);
                        _enqueuedEvent.Set();
                    }
                }
                catch (Exception ex)
                {
                    _logger.LogInfo("Error while Adding to index Queue in loop");
                    _logger.LogError(ex);
                }
            }
            RestartThread();
        }
        private static bool Dequeue()
        {
            lock (_queue)
            {
                if (_queue.Count > 0)
                {
                    _enqueuedEvent.Reset();
                    _currentPhoto = _queue.Dequeue();
                }
                else
                {
                    return false;
                }

                return true;
            }
        }

        private static void PerformIndex()
        {
            IndexWriter writer=null;
            try
            {
                writer = new IndexWriter("./SearchIndex");
                while (Dequeue())
                {
                    IndexData indexData = new IndexData();
                    writer.DeleteFromIndex("ImageId:" + _currentPhoto.ImageID.ToString());
                    writer.WriteToIndex(indexData.GetIndexingData(_currentPhoto));
                }
            }
            catch (Exception ex)
            {
                _logger.LogInfo("Error while indexing data in worker thread");
                _logger.LogError(ex);
            }
            finally
            {
                if(writer!=null)
                    writer.Dispose();
            }
        }
    }
}

Other process can add Photo entity for indexing just by calling the static Add function and pass the required Photo( s ) object as parameter. Once the photo got added to queue, the Add function will call RestartThread function. The RestartThread function will check whether the worker thread Stopped, if Stopped then Recreate and Start the thread.

The function PerformIndex will get called when the worker thread starts. The PerformIndex function will dequeue the photo one by one from the Queue. Once the photo is dequeued successfully then one helper class will create and format the data for indexing and then  pass it to Lucene.Net class for further processing. The dequeue process will remove the photo object from the queue as well.

With the above approach I achieved a complete asynchronous messaging system. Any suggestion or better approach is always welcome.

Written by Sony Arouje

February 21, 2011 at 6:30 pm

db4o A light weight Object Oriented Database

with 8 comments

Now a days I am spending my free time in rewriting one of my freeware for Flickr users called iTraveller. One of the design goals was the application should be scalable. My major concern was, I cannot freeze my db design before my first release because I am planning to have multiple version with added functionalities, so the db structure may change in the future. If I use conventional db’s then I have to write functionality to add or remove columns. iTraveller is not very data centric application, so managing db change scripts and writing a functionality to apply these changes when user upgrade to new version will be a big pain for me.

First I thought of using MongoDB, because I used Mongo db in a Silverlight application. But then finally dropped that idea. I want some thing more lighter than Mongo and NoSql sort of database. I evaluated several db’s but nothing suites what I am looking for. Finally my search ends at db4o. The usage of db4o is very straight forward and very easy to learn, db4o installer comes with a very good documentation to start off with the development.

db4o is a light weight Object Oriented database. Using db4o I can persist my entities very easily without any configuration. I can add or remove new properties to my entities without affecting the existing persisted entities.

I created a generic repository sort of class around db4o, the same way I did one for Entity Framework. This generic repository reduced lot of my work and act as the entry point to persist my entities to db4o data file. Below is the Generic repository I created for db4o.

using Db4objects.Db4o;
public class GenericRepository:IRepository,IDisposable
{
    private static IRepository _singleRepoInstance;
    public const string DBNAME = "Data.dat";

    public static IRepository GetRepositoryInstance()
    {
        if (_singleRepoInstance == null)
            _singleRepoInstance = new GenericRepository();

        return _singleRepoInstance;
    }

    IObjectContainer _context = null;
    private IObjectContainer Context
    {
        get 
        {
            if(_context==null)
                _context = Db4oFactory.OpenFile(DBNAME);
            
            return _context;
        }
    }

    private IList<TEntity> Query<TEntity>() where TEntity : class
    {
        return this.Context.Query<TEntity>();
    }

    public IList<TEntity> GetAll<TEntity>() where TEntity : class
    {
        return this.Query<TEntity>().ToList();
    }

    public IList<TEntity> GetAll<TEntity>(TEntity entity) where TEntity : class
    {
        return this.Context.QueryByExample(entity).Cast<TEntity>().ToList();
    }

    public IList<TEntity> Find<TEntity>(Func<TEntity, bool> criteria) where TEntity : class
    {
        return this.Query<TEntity>().Where<TEntity>(criteria).ToList<TEntity>();
    }

    public TEntity Single<TEntity>(Func<TEntity, bool> criteria) where TEntity : class
    {
        return this.Query<TEntity>().Single<TEntity>(criteria);
    }

    public TEntity First<TEntity>(Func<TEntity, bool> criteria) where TEntity : class
    {
        return this.Query<TEntity>().FirstOrDefault<TEntity>(criteria);
    }
    public TEntity First<TEntity>(TEntity criteria) where TEntity : class
    {
        return this.Context.QueryByExample(criteria).Cast<TEntity>().First<TEntity>();
    }
    public void Add<TEntity>(TEntity entity) where TEntity : class
    {
        this.Context.Store(entity);
    }

    public void Add<TEntity>(List<TEntity> entities) where TEntity : class
    {
        foreach (TEntity entity in entities)
        {
            this.Add<TEntity>(entity);
        }
    }

    public void Delete<TEntity>(TEntity entity) where TEntity : class
    {
        this.Context.Delete(entity);
    }

    public void Update<TEntity>(TEntity entity) where TEntity : class
    {
        this.Context.Store(entity);
    }

    public void Dispose()
    {
        this._context.Close();
        this._context.Dispose();
        this._context = null;
    }
}

The above repository is created in singletone mode. I did that way because I noticed that opening the database file is taking some delay and wanted to avoid that delay. So I wont close the data file until iTraveller get closed.

Let’s see how we can call the above repository to save an entity

GenericRepository repository = GenericRepository.GetRepositoryInstance();
repository.Add<Photo>(photos);

 

Below is the way we can retrieve some entities from the data file

IRepository repository = GenericRepository.GetRepositoryInstance();
List<LocalCategory> categories = repository.GetAll<LocalCategory>().ToList<LocalCategory>();
 

 

You can also retrieve data based on any condition as well. The below code will return a list of Photo object whose categoryId is 10.

IRepository repository = GenericRepository.GetRepositoryInstance();
return repository.Find<Photo>(p => p.Category.ID == 10).ToList<Photo>();

 

As you can see the db4o is very easy to use light weight database without the hurdles of mapping to tables and all sort of stuffs.

Written by Sony Arouje

January 5, 2011 at 4:26 pm

Posted in .NET

Tagged with , , ,

Follow

Get every new post delivered to your Inbox.

Join 181 other followers

%d bloggers like this: