Market Structure

 

Problem:

 

Existing software was performing too slowly and adding new functionality was not possible due to the lack of internal architecture. Large databases (over 1 million records) would take overnight processing to get results, sometimes the system would not be able to handle them and crash.

 

Solution:

 

  • Rewrote the software using C# and allowing the user to choose between SQL server or SQL CE as data storage. This converted the system to a file base system where the data is easily shared among users like an excel spreadsheet.
  • Accelerated results processing and reporting by a factor of 100x, by creating column bitmap indices on top of the user chosen database back end, this allows the user to discover multi-level structures in the underlying data of more than a million of records in a few seconds.
  • Used the performance increase to stream line the usage workflow, this reduced the number of steps required to obtain reports, and made some of them to be displayed live as the user were working with the data
  • Included multiple ways to exchange data with other packages including Excel, and SPSS
  • Added the ability to edit variables by a user friendly GUI, or by directly typing its meaning. This provided a much more flexible way to work with the data.
  • Used a pure MVC pattern on the Win Forms application to organize the organically growth of features.
  • Full Office 2010 style Look and Feel that reduced the amount of training.
  • More features have been added without taxing the performance (I.e. structure sizing, asymmetrical structures, etc.)
  • Created and implement a scripting language which allows creating, manipulating, and reporting over structures in a 4GL fashion. This is a feature for advanced users since the UI is pure text, but it can create much complex structures and a faster processing than the UI counterpart

 

Technologies used:

SQL Server, SQL CE, C# 4.0, Visual Studio 2010, DevExpress controls suite, External DLL integration, Cruise Control for continuous integration, Vault for source code management. Modified TinyPG as LL(1) Lexer and Parser for text editor