I have worked on a project for around 4 years, from full-time development, phased releases and ongoing maintenance. The project is called Online Registration for our client IMG and as I write this I’m currently involved in my 3rd iteration of this system. As this new build is in full swing I felt it would be a good idea to look at where we started, and how we got to where we are today with the project.
In 2012 I joined ATLARGE as an engineer and inherited a project called Online Registration for IMG Academy. The existing system allowed the customer to select a date, pick a camp type and complete their purchase online. It was built using native PHP from the ground up, the UI had become dated and maintaining the system required uploading and downloading products and order data via CSV. The client would then input their orders into another internal system manually.
The urgency of replacing the system from the ground up was reaching critical. Internally the client was getting close to releasing a new customer relationship management system to streamline management of the business. The biggest challenge for the new customer-facing system would be integrated directly with the new CRM system, and we needed to launch the site shortly after the CRM was launched.
After a UI had been agreed on, functional requirements were defined, and an estimate was produced, we decided to go with Symfony2 as the application framework. Our team had considered Drupal, but it didn't make sense because the requirements for the user interface and the content itself didn't lend themselves well to a content management framework. We needed a Model-View-Controller (MVC) framework. MVC is a pattern that separates an application into three components: the model (object/database schema), the view (the frontend/UI), and the controller (which essentially controls both the model and the view). Each of these components is built to handle specific development aspects of an application.
All of the products, or as we call it “camp data,” exists in Microsoft Dynamics CRM. This data was accessed via an external Microsoft Azure database. We needed to grab the data, pull it into our system, format it, display to the customer, and then (hopefully) take payments.
Simple enough, right?
The CRM data we had access to came in the form of a database view, which is a single table, flat set of data built using a query (often to multiple tables). Essentially, think of it as a huge CSV with 40 columns and 20,000 rows. The database was also hosted on a Microsoft SQL server, and we wanted to use a PHP framework. This led to using a custom PHP plugin called freetds, that allowed PHP to communicate with MSSQL using Doctrine.
Much of what we produce centres around using Drupal to help us create high-quality content management systems and using a set of contributed modules for the central framework. On top of the contributed code, we write tons of custom modules and have vast experience working with APIs. Still, many of the members of our team did have experience with MVC frameworks such as Zend and Cake. So, even if Drupal was our biggest strength, we were confident we could deliver a high-quality solution using the Symfony framework.
After 2 months of development (at breakneck speed), the site was ready for phase 1 launch! It was a MASSIVE user interface improvement on the previous system:
During testing, we'd hit some UX issues related only to speed. This was something we expected, the client was also aware. Early in development, it became clear the response speed of the Azure data wasn't adequate for any kind of enjoyable user experience.
However, we had a very tight timeline with CRM being launched. So we launched the site knowing we’d need to continue development and come up with a solution to the speed issues in parallel to the next phase of releases.
The quickest solution we could come up with was to cache as much of the data returned from CRM as possible. We did this using the Symfony2 built-in cache system. It worked well - we built scripts to “warm” the cache with each deployment, but we couldn't cache everything. We needed to move fast with solutions and phase 2 needed to take priority. Therefore, we put further development of the caching system on hold.
The requirements for this functionality were:
- Construct XML based on the order info
- Transfer the XML to Azure
- Check the status of transfer every few minutes
- Send admin email after each transfer
The Online Registration system allows the customer to book stays for multiple participants as part of a single transaction, in that scenario, XML needs to constructed for each participant and transferred to CRM individually.
Because the transfer needed to be automated chronologically in the background. We decided to use the Symfony2 Command console framework. This allows the creation of PHP scripts executable via command line. We built one script to move successful orders to a database table in Azure to be processed by CRM and another script, to then monitor the status of the order processing. The scripts would be executed every 5 minutes. The order transfer functionality for phase 2 was launched around a month after phase 1.
Development of the system continued over the course of the next 12 months. For the first 6 months, we had very specifically phased deliverables. After that, work continued as maintenance, fixing small issues, and extending functionality.
During this time, performance was a recurring complaint - the caching implementation we had produced solved about 70% of our issues. Our team looked at improving this by adding further layers of caching, but this didn’t give us the complete solution we were looking for, so we put together proposals for complete solutions to the performance issues.
As we the investigated and estimated the scale of work involved to give us the kind of performance we were looking for, it became clear that we’d essentially need to rebuild the entire system because, at the same time as discussing performance issues, the client wanted a complete UI overhaul. This led to using elasticsearch search to Cache the entire catalogue of data from CRM, giving us lightening performance speeds, as well as using an SPA framework for the UI, which gives us the ability to produce an intuitive and enjoyable UX.
To read more about how that project went, take a look at Danny’s blog post about E-Commerce and IMG!