Building an enterprise analytics platform from scratch

Built a production Spotfire deployment and supporting environments from scratch supporting approximately 600 research analysts, account managers, and executives across the organization, as well as public-facing reporting. Conducted regular architecture reviews and load testing, designed caching strategies, and led ongoing Spotfire training sessions.

The organization relied primarily on Excel spreadsheets and a collection of disparate visualization tools. Data was shared through network drives and deep folder structures, making it difficult to discover, share, and maintain analytics across the organization. Core data lived in Salesforce and several on-premises SQL Server databases, with access fragmented across systems and often gated behind separate approval processes.

I worked closely with the organization's IT team to build out the Spotfire platform and supporting environments, including Spotfire Server, Web Player, Automation Services, TERR/S+, Python, and a data virtualization layer that provided a common point of access to otherwise disparate data sources. I conducted regular architecture reviews and load testing as usage grew, and designed caching strategies to keep frequently used analytics responsive. Caching was an integral part of the solution, since we couldn't simply hammer away at the Salesforce API every time someone needed data.

Getting the technology in place was only half the problem; we also needed people to actually use it. To drive adoption, I conducted Spotfire Analyst training for students ranging from "I've got some experience in Excel" to "I've built analytics applications with Tableau," and for roles ranging from Research Analyst to CxO. I also acted as inside sales, presented analytics to the organization's executives and constituents, and worked with and advised internal IT on software and infrastructure. This occasionally gave me the pleasure of helping break down and pack up after events—which I genuinely enjoyed; I'm the kind of nut who likes organizing wires.

A multi-node Spotfire deployment centered on Spotfire Server, with dedicated supporting nodes for Web Player, TERR/S+, Python, and Automation Services. Spotfire Data Virtualization (SDV)—known at various points as Composite Studio and TIBCO Advanced Data Services—provided a common data layer across Salesforce and on-premises SQL Server sources, with caching used to manage API constraints and performance.

The information architecture relied heavily on SDV for caching, access management, and row-level security. We worked closely with Salesforce and TIBCO's adapter team, resulting in several enhancements that were incorporated into the product.

On a personal note, this remains one of my favorite engagements. I had the privilege of living in Europe for four years, meeting people from all over the world, and traveling with the organization to its events. I got to experience those places while learning an enormous amount about the world and international organizations—and, for much of it, feeling completely over my head and loving it. I'd only joined TIBCO the previous year. What a time to be in my 20s!

The challenges were many, beginning with the fact that I was so green! I had to quickly learn the etiquette of international organizations (when you point, use your whole hand with your palm open; never a finger), get a whole new wardrobe, and learn to speak French!
Caching was our biggest technical challenge, however, by far. We spent a great deal of time testing and tuning the architecture to keep data as fresh and dynamic as possible without hammering the Salesforce API. We had to balance freshness against performance and API constraints while still respecting access controls and row-level security. We ultimately relied heavily on SDV's caching layer, carefully deciding what could be cached, how frequently it should be refreshed, and where requests needed to remain dynamic.

My pièce de résistance was a document I created that captured essentially every detail of the implementation: architecture and configuration, training notes, load testing, lessons learned, and best practices. It was eventually adopted within Spotfire as the baseline for conducting architecture reviews with other customers.