Integrate.io Integrate.io
GEOFF HARCOURT
CTO, COMMONLIT
Industry: E-Learning
Location: Washington, US
Company Size: 100+

CHALLENGES

  • We must replicate transactional database information into a secure and sophisticated data warehouse.
  • Our home-built ETL pipeline was causing more headaches than solutions. We needed to break free from this system of instability and explore options that would bring us a reliable solution.
  • Engineering resources were being burned up through constant firefighting - it was time to find a more dependable replication solution and put an end to the drain on these critical assets.

SOLUTION

  • CommonLit implements Integrate.io’s data replication solution, which replicates millions of rows a month with zero issues.
  • Industry-leading tool replicates data quickly and consistently; predictable pricing makes it easy to manage team budgets; and white-glove support ensures zero outages or problems.

RESULTS

  • 700M rows of data replicated in a month
  • 22 days of engineering resources saved each year
  • More time to focus on core engineering priorities

Share This Story

How CommonLit Saves 22 Days of Engineering Resources a Year with Integrate.io

WATCH VIDEO

About

CommonLit is a nonprofit EdTech company that provides a high-quality literacy curriculum to help students graduate high school with the reading and writing skills they need for success in college and their careers.

Use Case

CommonLit’s pipeline for moving data from a Postgres database to Amazon Redshift had lots of issues. Replications constantly broke down and developers had to drop their critical work to debug these problems. In addition, data wasn’t available for critical business intelligence and analytics needs across the company whenever pipelines were down. CommonLit needed a much more stable solution.

Slow, unreliable data replication leads to wasted engineering resources and constant organizational disruption

What CommonLit needed was an easy way to continually replicate data from its transactional database into a data warehouse. This change would allow its analysts and business intelligence experts to get a holistic view of organizational performance and gather analytics needed for better decision-making.

Before they found Integrate.io, CommonLit used a home-grown ETL pipeline stopgap to move their data. The solution was heavily reliant on AWS DMS, Amazon’s Database Migration Service.

Unfortunately, this solution was challenging to configure and almost impossible to maintain. The organization had frequent issues with the replication pipelines breaking, meaning data wasn’t getting to the data warehouse—and ultimately to the people and tools that needed it most.

When replication broke down, an engineer from CommonLit’s lightly resourced team had to drop whatever project they were working on, and race to debug and fix the pipeline.

The task of fixing the pipeline was made even more difficult because Amazon DMS provided almost no data about what had broken and where.

Even more problematic, the sync would often break in such a way that engineers needed to resync the entire data warehouse from scratch. This was a nightmare for both the data team—and the wider business.

With a full resync typically taking 12 hours or more, engineers were pulled away from their daily responsibilities for long periods, losing countless hours and even more in focus switching and opportunity cost losses. As data volumes continued to grow, these critical business disruptions were only going to increase.

“Resyncing so much data had a big impact on our business,” says CommonLit’s Chief Technology Officer (CTO) Geoff Harcourt. “I shudder to think what the impact might have been if we’d continued along that trajectory.”

During periods of extended pipeline maintenance, critical information users needed for analytics and business intelligence was either out of date—or totally inaccessible.

“If the data our teams needed was old or unavailable, we couldn’t answer basic, critical questions required to complete grant applications or carry out other research our C Suite relies upon,” says Geoff.

To cushion the organization from the impact of unreliable replication, Geoff created detailed runbooks for debugging the system and even implemented a whole set of alerts for engineers to jump on out-of-hours fixes. This added more disruption for engineers and created extra costs for off-hours work.

The business had lost total trust in its data replication system.

“We were losing thousands of dollars of engineering time a month to disruptions. With a team as small as ours, that hit was serious and was obstructing our ability to do great ongoing work,” says Geoff.
Geoff Harcourt
Geoff Harcourt
CEO, CommonLit

Rebuilding trust in data replication with Integrate.io

CommonLit urgently needed a more stable alternative for replicating data quickly and efficiently. They had conversations with a number of potential vendors, including Integrate.io, Rivery, and Fivetran.

CommonLit ultimately selected Integrate.io, because it “had the best combination of price, reliability, attention to personalized service, and functionality.”

Another differentiator was Integrate.io’s predictable pricing, which made it simpler for Geoff to control team budgets.

“It was really important to us to have predictability and transparency around pricing,” says Geoff. “With Integrate.io, we pay for the number of rows of data that we move, which makes it easy to figure out how much it will cost on an ongoing basis.”

To ensure Integrate.io achieved the reliability and consistency they needed, CommonLit kicked off with a 14-day free trial. 

“The trial was valuable because it helped us sort out our permissions and the basic structure of our pipelines, along with how we wanted to configure them. It also helped build trust with the Integrate.io team,” explains Geoff.
Geoff Harcourt
Geoff Harcourt
CEO, CommonLit

The process of getting up and running with Integrate.io was fast and disruption-free. They provided user-friendly documentation for CommonLit’s specific use case along with an elevated level of ‘white-glove’ localized support.

“The documentation Integrate.io provided clearly laid out the steps to follow, plus we received some code to run, which got our permissions properly set up, so we could get things moving,” says Geoff.

“Their support team was super responsive. Whenever I sent a message, I’d hear back in 10 minutes maximum, which was really impressive."

With close support from Integrate.io’s skilled data experts, CommonLit got a test pipeline up and running in the same day. They synced their data to see if there was any impact on CommonLit’s production systems and immediately noticed a dramatic improvement in the reliability and consistency of replication. They moved ahead with a full sync of their data on an ongoing basis.

Soon, they were replicating millions of rows a month with zero issues.

“I tend to be a cautious person about how freely I give my trust to vendors, but after two weeks without any disruptions to service, we knew we’d made the right choice,” says Geoff.
Geoff Harcourt
Geoff Harcourt
CEO, CommonLit

One aspect of Integrate.io’s support that Geoff appreciates is how they constantly monitor pipelines, meaning Geoff doesn’t have to worry about missing a small issue that spirals into something bigger.

“Our pipelines are extremely reliable and the Integrate.io team takes immediate action whenever there’s the slightest issue, to ensure that things always move along for us,” he says.

“We haven’t had any outages or problems that have held us back in the entire time that we’ve been on the platform.”

Integrate.io’s ongoing support, long after the initial ‘sale’, continues to blow Geoff away.

“One time, we did a huge data migration on a Saturday night, but it caused an interruption in our replication pipeline,” he says.

“The Integrate.io team immediately responded in the middle of the night and took the necessary steps to keep the sync going. When we got around to Monday morning, nothing was broken and our entire organization was ready to go!”

Reliable, disruption-free replication saves 22 FTE (full-time equivalent) days of engineering resources, and tens of thousands of dollars every year

With Integrate.io, CommonLit replicates as many as 700M rows of data every month, with no issues, and almost as importantly, no disruption to their engineering team.

“Integrate.io has been a very transformational part of how we move data between our data stores,” says Geoff.

The platform’s speed, reliability, and consistency mean engineers no longer have to carry out time-sucking debugging and resyncing.

CommonLit easily saves 15 hours a month of engineers’ time thanks to Integrate.io. That’s 180 hours a year, or over 22 FTE (full-time equivalent) days, from engineers that are recaptured and channeled back into core projects.

“Our engineers are able to work on the projects and obligations they’re assigned to, knowing they won’t have to be dragged off to firefight an ETL pipeline breakdown,” says Geoff.
Geoff Harcourt
Geoff Harcourt
CEO, CommonLit

With Integrate.io, data is now always near real-time, and always accessible, supporting better critical business decisions and organizational growth.

“The data sync is always prompt and our business intelligence users are able to treat data in our warehouse and business intelligence tool as 100% reliable and up to date all the time,” says Geoff.

“Integrate.io’s reliability is critical to our success. The grants that our CEO and COO are pursuing now are higher stakes and larger dollar values than before. It’s absolutely critical that decisions are based on timely data, and Integrate.io ensures we get it right every time, the first time.”

Geoff recommends Integrate.io to any data team looking to replicate data quickly and dependably, so they can extract maximum business value.

“With Integrate.io we have a solution that’s plug-and-play. You set it up, you turn it on, it runs, and it’s supported by a team that’s responsive to any issues that crop up, which is invaluable,” he says.

“I didn’t realize when we started with Integrate.io how much their reliability would be a game changer for us.”

Get ultimate value from your data warehouse today!

Start your free 14-day trial now.

hello@integrate.io
+1-888-884-6405

Ensure Data Quality Ensure Data Quality
Ensure Data Quality
Ensure Data Quality Ensure Data Quality

©2024 Integrate.io

Solutions
Support
Company
Language

©2024 Integrate.io