What's the best way to store the growing amount of data organisations collect on a daily basis?
How do you cope with storing and analysing the broadening range of data and returning it through a broadening range of interfaces? We've been working with Microsoft Azure's Cosmos DB and have put together a few questions to ask yourself in tackling these questions. (Game of Thrones spoilers ahead!).
What is Cosmos DB?
Cosmos DB is an Azure-native database solution that is optimised for global low-latency performance and supports multiple data structures. It has SLA guarantees on performance, for all those time-critical applications, and has a tuneable consistency to get the best performance whether you're building social features into your app, where instant feedback to one user matters, or whether you’re tracking prices in real-time, where the data has to be correct for everyone.
What's in your data?
Azure SQL and other databases are powerful, mature tools, and most existing applications that use such a database should stay as they are. Cosmos DB forces you to think about data slightly differently. Traditional databases treat data as tables, like a spreadsheet, with fixed columns for product name, price and stock level, for example, and then set up queries to find everything under £10. They take time to configure correctly to make sure they're fast, but once configured, they are extremely powerful because the shape of the data is defined.
SQL databases are well structure and work like tables, where each item has the same structure
Cosmos DB and other NoSQL databases are less structured. It's more like a notebook than a spreadsheet. You can choose to put the date on the same place on every page. You can make every page look like a calendar, or you can do something different on each page. This makes it easier to store data that doesn't fit neatly into a table, so the phones in your catalogue can show how many cameras they have, what colour they are, whether they support wireless charging, whilst the smart speakers can show whether they have a screen, if they have a camera, their shape, etc. It's a lot more flexible, but you need a good team to help you create and use that data effectively.
Cosmos DB is great for IoT devices. Imagine if you have 7 thermostat models on the market, and 4 security cameras, and 19 alarms systems with 153 different sensors, all with different capabilities, all sending back different data, but you still want to give your customer one simple dashboard. Cosmos DB allows you to shape your customer data to fit the devices they own, and introduce new models, without needing to change anything in your data layer.
NoSQL databases, like Cosmos DB, allow you to shape your customer data to suit multiple needs
What relationships are in your data?
The shape of the relationships in your data matters too. Traditional relational databases are great at mapping belonging relationships: employees belong to a department, orders and addresses belong to a customer. Most business cases can be devolved into this type of relationship, but it can make it harder to discover the underlying patterns in the data.
NoSQL databases, such as Cosmos DB, are capable of modelling more complex relationships such as graphs (employee works with employee, site is 10 miles from site) or hierarchies. These are traditionally harder to work with and can result in losing valuable information when trying to fit them in to a standard database.
Cosmos DB has no problem helping you figure out how Jon Snow fits into everything
How does your data change?
To get more customers, you'll find yourself needing to support more platforms. Website visitors will want an iOS app and an Android app. Users will want to order on Alexa, Siri and Google Assistant. And you’ll need to think about integration with chat platforms.
As data changes, how do you keep them all in sync with your data? How do you add images for your customers with Retina Displays without breaking things for everyone else? By accommodating a flexible, extensible data model, Cosmos DB minimises the amount of changes you need to make to the data to support new use cases, by moving more of the logic into the application.
More customers means more data: more orders to store, more customer details to save. And more work every time you need to change the database. You don't want to take a global website offline for 6 hours to add a 3D AR image field for every product.
Can you scale?
Congratulations on your millionth customer. How is your website coping?
In a traditional database scenario, you'll likely need more servers, and you'll need to start separating your data. Maybe your Australian customers see a different set of products, so you can create a database just for them, so they don't need to use the slow connections back to your data centre in Dublin. But how do you keep the data correct for all regions?
Data relationships can be geographical as well as hierarchical
In Cosmos DB, you can scale automatically, only paying for usage, so you don't need to pay for Black Friday-ready infrastructure all year round. Cosmos DB can also handle data synchronisation worldwide, so when you add the "buy now, only 5 left" warning, every customer can see accurate, real-time stock information.
Cosmos DB charges by transaction, not data volume. Cosmos DB can scale quickly, so the costs can be lower, but also less predictable if you have spikes in traffic. You should always discuss expected load with your architect to set expectations on how costs between Cosmos DB and alternatives will compare. You should also discuss which data will scale. More customers may mean more orders, but the product catalogue may not be updating every minute of every day.
Is Cosmos DB right for you?
Cosmos DB is not the right solution for everyone. If your data fits neatly into a traditional SQL database, and isn't massive, then that's fine. If you only ever need 1 data centre, or you have access to experts in SQL replication, that's fine too.
Cosmos DB also has lower data integrity controls, so if you need everything to be consistent, all the time, and you need the database to validate its own data, you’ll likely want to stick with a traditional database solution and accept the performance and maintenance costs required for that. Like SQL databases, Cosmos DB can guarantee data security however and a well architected solution can be completely GDPR compliant.
If you want to report on your data, you may find Cosmos DB doesn’t offer all the tools you expect, but there are other tools available that you can use to extract the data so you can view trends and generate reports.
If you have a new application with data that doesn't fit neatly into a table, or you have customers around the world that need up-to-date data, then Cosmos DB might be the solution for you. Get in touch to see how we can help you make the right decision, we'd love to talk.
Craig
Technical Architect