Benefits of Our Datasets Feature

Authors
  • Name
    Thomas de Zeeuw
    Github
Published on

In this blogpost we'll go over how Slight datasets provide additional benefits over using plain SQL tables.

If you're not yet familiar with Slight's datasets, they are a source of data that can be accessed by a Slight app. Typically these are SQL tables. which can be used by apps that use SQL. More information about Slight datasets can be found in our dataset documentation.

The most common use for this feature is that it allows you to document tables from your databases within Slight. This ends up looking similar to a data catalog. One immediate advantage is that these documented datasets are in the same tool they're used in, rather than somewhere else.

But this alone doesn't answer the question: why bother using this feature? When e.g. making apps in Slight, why not just reference the SQL table directly?

Database Migrations

Whenever you run a database migration that makes breaking changes, for example deleting a table or deleting/changing a column, there's always a question of "What is going to stop working?" Is there a SQL script somewhere, owned by someone, that is critical to the company, but is run only once year and is only going to break six months down the line? Slight can help with these kind of migrations.

If you're using Slight apps and datasets it's very easy to get a list of impacted apps. Simply navigate to the (to be) affected dataset, for example by finding it in the dataset catalog (example catalog on slight.run), and open the "Apps Using" tab. This will show a list of all apps using the affected dataset.

List of apps using a dataset

With this simple action you have a list of affected apps and the owners of the apps so you know who to contact. Furthermore this gives quick impression for the scale of the migration.

Dataset Usage

At times it can be useful to understand how often a dataset is accessed. Most database systems provide a way to track which queries are being made and by who. This allows database administrators to keep track of (im)proper use of the data. However, the query results are not always meant for the person running the query. It's quite common for someone more familiar with SQL to run a query for a colleague with less SQL knowledge. This means that we don't have proper insights into where the data ends up.

Slight changes this by making the power of SQL available to everyone in the form of Slight apps. Slight apps can be by anyone (given proper access of course), even by those know don't know how to use SQL. This means that it's no longer necessary to run queries for colleagues as they can do it themselves, resulting in the proper user being recorded in the audit log.

With Slight datasets it's also possible to get a quick insight into the usage of a dataset. Available on the dataset page are the number of times a dataset was viewed and the number of apps that use the dataset. The example below shows that a good number of apps are using the dataset, indicating it's an important one.

Example of dataset usage

Similar information is available for apps; the number of views and the number of times the app was run.

Clear Documentation

In some cases table columns are (re)used in ways that... let's say the original creator didn't intend them to be used. A quick hack and a workaround there and you can end up with a pretty messy database scheme. Although we recommend that you don't end up in this situation, in some cases this is simply the status quo we have to deal with.

Most database systems provide a way to comment on various objects, such as tables and columns, but this can be limited in size and usage. Slight puts documentation front end and center. On nearly all our resource, e.g. apps, graphs, datasets and dataset columns, we support a short form description and long form documentation. This allows you to document all the messy details in place where the users will see it; right on dataset information page.

The example below shows some non-obvious column names with the documentation that clear up any ambiguities.

Example of the documentation of dataset columns

Creating Apps

We can provide a great autocomplete experience using the information stored in the datasets. We can show not only the columns that are available, but also any description included in the dataset:

Autocomplete showing two possible columns, and a description for the highlighted column
If there are multiple datasets in use, for example if a JOIN is used, we can show which dataset each autocompleted column is from.

We've discussed above how you can quickly see what other apps use a given dataset. This allows app creators to build off existing work fast, without having to dig for what's relevant.

And Even More Benefits

  • Quick previews (of actual data!) of all your datasets in a central accessible location.
  • Organization: your tables may have a more naturally grouping than the one specified by database schemas for example. You can also add tags to your datasets, taking this even further.
  • You can give datasets a friendly easy-to-remember name, regardless of the table's name.
  • Hide boring columns that only add noise for dataset users.

If this sounds productive to your team, reach out at team@slight.co or start your free trial, whichever you prefer.

Related Articles