🎨 Database review and refactoring the importer

13th January 2021

Database Review

• Prisma created and modified fields - @updatedAt on schema
• Migrations friction - not sure how to do non-destructive migrations and prisma migrate doesn't seem to be well-documented yet. Fair enough, it's pretty hot off the press. It doesn't matter so much yet, and hopefully it's matured by the time it does.

Refactoring the Data Importer

Following that, I started to look at refactoring the import process. In the current version, the logic is split between the client- and server-side, which looks beautiful (we get a nice web UI to follow the progress), but it splits the logic between the admin page and the API, and I suspect it makes the process an order of magnitude slower than it needs to be (the effort linking takes ~20s on Lachlan's data).

An import process ideally belongs fully server-side, to avoid having to create API endpoints just to service that task. In a perfect world, we'd hit a single endpoint and it would stream us the same message and progress data as we receive currently (perhaps via websockets). However, Vercel doesn't support websockets, so that rules that one out.

I'm thinking that I'll give this approach a shot. If it's as fast as I think it'll be, we won't need to stream anything - we can just show a loading indicator, and then receive and display all of the messages as a single response from the server. It wouldn't be quite quite as responsive an experience, but the process itself would be quicker.

Looking forward, it'd be nice to have a npm run command that would do the same thing from the command line. I'll aim to build a library that performs the import, and then make use of it for both the web-based import and the command-line script.