My chaotic journey to find the right database

idk.. syncing was simple for me. Any table you want synced, just add an "updated_at" timestampz, then on front-end when you want to sync, you just request all rows with updates after the "last_update" timestamp that you have locally. Yes, still keeping soft deletes on server, but locally, deleted them when you get the sync showing it deleted. That way local data doesn't keep growing with deletes. Then I also just refresh sync everything for data over 30 days old with updates, that way I can delete "deleted" rows after 30 days and any local client not updates within the last 30 days, will just re-sync it all.

The only issue with this approach but I rarely see it as an issue, users can change the data they have locally, which it won't sync those changes if use doesn't update the "updated_at" column, or they can delete data stored locally and that won't get synced again so it will just be missing. But, that could be worked around if you really needed per row syncing guaranteed. But come on, it's not that important. Sync on load, sync server to client when rows greater than updated_at appear in server, or client to server for the same. so simple. User messes around with their data directly, o-well, they can logout and login to refresh it all.

I wonder if Deno Kv would come in handy here.

Are you trying to convince us that $0.12 * 300 / month is expensive, but quater CPU VM with 512GB for $1500/month is a best deal?

Try rxdb

Also why websocket solutions aren't viable for such problem?

sounds like a skill issue.

Damn. I'm impressed just throwing everything back and forth for sync worked that well for that long.

Dexie deserves more praise than this video does. :) Dexie is what Whatsapp web and chatgpt web use to deliver ultra fast local first experiences. To me, Whatsapp web is the standard. It's really good and works on mobile. The only caveat is I don't understand why you'd load the full chat history in indexedDB when most use cases would be concerned with the 20 most recent chats. I do preload Dexie cache (last 20) + lazy load + Supabase realtime. But I didn't hit scale yet. So we'll see.

As of now, when you hit "chat" it's a near instant experience.

Kudos for memorizing everything that should rerender. When using dexie uselivequery, this is something you should take responsibility for.

+ You're using virtualization! Awesome that you mention that, I have yet to implement it.

i am nostalgic for this look of yours

I can also recommend RxDB (e.g. with dexie)

We did something like this in firebase. We stopped because even like this it was a ton of work

I always thought the way of doing this time of sync is using event sourcing.

This can potentially solve some of the sync problems and paginate a little bit of the information

You should have just used Triplit! Would have solved all of your issues (real time sync, partial sync, authorization, etc) and is all managed in Typescript

Exactly what i been looking for.

have you consider using Event Sourcing for state?

Ferretdb ???

I have done this differently - with the yjs and an indexeddb provider. That being said, it might be too slow for the scale of the data here

i have a question, I hope someone can answer,if the userID is ever exposed on client side, and you already know the threadID isnt it possible for someone to guess a userID if there are enough users to edit another Users chat? I know its quite unlikely but just wanna confirm I understand

this was very interesting

Totally agree that “does the code look cool” is a great metric (not sarcastic; programmer intuition matters).

AI has broken language, I bet if you chart the number of times people on you tube videos say 'dive' it has hockey sticked in the last couple years.

We had nothing but trouble with PlanetScale and moved to Neon and couldn’t be happier. For us, Neon has been amazing and PlanetScale is nothing but nightmare fuel. I am having trouble fathoming the praise for it.

Gun js can solve ur problem of real time sync

What is the problem with Websockets?

this is the type of video I want to watch

Every time he said "sync layer," my brain had to momentarily wonder: "what does some random guy named 'Sinclair' have to do with any of this..?"

What about YJS my g

I’m not a Dev but I like to watch them squirm in pain and question their life choices 😎😂

A lot of this in-client DB logic is what we use in mobile apps to not show as much loaders as possible but the drawback is maintaining two different databases and syncing downstream clients whenever smth is changed upstream

I'm building an app now with a dexie layer that syncs up with the server, I thought that was a great idea but now you're giving me second thoughts lol

Is redis only in memory databae so if it's go down all data is lost ?

Didn't watch video. Answer is postgres, right?

Would SQLite client side and mysql server side using migrations?

99.9999% of apps made today could use sqlite or basically read from a file dump + memory and be fine.

Thanks for mentioning Legend State! Definitely has piqued my interest. Seems to have a pretty low overhead sync solution with a backend compared to other solutions I have come across so far.

Always funny when Theo makes a video about something he has no freaking idea about! Should it not be possible to at least know the proper terms after spending 1 months on a topic?

- "offline first" means "the main workflow works offline "with the latency and resilience benefits that brings
- "local first" means specifically that the app needs to be resilient to the company and servers going away and having realtime and collaborative features, so nothing theo is remotely interested in
- linear is neither of those, it just uses limited amount of offline first concepts to improve UX and apart from that it tries to piggyback on the local first movement without sharing any of the values or other aspects
- caching concepts are enough for the use-cases where you want to mainly view and create something. Local first comes into play when you need TO WORK with the data in an app (eg. sorting emails, managing calendars, editing notes)

Use RxDB

Perhaps I’m unskilled, but if I start a new project I’m going for PostgreSQL or DynamoDB.

I had pretty good luck back in the day with Couchbase. We did something very similar to what you ended up doing here. It was possible to query thousands of keys at once and sending them down as a package.

caring about alpha nerd stuff instead of things your users actually care about... never fall in that dev trap. No one cares that your sidebar state isn't finalized after 0.25 seconds

Excellent video, appreciate all the knowledge share from this migration

MYSQL, I'm only gonna use MYSQL. It can do all the things and you lernt it in school and you can around with the boys.

This video has 1% DB and 99% FE hacks

Why was using websockets for syncing such a bad thing?

Json by key on disk like dynamo and etc instead of redis/mysql should be better. But it also should be almost free with event stream / "big-table" instead of full sync.