Day 170: Sailing in space

Coding sometimes feels building a ship, expect during the construction process, I keep on discovering leaks.

I roll up my sleeves and patch the hole, only to find that two more have appeared out of nowhere.

Building software usually doesn’t feel so hard, but in the case of Browserflow, I’m designing a ship to handle any environment. Sure, it can easily sail in easy waters. But can it handle 20-foot waves? What about a hurricane?

Today, I felt great about some refinements I made to a command that lets you loop through all elements with a given selector. It even scrolls for you so that it automatically fetches new elements in infinite feeds!

Check out my sweet baby logging the usernames of everyone I follow on Instagram (I haven’t used Instagram in years but it’s great for testing Browserflow):

I thought I had successfully upgraded my dinky little boat to a luxury yacht, and then, of course, I discovered that the yacht is taking water.

Modern sites can be too fancy now. For example, when you scroll your Twitter feed, the site doesn’t simply add tweets to the bottom of the page like it might appear. Inspect the DOM, and you’ll see that it’s actually only keeping around a handful of elements in a list and throwing away the rest. If you scroll a bunch, it might look like the page contains all the tweets you’ve already seen, but they get purged and only get added back when you scroll back up. (Try pressing Ctrl+F to search for a tweet you know you’ve scrolled past — if you’ve gone too far, it won’t appear.) It’s a clever optimization to avoid bloating the page with tweets that are no longer visible.

Unfortunately for me, this makes my job 100x harder because I can’t rely on a selector to deterministically return the same element. The element that it referred to before can get removed and replaced with a different one as I scroll down and load more tweets.

I still don’t have a good solution, and this scenario is representative of the challenge of building a system to handle any site. After having spent so much time designing my ship to smoothly navigate the open seas, it feels like I’m discovering that my ship also needs to be able to sail in space.

Love programming, but it can be such a pain to do it well!