Day 171: Slaying the beast

Today was definitely one of the most challenging coding days I’ve had.

I worked on the problem I described yesterday: How do you reliably parse a page when the DOM elements are constantly switching out from under you?

I spent 10 hours straight trying to come up with a working solution and failed time after time.

If I were writing a one-time script just for Twitter, I could come up with a solution really easily: Find something unique about the item (ID, the URL, etc.) to determine what I’m looking at and ignore it if it comes up again.

But that doesn’t work for a general purpose solution. What if someone wants to process a list that does have duplicates?

I thought about giving up several times — I don’t know of other sites that do the same weird trick that Twitter uses where they limit the number of DOM elements, and I wasn’t sure that all this effort was worth it.

Part of the reason why this is so hard is because I refuse to settle for a solution where someone has to use WAIT commands. If you’ve ever automated things using something like Selenium, you know what I’m talking about. You don’t know when things are going to finish, so you add an instruction to wait for 300ms and hope for the best. It works, but only half the time, so you increase it to 500ms. It still fails once out of every 10 tries, so you bump it up to a second just to be safe.

That’s an awful developer experience, and I don’t want to create something like that. I’ve been building Browserflow using an event-driven architecture so that there’s absolutely no arbitrary “wait half a second here and hope that this other thing finished” sort of logic in the codebase.

Going this route makes things significantly harder. For example, consider the case of scrolling down a page and clicking on a button. It’s easy for us humans because we know when the button has shown up on the screen, but there are always unexpected considerations when trying to automate things like this. You write a script to click the button. But what if the button is not even visible on the page? Okay, so you add a check to see if the button is visible, and if it’s not, scroll it onto the screen and then click it. But when you try clicking it, you find that the command to scroll the page was issued but that the page hasn’t scrolled yet, so the click does nothing.

Automating websites is synonymous with race conditions like this, and it’s why most people succumb to the temptation of throwing in a “wait for a second after scrolling before clicking” and call it a day.

Browserflow is going to be different. When someone writes a flow using Browserflow, I want it to work 100 out of 100 runs. No flakiness. No random waiting.

After many hours of wrestling with the Twitter conundrum above, I finally got it working. I had to revert a ton of code that I had spent hours writing earlier in the day, but the solution is elegant, dependable, and fast. The whole day, I felt like I was dying (this also has to do with the fact that I went trail running with my housemates yesterday and my allergies came back in full force), and it didn’t help that no matter what I tried, I couldn’t get past the infinite race conditions involved in getting a selector, scrolling, and having that selector point to the wrong thing.

But I freaking did it, and I’m incredibly proud of myself. I still don’t know whether it’ll have any use outside of interacting with Twitter, but it works beautifully and I am pleased. The beast has been slain, and I can finally move on.