A year before we shipped the DeepSource integration for VS Code, we were working on something called - Manual Issue fixing. The idea was that besides Autofixing code quality issues, users should be able to manually fix the issues for which an Autofix does not exist without having to go separately to their code editor or their code hosting dashboard.

It is necessary to note here that DeepSource’s Autofix feature was released in late 2020 and the initial version did not use LLMs. Engineers in the team manually defined what possible fixes can be for a comparatively smaller set of issues (think of a paradigm where our javascript analyzer can detect ~700 issues, while Autofix exists for only ~100 of them). Manually defining fixes wouldn’t have let us achieve parity between issues that can be Autofixed and total issues that are detected by an Analyzer anytime soon. To achieve parity between these 2 sets, we prototyped an approach to enable Manual issue fixing, spiced up with LLMs using an approach called Suggest a fix (We did not want to use the Autofix brand for an experimental feature - and also Autofix 1.0 applied deterministic code patches, while LLM outputs are anything but deterministic).

As we were crafting prompts and playing with the results and prototypes, the starting line to work on the VS code extension was also coming closer, and alongside, code generating LLMs were getting even better (Imagine going from Codex to GPT-3 to CodeLlama to Claude to GPT-4 to CodeLlama 2). After working on manual issue fixing for four weeks, we decided to shelve it in favour of taking LLM-powered code fixes directly to where our users worked - the code editor. Some parts of the manual issue fixing feature were transitioned to building Bulk Autofix.

But we took the learnings we had from this project, and applied them first-hand to the DeepSource extension for VS Code. Learn more about the extension here 👇

<aside> 🎊 DeepSource for VS code ↗

</aside>

Workflow to run Autofix inside VS Code 👇

https://vimeo.com/852976425

<aside> 👉 You can read an in-depth account of how you can take the extension for a spin here ↗

</aside>

The rest of this post is more of a story of the things that did not make it to the final version, the issues we ran into, and a small walkthrough of why we took certain decisions.

When we started building the extension, we proceeded with 2 key restrictions -

1. The analysis wouldn’t happen on-device. We would be streaming the code from the editor to our scanning containers in the cloud, and stream back the results. Because of this architectural restriction, we knew analysis runs on the VS code extension would be extremely slow (much slower than other linters that run on-device in a split second)
2. There would be a difference in the state of code getting analyzed on the cloud vs the code that is getting analyzed from the editor. So we have to establish a baseline where the latest scan from the cloud provides the initial set of issues, and the state of the code in the editor would be compared against that.

To address 2, our initial prototype had a pretty big section dedicated to informing users about which baseline (with commit ID and branch) the code in the editor is being compared against.