GitHub Copilot preview gives me hope


People have predicted the death of computer programming for as long as I can remember. This has not (yet) happened for a variety of reasons, the most important of which is that programming is as much an art as it is a science or an engineering discipline.

GitHub Copilot, billed as “your AI pair programmer” and currently in a limited technical preview, attempts to help automate programming in a way that goes a little beyond what IntelliSense and others can provide. It is not completely autonomous. You need to declare (type in) your intentions before Copilot can generate meaningful code, as we’ll see, and you also need to supervise Copilot to get it back on track when it inevitably goes wrong.

Copilot is a cloud service with interfaces to Visual Studio Code (running on your own machine or running in the cloud on GitHub code spaces); JetBrains IDEs, such as IntelliJ IDEA; and to Neovim. The cloud service is a code prediction engine powered by OpenAI Codex, a language model trained on billions of lines of public code.

Yes, there has been controversy over Codex and Copilot. Before you start to fret about the potential copyright and privacy violations of Copilot (I’m looking at you, Free Software Foundation), however, you should understand that Codex was formed on publicly available code of a way often seen as fair dealing. within the machine learning community.

You should also understand that Codex is a code synthesizer, not a search engine. The developers of Copilot admit that this may not be the last word on the subject:

… This is a new space, and we want to initiate a discussion with developers on these topics and lead the industry in defining appropriate standards for training AI models.

How the GitHub co-pilot works

According to GitHub, “OpenAI Codex was formed on publicly available source code and natural language, so it understands both programming and human languages. The GitHub Copilot editor extension sends your comments and code to the GitHub Copilot service, which then uses OpenAI Codex to synthesize and suggest individual lines and entire functions. Additionally, the service uses user choices to improve future suggestions.

co-pilot github 01 GitHub

As this diagram shows, GitHub Copilot is a service that uses the OpenAI Codex language model to provide suggestions based on content from the Visual Studio Code editor and a few other editors.

Testing GitHub Copilot on Visual Studio Code

Copilot is currently in a limited technical preview phase. Before you can install it usefully, you must register on the waiting list for the preview program.

Once you have received your welcome email, you can access the GitHub co-pilot extension page on Visual Studio Code Marketplace and install the extension. Then you will need to authorize the extension in Visual Studio Code. The Home Page has a tutorial you can do, starting from point number 2. In this tutorial you create a .JS file, type

   function calculateDaysBetweenDates(begin, end) {

and end up with a fully implemented function deduced from the function name. In the next tutorial on this page, you type a comment to summarize what a function should do, and you end up with a fully implemented function inferred from the comment, even though the function name is too general to be useful.

co-pilot github 02 IDG

The GitHub Copilot extension page on the Visual Studio Code Marketplace. As you can see at the top, I have already installed the extension.

github co-pilot 03 IDG

A screenshot of Visual Studio Code with the GitHub co-driver active. I just started the first tutorial in the documentation, and you can see the “ghost” code suggestion under what I typed, as well as the Copilot contextual control bar. I created the new file as TypeScript rather than JavaScript, mainly because I’m screwed up. In this case, Copilot still generated JavaScript code.

GitHub co-pilot skills

In addition to inferring function bodies from the function name and a summary comment, Copilot can take inspiration from other codes in the file you are editing and from variable names. For example, if I type a colon after a variable name in TypeScript, Copilot will attempt to fill in the type. If I type “var test1 =”, Copilot will rely on the word “test” and generate an executable test for the previous function. If I type multiple lines that form a repeating pattern, Copilot will try to generate more examples of the same pattern.

Copilot works with a wide variety of frameworks and languages. It works best with Python, JavaScript, TypeScript, Ruby, Go, and more recently Java, with the C family of languages ​​(C, C ++, and C #) slated for the future. I’ve heard from others that it works great with popular JavaScript frameworks like React.

GitHub expects Copilot to consider the current code project more than the current file for its context in the future.

co-pilot github 04 IDG

I generated lines 8 and 9 by typing in the beginnings of the lines and a bunch of tabs. I typed line 10 and the beginning of line 11, and Copilot ended line 11. I was working in TypeScript; it was after compiling in JavaScript, that I ran under Node.js as you can see at the bottom of the screen. Note the incorrectly generated comments on the expected result values ​​in lines 8 and 9.

GitHub co-pilot limitations

First of all, Copilot doesn’t always generate good code. It does not always generate the correct code. Worse yet, it doesn’t always generate executable code. (I encountered all three cases in my testing.)

You absolutely must review the code generated by Copilot. Treat it as if it was written by a green intern programmer who is good with Google searches but needs close supervision.

One way to avoid accepting the first excerpt offered by Copilot is to use the “Open Copilot” option in its context menu, or use the Ctrl-Enter key combination, to display the Copilot suggestions window. in a separate tab. Take a look at the 10 suggested solutions and accept the one that comes closest to what you really want. With that done, you might want to edit the generated code a bit to improve its robustness.

GitHub benchmarked Copilot code generation:

We recently compared a set of Python functions that have good test coverage in open source repositories. We’ve hidden the function bodies and asked GitHub Copilot to perform them. The model did this correctly 43% of the time on the first try and 57% of the time when allowed 10 attempts. And it’s getting smarter and smarter.

Obviously, 43% accuracy is not a very good (or even acceptable) accuracy score for production use, although it is an impressive achievement for new generation technology. coded. Nonetheless, if you are a good code reviewer, you can modify the code generated by Copilot to be correct and robust much faster than you could write it yourself from scratch, especially if you are working with it. a library or framework that is new to you.

github co-pilot 05 IDG

The right tab displays 10 suggested snippets for the body of the function. You can accept whichever is closest to what you want.

Examples of GitHub co-pilots

There were about 25 small examples of Copilot code generation on its home page, and four larger examples with accompanying screen videos in the Co-pilot gallery when I watched November 5, 2021. It’s likely the Copilot team will release more examples in a wider variety of programming languages ​​over time. By the way, it is worth watching the animations in the examples on the homepage, as well as downloading and watching the MP4 videos from the gallery.

github co-pilot 06 IDG

GitHub Copilot example for sentiment analysis in Python, following the Gallery. I typed parts of about six lines, plus a lot of tabs to accept the code. I also rejected several suggestions, including test sentences that I considered negative generated for the positive_sentences list. The code didn’t execute until I installed the Python Requests package on my machine with pip3.

Overall, GitHub Copilot is somewhat useful in its current tech preview development phase. His current performance gives me hope that he will become even more of a time saver in the future. Whether or not the planned commercial Copilot product will be worth buying if and when it comes out is an open question that will depend not only on its evolved performance, but on your own skill set and role.

There are several products that claim to compete with GitHub Copilot. The most promising of these seems to be Tabnine, from a company of the same name in Tel Aviv. Tabnine looks like IntelliSense on steroids and can optionally train on your own body of code as well as open source code. A few of the other alternatives are basically looking for the relevant code in StackOverflow, which makes me somewhat wary of their methodology.

It’s definitely worth trying GitHub Copilot in your own environment and tracking its progress over time.

Copyright © 2021 IDG Communications, Inc.


Leave A Reply