Using Copilot with Python apps

I've been hesitant to try Github Copilot, the "AI pair programmer". Like many developers, I don't want to accidentally use someone's copyrighted code without proper attribution.

Fortunately, Github is continually adding more features to Copilot to make that possibility both rarer and easier to spot. Plus, I'm now on the Cloud advocacy team at Microsoft (Github's parent company), so I keep hearing about the benefits of Copilot from my colleagues. I decided that it was time to try it out! 🤖

I enabled Copilot while developing a Flask + PostgreSQL demo app, and wow, I am a huge fangirl already. 😍 Here's how it helped me out:

Writing ORM queries

My app uses SQLAlchemy, a popular package that's been through a few iterations. I've only used SQLAlchemy a few times, so I often find myself unsure how to form the correct ORM queries. I'm much better at SQL than SQLAlchemy, as it turns out. Fortunately, Copilot has seen enough examples of SQLAlchemy queries that it was able to form them for me.

Copilot wrote the queries after I provided the route function header and variable names. It's worth noting my models.py file already existed at this point.

@bp.route("/surveys/<int:survey_id>", methods=["GET"])
def survey_page(survey_id):
    survey = Survey.query.where(Survey.id == survey_id).first()
    answers = Survey.query.where(Answer.survey == survey_id)

Yes, those are pretty straightforward queries, but it still would have taken me a web search first to remember the SQLAlchemy ORM methods. It also was able to write queries with filters, especially if I wrote the comment first:

# Count matching answers in the database
answer_count = session.query(models.Answer).filter_by(
    selected_option="strawberry").count()

Would I have learned more of the SQLAlchemy API had I written those queries myself? Yes, probably, but 1) I don't know how long that knowledge would have lasted, given I bounce between multiple ORMs across projects, and 2) we're at the point of web development where there are too many APIs in play to memorize, and our time can be spent on gluing together apps.

Of course, we need to make sure these queries work! That brings me to my favorite use of Copilot...

Writing tests

My app uses Pytest to test the routes and models. I started by creating a test_routes.py file with this comment at the top:

# Test the routes in app.py using pytest

Copilot immediately took care of the imports for me. Interestingly, I didn't need pytest imported at first, since it's only necessary if you define fixtures or use other special features, but I did end up writing a few fixtures later.

import pytest

from app import app

Now I wrote the signature for the first test:

def test_index_redirect():

My goal was to test this route, whose code was already written:

@bp.route("/", methods=["GET"])
def index():
    return redirect(url_for("surveys.surveys_list_page"))

Copilot filled in the rest of the code:

with app.test_client() as client:
        resp = client.get("/")
        assert resp.status_code == 302
        assert resp.location == "http://localhost/surveys"

I ran the tests then, and discovered only one issue (which I suspected when I saw the suggested code): the location needed to be a relative URL, just "/surveys". I was very happy to have this test written, as I'm relatively new to Pytest and had already forgetten how to write Pytest tests against a Flask app. If Copilot hadn't written it, then I would have dug up a similar app of mine and adapted those tests.

For my next test, I wrote this function signature and comment:

def test_surveys_create_handler(client):
    # Test the create handler by sending a POST request with form data

Copilot filled in the next line, complete with a fake survey question. That's part of what makes Copilot particularly great for tests, it loves making up fake data. 😆

resp = client.post("/surveys", data={
        "survey_question": "What's your favorite color?",
        "survey_topic": "colors",
        "survey_options": "red\nblue\nyellow"})

For the rest of the tests, my general approach was to write a function signature, write comments for the stages of the test, and let Copilot fill in the rest. You can see many of those comments still, in my test_routes.py file. The only place it flailed was properly setting a cookie in Flask, so that was something I had to research myself. At some point, I refactored the common app.test_client() into a test fixture, since so many tests used it. Copilot may not always be the DRYest! 💦

Reflections

I like Copilot because it really is like a pair programmer, except without the feeling of being watched (which is uncomfortable for me, personally). There's also no judgment. I sometimes will correct a Copilot suggestion, run the tests, and then realize Copilot was actually right. I'm more amused than embarrassed when that happens, since I know Copilot really doesn't care at all.

I also don't feel like Copilot has copied the code of any particular developer out there, in the suggestions that it gave me. What I saw instead is a model that has seen a lot of similar code, and also has seen the code inside my project folder, and it's able to match those patterns together. The suggestions felt like the results of a StackOverflow search, but quicker and personalized.

I think it's interesting that using Copilot really encourages the writing of comments, and I wonder if this will lead to a future where code is more commented, because people leave in the comments they used to prompt the suggestions. I often strip mine out before committing, but not always. I also wonder if comments-first code writing will generally lead to people coding faster, because we will first think through our ideas in an abstracted sense (using English) and then implement them with syntactic constraints (code). I suspect that coding is actually easier when we describe it in our natural language first.

Those are my musings. I'd love to hear about your experimentation and how you think it will affect the future of coding.