Python is such a popular language for good reason: Its principles are strong. However, if Python is “the second-best language for everything”… that means the first-best is often chosen instead. Oh no! How can Pythonistas survive a project or workplace without our favorite language?
Personally, even though I love Python, I don’t use it daily at my full time job. Nevertheless, Pythonic thinking guides my whole approach to software. I will talk about how the things that make Python great can be applied to non-Python places in three primary ways:
Principles from the Zen of Python
Projects that partially use Python
People who build strong, healthy community
Check out my talk, Surviving Without Python, from PyOhio 2019! It was one of the most meaningful talks I’ve ever given.
Software frameworks are great because they apply the principle of Separation of Concerns. A framework’s tools and code handle a specific need in a standard way for developers to write other code more easily. For example:
Web frameworks support receiving requests and sending responses.
Test frameworks include test case structure, runners, and reporting mechanisms.
Logging frameworks control how messages are gathered and stored.
Dependency injection frameworks create and manage object instances.
Recently, a question hit me: How far should a framework go to separate concerns? Should a framework try to do everything all-in-one, or should it behave more like a library that focuses on doing one thing well?
Let’s look at Python Web frameworks as an example. Django, the “Web framework for perfectionists with deadlines,” provides everything a developer could want out of the box. Flask, on the other hand, is a “microframework” that prides itself on minimalism: any extras must be handled by extensions or other packages. The differences between the two become clear when comparing some of their features:
Feature
Django
Flask
HTTP Requests and Routing
Included
Werkzeug (bundled)
Templates
Included
Jinja2 (bundled)
Forms
Included
None (Flask-WTF)
Object-Relational Mapping (ORM)
Included
None (SQLAlchemy)
Security
Included
None (Flask-Security)
Localization
Included
None (Flask-Babel)
Admin Interface
Included
None
Clearly, Django is all-in-one, while Flask is piece-by-piece. To make a serious Flask app, developers must pull in many extra pieces. There are many other frameworks with similar competitions:
JavaScript testing: Jasmine vs. Mocha
JavaScript development: Angular vs. React
Java BDD testing: Serenity vs. Cucumber-JVM
I think each approach has its merits. All-in-one frameworks are more convenient to use, especially for beginners. For developers who are new to a domain or just need to get something up fast, all-in-ones are the better choice. They come with all units already integrated together, and they often have good documentation. However, developing an all-in-one framework takes much more work because it covers multiple concerns. Developers may also feel shoehorned into the framework’s way of doing things. All-in-ones typically dictate what they believe to be the “best” solution.
Piece-by-piece frameworks require more expertise but offer greater flexibility. Developers can pick and choose the pieces they need, and they can change the packages used by the solution more easily. Found a better ORM? Not a problem. Need to localize the site in Chinese? Add it! Solutions can avoid excess weight and stay nimble for the future. The big challenge is successful integration. Furthermore, a library or framework for a singular concern tends to solve the concern in better ways simply because project contributors give it exclusive focus. The more I learn about a space, the more I lean towards a piece-by-piece approach.
As always, pick frameworks based on the needs at hand. For example, I like to use Django to make websites for my wife’s small businesses because the admin interface is just so convenient for her, even though I could get away with Flask. However, I’ll probably pick Mocha (piece-by-piece) over Jasmine (all-in-one) whenever I return to JavaScript testing.
The Python community, like many groups, has its own language – and I don’t mean just Python. There are many words and phrases thrown around that may confuse people new to Python. I originally shared some terms in my article, Which Version of Python Should I Use?, but below are some more of those colloquialisms for quick reference:
I am a Software Engineer in Test (SET). Many people don’t know quite what that means, though. Developers frequently refer to me as a “tester” or “QA,” and a former director once thought I did DevOps. While my work covers these areas, they aren’t the main focus. Let’s clarify what it means to be a SET.
What is a Software Engineer in Test?
A “Software Engineer in Test” (a.k.a. “Software Development Engineer in Test”) is a software developer who develops software for testing: tools, frameworks, and automated tests. SETs focus primarily on automation for running tests quickly and repeatedly. Test automation is a software product: just as front-end developers write web pages and back-end developers write microservices, SETs write automated tests. The same practices and coding skills apply. I frequently say that a Software Engineer in Test must have the heart of a developer.
So they just write test scripts?
No. Never say this to a SET. Test automation involves much more than just “writing test scripts.” A serious testing solution requires serious design and effort. The top-level automation of a test case is usually just a small piece. SETs are responsible for:
Collaborating with developers and product owners
Contributing to planning and design
Reviewing product code
Formulating test scenarios
Developing test automation frameworks
Automating test scenarios using the frameworks
Knowing and using design patterns where appropriate
Setting up dashboards for reporting test results in real time
Teaching others good quality and testing practices
Developing tools to assist manual and exploratory testing
Saying that testing is “just scripting” belittles the role of the SET and underestimates the workload it requires.
How did this role start?
Software “tester” and “QA” roles have existed for decades, but the SET role first became distinct in the 2000s when large-scale test automation became both feasible and necessary. According to Wikipedia, Microsoft coined the title “Software Developer Engineer in Test” (SDET) in 2005, and others like Amazon and Apple quickly adopted it. Google coined the name “Software Engineer in Test” for the same type of role. I personally prefer the SET title over the SDET title simply because it is more concise.
How is it different from “QA” or “testing” roles?
To me, the SET role is distinct from the traditional “QA” or “testing” role. Software testers historically focused on manual testing and thus didn’t need strong programming skills. Their fortes were product domain knowledge, intuition, out-of-the-box thinking, test planning, and test system setup. And these certainly are important, indispensable skills! SETs, however, live in both the development and testing worlds. They use developer skills to provide software solutions for testing problems. Automation is much more central to the SET role. These days, almost all “testing” job openings are SET roles; manual-only test roles have been largely deprecated. Nevertheless, there will always be a need for manual testing because there are some problems a human can catch much more easily than a script (or an AI agent).
Personally, I avoid using the titles “QA” and “software tester” for myself because they don’t accurately describe all that I do. I also avoid the title “automation engineer” because, again, it is reductionist. I tackle software testing with the heart of a developer, and I set up test automation solutions from the ground up. I’m proud to be a software engineer who specializes in testing.
As a bonus, check out Test and Code episode 47, in which Brian Okken and I discuss what it means to be a Software Engineer in Test (among other topics).
Git is one of the most popular version control systems (VCS) available, especially thanks to hosting vendors like GitHub. It keeps code safe and shareable. Sometimes, however, certain files should not be shared, like local settings or temporary configs. Git provides a few ways to make sure those files are ignored.
.gitignore
The easiest and most common way to ignore files is to use a gitignore file. Simply create a file named .gitignore in the repository’s root directory. Then, add names and patterns for any files and directories that should not be added to the repository. Use the asterisk (“*”) as a wildcard. For example, “*.class” will ignore all files that have the “.class” extension. Remember to add the .gitignore file to the repository so that it can be shared. As a bonus, Git hosting vendors like GitHub usually provide standard .gitignore templates for popular languages.
Any files covered by the .gitignore file will not be added to the repository. This approach is ideal for local IDE settings like .idea or .vscode, compiler output files like *.class or *.pyc, and test reports. For example, here’s GitHub’s .gitignore template for Java:
.git/info/exclude
As a best practice, .gitignore should be committed to the repository, which means all team members will share the same set of ignored files. However, some files should be ignored locally and not globally. Those files could be added to .gitignore, but large .gitignore files become cryptic and more likely to break other people’s setup. Thankfully, git provides a local-only solution: the .git/info/exclude file (under the repository’s hidden .git directory). Simply open it with a text editor and add new entries using the same file pattern format as .gitignore.
# Append a new file to ignore locally
echo "my_private_file" >> .git/info/exclude
skip-worktree
A .gitignore file prevents a file from being added to a repository, but what about preventing changes from being committed to an existing file? For example, developers may want to safely override settings in a shared config file for local testing. That’s where skip-worktree comes in: it allows a developer to “skip” any local changes made to a given file. Changes will not appear under “git status” and thus will not be committed.
Use the following commands:
# Ignore local changes to an existing file
git update-index --skip-worktree path/to/file
# Stop ignoring local changes
git update-index --no-skip-worktree path/to/file
Warning: The skip-worktree setting applies only to the local repository. It is not applied globally! Each developer will need to run the skip-worktree command in their local repository.
assume-unchanged
Another option for ignoring files is assume-unchanged. Like skip-worktree, it makes Git ignore changes to files. However, whereas skip-worktree assumes that the user intends to change the file, assume-unchanged assumes that the user will not change the file. The intention is different. Large projects using slow file systems may gain significant performance optimizations by marking unused directories as assume-unchanged. This option also works with other update-index options like really-refresh.
Use the following commands:
# Assume a file will be unchanged
git update-index --assume-unchanged path/to/file
# Undo that assumption
git update-index --no-assume-unchanged path/to/file
Again, this setting applies only to the local repository – it is not applied globally.
Comparison Table
Which is the best way to ignore files?
Method
Description
Best Use Cases
Scope
.gitignore file
Prevents files from being added to the repository.
Local settings, compiler output, test results, etc.
Global
.git/info/exclude file
Prevents local files from being added to the repository.
Local settings, compiler output, test results, etc.
Local
skip-worktree setting
Prevents local changes from being committed to an existing file.
Shared files that will have local overwrites, like config files.
Local
assume-unchanged setting
Allows Git to skip files that won’t be changed for performance optimization.
NuGet is a package manager for Microsoft .NET. It installs packages and manages dependencies for .NET projects. It is like Maven (Java) or pip (Python). The NuGet Gallery hosts thousands of popular packages like Json.NET, NUnit, and jQuery. If you develop .NET applications (like in C#), then you probably need to use NuGet.
Installing Packages
The easiest way to use NuGet is through Visual Studio, which includes NuGet features by default. Packages are managed per project. Right-click on a project in Solution Explorer and select “Manage NuGet Packages…” to open the project’s package manager page.
The Browse tab lets you search and install new packages.
The Installed tab shows which packages are installed and can uninstall them.
The Updates tab lets you update packages to their latest versions.
The NuGet Package Manager page for a project in Visual Studio
When packages are installed and updated, NuGet also pulls any dependencies they require. Visual Studio also creates a packages.config file for all dependencies. Then, just build and run!
NuGet Configuration
NuGet can be configured using a NuGet.Config file. This file can be placed under a project directory, a solution directory, or a system-wide location. One of the most common settings is the package sources: NuGet uses the public nuget.org repository by default, but others (like private company repos) can also be added. Check the nuget.config reference online for docs on all options. (Package sources can also be configured through Visual Studio under Tools > NuGet Package Manager > Package Manager Settings.)
NuGet Package Manager Console
Sometimes, it’s helpful to control NuGet directly through the Package Manager Console. From the menu bar: Tools > NuGet Package Manager > Package Manager Console. For example, when packages get messed up, I’ll run “Update-Package -Reinstall” to reinstall everything. (Right-clicking the solution and selecting “Restore NuGet Packages” never seems to work for me.) Check the help command or the official guide for more info.
The NuGet Package Manager Console in Visual Studio
NuGet CLI
The NuGet CLInuget.exe provides the full extent of NuGet features, including the ability to make packages. It is more powerful than the Package Manager Console. It must be installed independently – it does not come with Visual Studio. Check the NuGet CLI reference online for full details. The .NET Core CLIdotnet.exe can also be used for managing packages. See the feature comparison for the differences.
The NuGet CLI
Creating a NuGet Package
A NuGet package is basically a ZIP file with a .nupkg extension. It typically contains an assembly DLL and maybe other related files. Creating a NuGet package is pretty easy:
Install the NuGet CLI.
Create a .nuspec file for the project.
Add appropriate settings to the .nuspec file.
Run the “nuget pack” command to create the .nupkg file.
Publish the .nupkg file to the desired destination.
The .nuspec file can be created by running the “nuget spec” command in the project’s directory. The generated <project-name>.nuspec file will contain replacement tokens that will be substituted with values from the project’s AssemblyInfo when the package is built. Make sure to set AssemblyInfo values appropriately for the substitution. The version is especially important, and the automatic version format may be useful for guaranteeing uniqueness. Be sure to add any packages upon which the project depends as dependencies, too. (The .nuspec file can also be created manually.) Refer to the .nuspec reference for full details.
The standard package creation command is “nuget pack <project-name>.nuspec”. However, if the .nuspec file contains replacement tokens, then use “nuget pack <project-name>.csproj” instead. Once the package is created, it can be published publicly to nuget.org or to a private NuGet feed.
Below is an example .nuspec file with replacement tokens:
There were tons of great talks at PyCon 2018 – more than I could attend in person – that are now available on the PyCon 2018 YouTube channel. This post has links to my favorites. Enjoy!
By the Numbers: Python Community Trends in 2017/2018 (Dmitry Filippov, Ewa Jodlowska) – At the end of 2017, the Python Software Foundation teamed up with JetBrains to conduct an official Python Developers Survey. Data science is taking Python by storm, and Python 3 now has majority adoption. There are tons of other cool statistics, too!
How Netflix does failovers in 7 minutes flat (Amjith Ramanujam) – That speed at that scale is mind-blowing. This is a fascinating talk, even for non-engineers!
Solve Your Problem With Sloppy Python (Larry Hastings) – “If you ever start writing a shell script, delete it and write a Python script instead.” This talk is a jovial reminder that Python is a powerful tool, even for hack-n-slash jobs.
The AST and Me (Emily Morehouse-Valcarcel) – Emily gives a great overview of the inner workings of the Python language. This talk is a must-see for anyone into compiler theory.
Dataclasses: The code generator to end all code generators (Raymond Hettinger) – Dataclasses are new data structures to Python to generate classes based on specs.
Pipenv: the Future of Python Dependency Management (Kenneth Reitz) – Pipenv is a new tool that makes pip, Pipfile, and virtualenv easier to use together. Kenneth gives a good overview of Python packaging and why pipenv is awesome.
Type-checked Python in the real world (Carl Meyer) – Sometimes, I wish Python had static typing. Now, it can! Facebook has done some innovative things to make it possible.
Beyond Unit Tests: Taking Your Testing to the Next Level (Hillel Wayne) – Property tests + contracts = integration tests. Hillel gives a fantastic strategy for making tests smarter.
“WHAT IS THIS MESS?” – Writing tests for pre-existing code bases (Justin Crown) – This is a pragmatic guide to adding new tests to old code. Now, you’ll never procrastinate on your tests again!
Demystifying the Patch Function (Lisa Roach) – Mocking is a great practice for limiting scope in unit tests. Whether using unittest.mock or other patching/mocking packages, this is a great talk for learning why and how to do mocking in unit tests!
Automating Code Quality (Kyle Knapp) – One of Python’s most beloved traits is its elegance. Maintaining high standards of code quality can be challenging for large projects, though. Kyle shows how to use existing tools to drive higher quality.
Keynote (Ying Li) – [35:07] – Ying is a software security engineer at Docker. In her keynote, she urged all people involved in technology to learn the basics of security. Definitely watch the video recording – she used a fun story to illustrate her points.
Keynote: The People and Python (Qumisha Goss) – [1:07:35] – “Q” is a librarian at the Detroit Public Library who taught herself Python so she could teach coding classes to kids. She shared the highs and lows of her experiences, especially in light of many disadvantages her students had. My favorite takeaway was to “cultivate greatness in others.”
