Python Packaging

Table of Contents

• 1  Setup
  • 1.1  Virtual Environment
  • 1.2  Integrated Development Environment (IDE)
• 2  PEP 8: Style Guide for Python Code
  • 2.1  Code Linter
  • 2.2  Formatter
• 3  PEP 257: Docstring Conventions
  • 3.1  Code Linter
• 4  PEP 483: Type Annotation
  • 4.1  Code Linter
• 5  Code testing
  • 5.1  Measure execution time
  • 5.2  The Python Profilers
  • 5.3  Unit testing framework
• 6  Python Package
  • 6.1  main.py
  • 6.2  my_package
  • 6.3  __init__.py
  • 6.4  _module*.py
  • 6.5  setup.py
• 7  Documentation
  • 7.1  doc
  • 7.2  mkdocs.yml

Setup¶

This study note is based on:

• Udemy course: https://www.udemy.com/course/python-coding-guidelines-tooling-testing-and-packaging/
• Github Repository: https://github.com/franneck94/UdemyPythonProEng
• Python Enhancement Proposals (PEP): https://www.python.org/dev/peps/
• Packaging for Python: https://packaging.python.org/guides/packaging-binary-extensions/

Virtual Environment¶

Concepts:

• Virtual environment: an isolated Python environment where a project's dependencies are installed in a different directory from those installed in the system's default Python path and other virtual environments.
• Dependency managers: tools that enable easy management of a project's dependencies.

(Optional) Let [project_name]='pyUdemy'. At terminal, choose one of the environment managements:

• virtualenv:
  • Download: % pip3 install virtualenv
  • Create: % virtualenv [project_name]
  • Activiate: % [project_name]/Scripts/activated.bat
• Anaconda (suggested) with Dependency managers:
  • Download: https://www.anaconda.com/downloads
  • Create: % conda create -n [project_name] Python=3.8
  • Activiate: % conda activate [project_name]
  • Install: % pip install -r requirements.txt
  • check install: % pip list
  • exit: % exit

Integrated Development Environment (IDE)¶

• Visual Studio (Suggested): sufficient for package development
  • Download: https://code.visualstudio.com/
• PyCharm
  • Download: https://www.jetbrains.com/pycharm/

PEP 8: Style Guide for Python Code¶

• Import order: % isort mymodule.py
  • Future library imports.
  • Standard library imports.
  • Third party imports.
  • First party imports.
  • Local application/library specific imports.
• Indentation:
  • Use 4 spaces per indentation level.
  • Continuation lines should align wrapped elements either vertically
• Comments
  • comments in English
  • Comments should be complete sentences
  • Two spaces after a sentence-ending period in multi- sentence comments, except after the final sentence
• Naming Convention

Element	Naming style	Example
pacakge	lowercase	pacakgename
interface	CapitalizedWords	InterfaceName
class	CapitalizedWords, CapWords, CamelCase	ClassName
function	lower_case_with_underscore, snake_case, (sometimes mixedCase)	function_name, (functionName)
variable	lower_case_with_underscore, snake_case, (sometimes mixedCase)	variable_name, (variableName)
method	lower_case_with_underscore, snake_case, (sometimes mixedCase)	method_name, methodName
utility	mixedCase	utilityName
constant	UPPER_CASE_WITH_UNDERSCORE	CONSTANT_1

Code Linter¶

• pylint (suggested): check code error and PEP8
  • Install: %pip3 install pylint
  • Single file checking:% pylint mymodule.py
  • Recursively checking: % pylint .

Pylint category	Description	VS Code category mapping
Convention (C)	Programming standard violation	Information (green underline)
Refactor (R)	Bad code smell	Hint (light bulbs)
Warning (W)	Python-specific	problems Warning
Error (E)	Likely code bugs	Error (red underline)
Fatal (F)	An error prevented further Pylint processing	Error

• flake8: PEP8-only
  • Install: % pip3 install flake8
  • Single file checking:% flake8 mymodule.py
  • Recursively checking: % flake8 .

Flake8 category	Description	VS Code category mapping
F	An error prevented further Flake8 processing	Error
E	Likely code bugs	Error
W	Python-specific	Warning

Formatter¶

• autopep8 (suggested): check and correct to conform to the PEP 8 style guide (based on Flake8)

  • Install: % pip3 install autopep8
  • preview difference: % autopep8 mymodule.py --diff
  • Single file checking: % autopep8 mymodule.py
  • Recursively formatting: % autopep8 .

• black: not able to setup config file

  • Install: % pip3 install black
  • preview difference: % black mymodule.py --diff
  • Single file checking: % black mymodule.py
  • Recursively formatting: % black .

PEP 257: Docstring Conventions¶

• VS Code Extension: Autodocstring avaliable for the following conventions
  • Google (suggested)
  • NumPy (suggested)
  • ReST/sphinx

Code Linter¶

• pydocstyle: checking by convention
  • Install %pip3 install pydocstyle
  • Single file checking:% pydocstyle --convention=google mymodule.py
  • Recursively checking: % pydocstyle --convention=google .

PEP 483: Type Annotation¶

The function below takes and returns a string and is annotated as follows

def greeting(x: data_type) -> data_type:
    return some_function(x)

def greeting(x: union[data_type, data_type]) -> union[data_type, data_type]:
    return some_function(x)

• Generic concrete collections
  • Dict(dict, MutableMapping[KT, VT]): A generic version of dict. Useful for annotating return types. To annotate arguments it is preferred to use an abstract collection type such as Mapping
  • List(list, MutableSequence[T]): Generic version of list. Useful for annotating return types. To annotate arguments it is preferred to use an abstract collection type such as Sequence or Iterable.
  • Set(set, MutableSet[T]): A generic version of builtins.set. Useful for annotating return types. To annotate arguments it is preferred to use an abstract collection type such as AbstractSet.
• Special type
  • Any: Special type indicating an unconstrained type.
  • NoReturn: Special type indicating that a function never returns.
  • TypeAlias: Special annotation for explicitly declaring a type alias.
• Special forms
  • Tuple: Tuple type; Tuple[X, Y] is the type of a tuple of two items with the first item of type X and the second of type Y. The type of the empty tuple can be written as Tuple[()].
  • Union: Union type; Union[X, Y] is equivalent to X | Y and means either X or Y.
  • Optional: Optional type. Optional[X] is equivalent to X | None (or Union[X, None]).
  • Callable: Callable type; Callable[[int], str] is a function of (int) -> str.
• Collections Abstract Base Classes

Code Linter¶

• mypy: check the compatibility of type for input and return in a function/method
  • Install %pip3 install mypy
  • Single file checking:% mypy mymodule.py
  • Recursively checking: % mypy .

Code testing¶

Measure execution time¶

timeit: time a

• import: import timeit
• run timer for number times exucutions: timeit.Timer(stmt='pass', setup='pass', number=1000000)
• run timer for repeat times setup and repeatxnumber times exucution: timeit.repeat(stmt='pass', setup='pass', repeat=5, number=1000000)

The Python Profilers¶

cProfile

• import: import cProfile
• Profiling function cProfile.run("print('a')")
• profiling function multiple time:

import cProfile
import io
import pstats
import random
import sys
from functools import wraps

from vector import Vector2D


def profile(fn):
    @wraps(fn)
    def profiler(*args, **kwargs):
        profiler = cProfile.Profile()
        profiler.enable()
        fn_result = fn(*args, **kwargs)
        profiler.disable()
        stream = io.StringIO()
        ps = pstats.Stats(profiler, stream=stream).sort_stats('time')
        ps.print_stats()
        print(stream.getvalue())
        return fn_result
    return profiler


@profile
def test_addition_own_implementation():
    for _ in range(100_000):
        # your code being profiled
        v1 = Vector2D(random.randint(-10, 10), random.randint(-10, 10))
        v2 = Vector2D(random.randint(-10, 10), random.randint(-10, 10))
        v3 = v1 + v2  # noqa


def main() -> int:
    test_addition_own_implementation()
    return sys.exit(0)


if __name__ == '__main__':
    main()

Unit testing framework¶

• unittest: test object(variable,function,class) using inherited class of unittest.TestCase.

import unittest

class TestStringMethods(unittest.TestCase):

    def test_upper(self):
        self.assertEqual('foo'.upper(), 'FOO')

    def test_isupper(self):
        self.assertTrue('FOO'.isupper())
        self.assertFalse('Foo'.isupper())

    def test_split(self):
        s = 'hello world'
        self.assertEqual(s.split(), ['hello', 'world'])
        # check that s.split fails when the separator is not a string
        with self.assertRaises(TypeError):
            s.split(2)

if __name__ == '__main__':
    unittest.main()

• pytest: Can run unittest.TestCase at command line.
  • Install: pip3 install pytest
  • run all test_*.py files under current directory: % pytest

Python Package¶

Python Package is a directory structure.

.
└──  PACKAGING 
     ├── my_package  # this is the actual python package
     │   │
     │   │ #############
     │   │ # Main body #
     │   │ #############
     │   ├── module1
     │   │   ├── _module1_a.py
     │   │   │   ├── print_hello_world() 
     │   │   │   └── print_name()
     │   │   ├── _module2_b.py
     │   │   │   ├── vector 
     │   │   │   └── string
     │   │   ├── tests
     │   │   │   ├── test_print.py
     │   │   │   └── __init__.py
     │   │   └── __init__.py
     │   ├── module2
     │   │   ├── _module2_a.py
     │   │   ├── _module2_b.py
     │   │   └── __init__.py
     │   ├── module3.py
     │   ├── __init__.py
     │   │
     │   │ #################
     │   │ # Documentation #
     │   │ #################
     │   ├── doc
     │   │   ├── api.md
     │   │   └── index.md
     │   ├── mkdocs.yml 
     │   └── README.md
     └── setup.py or main.py

main.py¶

• setup.py: outside of the package, the import statemen in this .py file is the entering point to load and execute the package,
  • With __all__ in __init__.py and underscored _module.py as the package modularization , it accesses the functions in ./PACKAGING/my_package/module1/_module1_a.py via module1

from my_package.module1 import print_hello_world, print_name 


def main() -> None:
    print_hello_world()
    print_name('Chen')


if __name__ == '__main__':
    main()

my_package¶

• my_package: A regular package is typically implemented as a directory containing __init__.py files that labels sub-directories as a part of the Python package.

__init__.py¶

• __init__.py: a indicating file
  • When a regular package is imported, this file is implicitly executed, and the objects it defines are bound to names in the package’s namespace.
  • By defining __all__ in __init__.py, the main.py is able to access the method under the module folder. e.g.: ./PACKAGING/my_package/module1/init.py

from ._module1_a import print_hello_world, print_name


__all__=[
    "print_hello_world",
    "print_name"
]

_module*.py¶

• _module*.py: as part of the package, this .py files contain classes, functions, or methods.
  • With a underscore behind, it allow main.py access to the function via the its parent directory. e.g.: ./PACKAGING/my_package/module1/_module1_a.py

def print_hello_world() -> None:
    print("Hello world!")


def print_name(name: str) -> None:
    print(f"Hello {name}")

setup.py¶

• setup.py
  • Reference/link to package directory: % python setup.py develop
  • copy the package directory to the /Lib of python environment: % python setup.py install
  • To publish to pypi, we need to define global variable for distribution name, author, package version, license, python version, etc.

from setuptools import setup


CLASSIFIERS = '''\
License :: OSI Approved
Programming Language :: Python :: 3.7 :: 3.8
Topic :: Software Development
Operating System :: Microsoft :: Windows
Operating System :: POSIX
Operating System :: Unix
Operating System :: MacOS
'''

DISTNAME = 'my_package'
AUTHOR = 'Weiquan Luo'
AUTHOR_EMAIL = 'weiquan.luo1221@email.com'
DESCRIPTION = 'This is a my_package.'
LICENSE = 'MIT'
README = 'This is a simple my_package python package.'

VERSION = '0.1.0'
ISRELEASED = False

PYTHON_MIN_VERSION = '3.7'
PYTHON_MAX_VERSION = '3.8'
PYTHON_REQUIRES = f'>={PYTHON_MIN_VERSION}, <={PYTHON_MAX_VERSION}'

INSTALL_REQUIRES = [
    'numpy',
    'scipy'
]

PACKAGES = [
    'fastvector',
    'tests'
]

metadata = dict(
    name=DISTNAME,
    version=VERSION,
    long_description=README,
    packages=PACKAGES,
    python_requires=PYTHON_REQUIRES,
    install_requires=INSTALL_REQUIRES,
    author=AUTHOR,
    author_email=AUTHOR_EMAIL,
    description=DESCRIPTION,
    classifiers=[CLASSIFIERS],
    license=LICENSE
)


def setup_package() -> None:
    setup(**metadata)


if __name__ == '__main__':
    setup_package()

Documentation¶

• mkdocs: After preparing doc and mkdocs.yml,
  • Install: pip3 install mkdocs
  • See avaiable function: % mkdocs --help
  • Create documentataion: % mkdocs build
  • serve at local host: % mkdocs serve
  • after upload to Github, deploy github page: % mkdocs gh-deploy

doc¶

• doc: folder use to create html file

  • api.md: specify what api(class, function, method, etc) should be presented, then retrive the docstring of api to generate html documentation.

    # References
    ## Mehod
    ::: module1.print_name
    ::: module1.print_name
    ## Class
    ::: module2.vector
    ::: module2.string

  • index.md: homepage for the package website.

    # my_package
    
    This is a my_package.

mkdocs.yml¶

• mkdocs.yml: styling file.

site_name: FastVector
site_description: "This is a simple vector python package."

theme:
  name: "material"
  palette:
    primary: "blue"
    accent: "grey"

plugins:
  - search
  - mkdocstrings

markdown_extensions:
  - pymdownx.highlight
  - pymdownx.superfences

nav:
  - "Start": index.md
  - "API": api.md