#python
6 posts

Making Python programs blazingly fast

Find what parts of your code are slowing down the program. A simple & lazy solution is to use a Unix time command. You can also use cProfile for detailed profiling.
Read more

Making Python programs blazingly fast

  • Find what parts of your code are slowing down the program. A simple & lazy solution is to use a Unix time command. You can also use cProfile for detailed profiling.
  • Once bottlenecks are identified, time the slow function without measuring the rest of the code.
  • The most obvious way of making it faster is to use built-in data types.
  • Caching or memoization with lru_cache will help improve functions that perform expensive I/O operations or some fairly slow recursive functions.
  • You can improve performance, by using seemingly unnecessary assignments like local variables.
  • You can also speed up your code just by wrapping the whole code in the main function and calling it once.
  • Avoid or limit using dot operators (.) as they trigger dictionary lookup using getattribute, which creates extra overhead in your code.
  • Operations on strings like modulus (%s) or .format() can get quite slow when running in a loop. Go for f-string instead which is the most readable, concise and fastest method.

Full post here, 5 mins read

Common security gotchas in Python and how to avoid them

Prevent input injections (SQL or command injections) by sanitizing input using utilities that come with your web framework, avoid constructing SQL queries manually, and use shlex module to escape input correctly.
Read more

Common security gotchas in Python and how to avoid them

  • Prevent input injections (SQL or command injections) by sanitizing input using utilities that come with your web framework, avoid constructing SQL queries manually, and use shlex module to escape input correctly.
  • Avoid relying on assert statements except when communicating with other developers (such as in unit tests or to guard against incorrect API usage) because in the production environment it is common to run with optimisations and Python will skip the assert statements.
  • Python’s import system is very flexible, and installing third-party packages exposes security holes. You also need to consider the dependencies of your dependencies. So vet your packages: look at PyUp.io, check package signatures, use virtual environments for all apps, and ensure your global site package is as clean as possible.
  • Rather than the very powerful yaml.load, use yaml.safe_load.
  • Python can have overrun or overflow vulnerabilities related to memory allocation, so always patch your runtime, even with the latest version.

Full post here, 7 mins read

Python code optimization tips for developers

Optimize the slow code first. In the case of Python, PyPy helps you use less space and work faster than CPython’s typical bulk allows for.
Read more

Python code optimization tips for developers

  • Optimize the slow code first. In the case of Python, PyPy helps you use less space and work faster than CPython’s typical bulk allows for.
  • Profile codes (using CProfile or PyCallGraph, say) to analyze how they work in different situations and estimate the time taken.
  • Python strings tend to be immutable and slow. Concatenate them with the .join() method rather than relying on the memory-hungry (+) operator alone.
  • Use list comprehension rather than loops for faster coding and execution.
  • For memory optimization, prefer xrange over the range function to speed up the creation of integer lists.

Full post here, 4 mins read

How and why we switched from Erlang to Python

Mixpanel had coded one of their servers using Erlang. After 2 years, it became hard for them to debug downtime & performance issues. They switched to their de-facto language, Python. Having more code clarity & maintainability were the two main reasons for this move.
Read more

How and why we switched from Erlang to Python

  • Mixpanel had coded one of their servers using Erlang because of performance requirements.
  • After 2 years, it became hard for them to debug downtime & performance issues because they didn't have any Erlang experts on their team.
  • They switched to their de-facto language, Python. Having more code clarity & maintainability were the two main reasons for this move.
  • For the framework and networking library to scale, Mixpanel used eventlet’s raw WSGI library (instead of Python’s asynchronous I/O) since its”green threads” resemble Erlang’s “actors”.
  • For the JSON library they used simplejson coded in C, for roughly a ten times better performance.
  • Using the right Python libraries avoided adding more servers horizontally.

Full post here, 5 mins read

Dropbox’s journey to type checking 4 million lines of Python

A type checker finds many subtle (and not so subtle) bugs. Refactoring is much easier Type checking provides quick feedback and allows to iterate faster.
Read more

Dropbox’s journey to type checking 4 million lines of Python

Full original post

This post shares in detail the Dropbox team’s rationale behind migrating their code to static type checking; challenges the team faced through the process & how they navigated those; various approaches they used for adoption internally; and the impact it made on the engineering the team’s productivity. A few benefits that convinced the team to go full throttle with this:

  • A type checker finds many subtle (and not so subtle) bugs
  • Refactoring is much easier
  • Type checking provides quick feedback and allows to iterate faster.
  • There is no need to write fragile, hard-to-maintain unit tests that mock and patch the world to get quick feedback.
  • IDEs and editors such as PyCharm and Visual Studio Code take advantage of type annotations to provide code completion, to highlight errors, and to support better go to definition functionality

It is a great case study of the usefulness of static type checking in Python for large-scale projects.

20 mins read