76
Why Python Is So Slow (And What Is Being Done About It) (thenewstack.io)
submitted 2 years ago by to c/python@programming.dev
47 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 4 points 2 years ago (1 child)

Usually, but when it isn't then you've got a bottleneck. Multithreaded performance is a major weak point if you need to do any processing that isn't handled by one of the libraries.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 2 years ago* (1 child)

    Then you need to break up your problem into processes. Python doesn't really do multi-threading (hopefully that changes with the GIL going away), but most things can scale reasonably well in a process pool if you manage the worker queue properly (e.g. RabbitMQ works well).

    It's not as good as proper threadimg, but it's a lot simpler and easier to scale horizontally. You can later rewrite certain parts if hosting costs become a larger issue than dev costs.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 3 points 2 years ago (1 child)

    A process pool means extra copying of data around which incurs a huge cost and this is made worse by the tendency for parallel-processing-friendly workloads often consisting of large amounts of data.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 2 years ago

    Yup, which is why you should try to limit the copying by designing your parallel processing algorithm around it. If you can't, you would handle threading with a native library or something and scale vertical instead of horizontal. Or pick a different language if it's a huge part of your app.

    But in a lot of cases, it's reasonable to stick with Python and scale horizontally. That has value if you're otherwise a Python shop.

  • permalink
  • fedilink
  • source
  • parent