343
Pandas (mander.xyz)
submitted 2 years ago by to c/science_memes@mander.xyz
36 comments fedilink hide all child comments
 
you are viewing a single comment's thread
view the rest of the comments
[–] 8 points 2 years ago (1 child)

Pandas supports lazy loading and can read files in chunks. Hell, even regular ole Python doesn't need to read the whole file at once with csv

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 3 points 2 years ago*

    I didn't know about lazy loading, that's cool!

    Then I guess that the meme doesn't apply anymore. Though I will state that (from my anedoctal experience) people that can use Panda's most advanced features* are also comfortable with other data processing frameworks (usually more suitable to large datasets**)

    *Anything beyond the standard groupby - apply can be considered advanced, from the placrs I've been

    **I feel the urge to note that 60Mb isn' lt a large dataset by any means, but I believe that's beyond the point

  • permalink
  • fedilink
  • source
  • parent