β² 897 βΌ Oh... shit. Oops (lemmy.zip) submitted 1 day ago by Mistiygirl@lemmy.zip to c/linuxmemes@lemmy.world 104 comments fedilink hide all child comments
[β] bobzrkr@sh.itjust.works 67 points 1 day ago (6 children) In AWS S3, they're objects, not files. But you upload files to objects. And can download an object to a file. But they're not files. Trust me bro. permalink fedilink source parent hideshow 12 child comments replies: [β] agentTeiko@piefed.social 6 points 1 day ago There is a reason for that since S3 is not block storage and doesn't have a Hierarchal filesystem. That's why when you create folder in the S3 console what you are really creating is a prefix meaning its just a placeholder to auto add the prefix string to the beginning of all objects. But all objects in the bucket are flat when it comes to the API. Plus the way the files are stored in the shards in S3. You can limit Throughput by treating prefixes like a hierarchical filesystem too much. permalink fedilink source parent [β] AmyAye@nord.pub 26 points 1 day ago (2 children) You know, the same deal kind of applies about "losing newcomers". Because I avooded using AWS because everything seemed cryptic and weird with its terminology. So other companies got my business. permalink fedilink source parent hideshow 4 child comments replies: [β] zarkanian@sh.itjust.works 13 points 1 day ago Cryptic and weird and their documentation is the worst. permalink fedilink source parent [β] testaccount789@sh.itjust.works 14 points 1 day ago avooded Awoooo! permalink fedilink source parent [β] irotsoma@piefed.blahaj.zone 3 points 1 day ago The more confusing thing I've had to explain to too many developers is that it's not a directory regardless of what it looks like on the AWS website view. It's a "prefix" meaning the entire thing, including the slashes, is just part of the filename, there is no real structure inside a bucket. So there's no such thing as relative paths to search for things. When you call the list objects function with a prefix it's the same as doing a "starts with" on a string. You can't give it a partial "path" relative to another object that appears to be in another "folder". permalink fedilink source parent [β] ViatorOmnium@piefed.social 7 points 1 day ago (1 child) That's mostly because it doesn't have real directories, it just simulates them with string prefixes and if you forget that on large buckets you are going to have a bad time. permalink fedilink source parent hideshow 2 child comments replies: [β] merc@sh.itjust.works 1 point 1 day ago (1 child) What makes that different from a "real directory"? permalink fedilink source parent hideshow 2 child comments replies: [β] ViatorOmnium@piefed.social 1 point 19 hours ago Imagine you have dir1/dir2 with "directories" inside, each one with millions of "files"/objects. When you try to "ls" the dir1/dir2 by querying Prefix='dir1/dir2/' and Delimiter='/' S3 will still scan all the millions of objects starting with the prefix, and then filter out the ones that have a / after the prefix. permalink fedilink source parent [β] Viking_Hippie@lemmy.dbzer0.com 3 points 1 day ago In Magpiper, they're Blarts. permalink fedilink source parent [+] atx_aquarian@lemmy.world 1 point 1 day ago [deleted] permalink fedilink source parent
[β] agentTeiko@piefed.social 6 points 1 day ago There is a reason for that since S3 is not block storage and doesn't have a Hierarchal filesystem. That's why when you create folder in the S3 console what you are really creating is a prefix meaning its just a placeholder to auto add the prefix string to the beginning of all objects. But all objects in the bucket are flat when it comes to the API. Plus the way the files are stored in the shards in S3. You can limit Throughput by treating prefixes like a hierarchical filesystem too much. permalink fedilink source parent
[β] AmyAye@nord.pub 26 points 1 day ago (2 children) You know, the same deal kind of applies about "losing newcomers". Because I avooded using AWS because everything seemed cryptic and weird with its terminology. So other companies got my business. permalink fedilink source parent hideshow 4 child comments replies: [β] zarkanian@sh.itjust.works 13 points 1 day ago Cryptic and weird and their documentation is the worst. permalink fedilink source parent [β] testaccount789@sh.itjust.works 14 points 1 day ago avooded Awoooo! permalink fedilink source parent
[β] zarkanian@sh.itjust.works 13 points 1 day ago Cryptic and weird and their documentation is the worst. permalink fedilink source parent
[β] testaccount789@sh.itjust.works 14 points 1 day ago avooded Awoooo! permalink fedilink source parent
[β] irotsoma@piefed.blahaj.zone 3 points 1 day ago The more confusing thing I've had to explain to too many developers is that it's not a directory regardless of what it looks like on the AWS website view. It's a "prefix" meaning the entire thing, including the slashes, is just part of the filename, there is no real structure inside a bucket. So there's no such thing as relative paths to search for things. When you call the list objects function with a prefix it's the same as doing a "starts with" on a string. You can't give it a partial "path" relative to another object that appears to be in another "folder". permalink fedilink source parent
[β] ViatorOmnium@piefed.social 7 points 1 day ago (1 child) That's mostly because it doesn't have real directories, it just simulates them with string prefixes and if you forget that on large buckets you are going to have a bad time. permalink fedilink source parent hideshow 2 child comments replies: [β] merc@sh.itjust.works 1 point 1 day ago (1 child) What makes that different from a "real directory"? permalink fedilink source parent hideshow 2 child comments replies: [β] ViatorOmnium@piefed.social 1 point 19 hours ago Imagine you have dir1/dir2 with "directories" inside, each one with millions of "files"/objects. When you try to "ls" the dir1/dir2 by querying Prefix='dir1/dir2/' and Delimiter='/' S3 will still scan all the millions of objects starting with the prefix, and then filter out the ones that have a / after the prefix. permalink fedilink source parent
[β] merc@sh.itjust.works 1 point 1 day ago (1 child) What makes that different from a "real directory"? permalink fedilink source parent hideshow 2 child comments replies: [β] ViatorOmnium@piefed.social 1 point 19 hours ago Imagine you have dir1/dir2 with "directories" inside, each one with millions of "files"/objects. When you try to "ls" the dir1/dir2 by querying Prefix='dir1/dir2/' and Delimiter='/' S3 will still scan all the millions of objects starting with the prefix, and then filter out the ones that have a / after the prefix. permalink fedilink source parent
[β] ViatorOmnium@piefed.social 1 point 19 hours ago Imagine you have dir1/dir2 with "directories" inside, each one with millions of "files"/objects. When you try to "ls" the dir1/dir2 by querying Prefix='dir1/dir2/' and Delimiter='/' S3 will still scan all the millions of objects starting with the prefix, and then filter out the ones that have a / after the prefix. permalink fedilink source parent
[β] Viking_Hippie@lemmy.dbzer0.com 3 points 1 day ago In Magpiper, they're Blarts. permalink fedilink source parent