Topic: [Feature Request] Dynamic WEBM file size limits

Posted under Site Bug Reports & Feature Requests

SCTH

Member

Currently, there is a flat 100MB cap on video sizes.
This is rather restrictive. There's a rather large amount of videos that have had to be compressed to fit here and have significantly better off-site versions.

This limit exists partially to prevent upload of videos that are absurdly large for what they contain. For instance, a lossless conversion from MP4 to WEBM will be absolutely huge for the content. However, it does not do this well - short but extremely badly compressed videos slip through, while preventing proper archival of longer or higher resolution videos.

This proposal is to instead limit WEBM file size by contents, in particular the number of frames and the resolution. This would allow for preventing uploading video far too large for what it is, while allowing uploading high-quality videos that otherwise could not be archived without destroying the quality.

This is doable - the site already gets upload statistics to determine duration and resolution, which could be reused for such a system.

There would still have to be an upper limit of course, and the actual details would be up for discussion. My first thought is simply a linear relation between the total number of displayed pixels and allowed file size, along with lower and upper bounds.

Donovan DMC

Former Staff

I'd like to mention with e6 using cloudflare, each tier has a hard limit on upload sizes
The free/pro tiers (which I believe is likely used by e6) only allow up to 100MB (pro is $25/month or $240/year
The business tier, which is $250/month or $2400/year allows up to 200MB
The enterprise tier which requires getting a quote allows 500MB by default

I in no way believe e6 is paying $2400 a year or more for cloudflare

SCTH

Member

donovan_dmc said:
I'd like to mention with e6 using cloudflare, each tier has a hard limit on upload sizes
The free/pro tiers (which I believe is likely used by e6) only allow up to 100MB (pro is $25/month or $240/year
The business tier, which is $250/month or $2400/year allows up to 200MB
The enterprise tier which requires getting a quote allows 500MB by default

I in no way believe e6 is paying $2400 a year or more for cloudflare

Maybe, but this can definitely be worked around. If it would be accepted other than actually coding it, I can do that.

Donovan DMC

Former Staff

scth said:
Maybe, but this can definitely be worked around. If it would be accepted other than actually coding it, I can do that.

The only "workaround" is bypassing cloudflare, which requires either infrastructure changes or very specific routing of an unproxied domain, and also in both circumstances opens up an attack surface for unmitigated DDoS attacks

SCTH

Member

donovan_dmc said:
The only "workaround" is bypassing cloudflare, which requires either infrastructure changes or very specific routing of an unproxied domain, and also in both circumstances opens up an attack surface for unmitigated DDoS attacks

My first idea would be splitting the file into multiple parts then combining server side.

Donovan DMC

Former Staff

scth said:
My first idea would be splitting the file into multiple parts then combining server side.

Ignoring that this all has to be done in the browser in memory, and presuming the tab doesn't get OOM killed for using hundreds of megabytes of memory while looking like it's doing nothing, this just opens the doors for more timing/network issues, and is definitely not worth the hassle

SCTH

Member

donovan_dmc said:
Ignoring that this all has to be done in the browser in memory, and presuming the tab doesn't get OOM killed for using hundreds of megabytes of memory while looking like it's doing nothing, this just opens the doors for more timing/network issues, and is definitrly.not worth the hassle

Not worth the hassle? Archiving the best available version is a pretty important part of e621's goal.

It also doesn't necessarily have to be done in memory, since there's an API for slicing files.

I'm confident that it is a solvable problem, and that I could code it. Perhaps not for all browsers necessarily, but uploading such large files would be a relatively niche use and requiring a proper desktop browser might be fine.