Best practices for handling temporary video URLs in a web application?

Hi everyone,

I work on a browser-based video utility and am reviewing how we handle user-submitted media URLs. I’m not sharing the product link here; I’m mainly looking for technical advice.

Some media URLs expire quickly, redirect several times or return different formats depending on the request headers. What would be the best architecture for handling this securely and reliably?

I’m particularly interested in:

Preventing SSRF and unsafe URL requests
Validating redirects and file types
Handling rate limits and temporary URLs
Processing large files without consuming excessive memory
Using queues for concurrent download requests
Deleting temporary files after processing

Would you process the file through the application server, use a separate worker service or stream it directly to the user?

Thanks for any suggestions.

As all of these are valid concerns that might require a considerable amount of computing power, I'd suggest building a service based on multiple queue-based workers. Suggested architecture is Browser -> Web API (URL validation, pushing to queue) -> Download worker -> Processing worker if needed -> Object storage/Direct streaming (depending on the case). In such a case user doesn't need to wait for long processing pipeline to complete, to get HTTP response, each component can be scaled independently, depending on it's workload, backpressure during traffic spikes can be applied.

When it comes to preventing SSRF, redirect validation, you can check this checklist by OWASP.

When downloading files, it's worth employing content sniffing instead of just relying on the file name and extension.

When it comes to rate limits and temporary URL it's worth keeping track of all the download jobs, say in some sort of object storage and to have a limited set of retries for each job.

Regarding processing large files, streaming is generally preferable to downloading entire file and keeping it in RAM.

When it comes to streaming file directly to user this mithg be a preferable option if no transformation or file analysis is performed as in such a case, it can reduce latency and save up the storage. However, in such a case, it is still worth employing SSRF validation.

1 Reply

Know the answer? Post it — somebody with the same question will find it here.

Bohdan Stupak1mo ago
As all of these are valid concerns that might require a considerable amount of computing power, I'd suggest building a service based on multiple queue-based workers. Suggested architecture is Browser -> Web API (URL validation, pushing to queue) -> Download worker -> Processing worker if needed -> Object storage/Direct streaming (depending on the case). In such a case user doesn't need to wait for long processing pipeline to complete, to get HTTP response, each component can be scaled independently, depending on it's workload, backpressure during traffic spikes can be applied.
When it comes to preventing SSRF, redirect validation, you can check this checklist by OWASP.
When downloading files, it's worth employing content sniffing instead of just relying on the file name and extension.
When it comes to rate limits and temporary URL it's worth keeping track of all the download jobs, say in some sort of object storage and to have a limited set of retries for each job.
Regarding processing large files, streaming is generally preferable to downloading entire file and keeping it in RAM.
When it comes to streaming file directly to user this mithg be a preferable option if no transformation or file analysis is performed as in such a case, it can reduce latency and save up the storage. However, in such a case, it is still worth employing SSRF validation.