Description
I noticed while working on the most recent newsletter that the size of the repo is starting to get a bit unwieldy - a fresh clone is 800MB (half of which is the index, half of which is the actual files).
Not only does this make it more of a pain for people to contribute, there is a hard 1GB limit on published GitHub Pages sites which we're eventually going to hit. The index wouldn't factor into that though (I hope...), so I don't think we're in imminent danger.
In general, Git is just not a good solution for storing lots of binary content, and I think we'll need to deal with this eventually if the newsletter is going to stick around long term (which I hope it does!)
Some potential fixes could be:
Fix | Pros | Cons |
---|---|---|
Stop accepting GIFs (they take up the majority of the repo's space) | Only a policy change, not a technical one. | Would only slow down the issue, not fix it. |
Move to a different hosting solution with a higher size cap (Netlify?) | Wouldn't need changes to the site or to the workflow. | Doesn't solve the repo size issue long term. |
Host images externally (maybe on a CDN?) | Wouldn't need changes to the site, minimal change to the workflow. | Might make it more difficult for people to submit images. |
Move away from Git-based hosting altogether | Might be easier to edit the site through a CMS instead of markdown. | Would require us to rethink the whole contribution workflow. |
Something else I've not thought of? | ??? | ??? |
Don't think we need to deal with this immediately, but just wanted to start the conversation sooner rather than later 😄
Activity
ozkriff commentedon May 14, 2021
Another option is to stop accepting GIFs but start accepting video files and limit the image and video sized drastically. The downsides are that we need to somehow process all the published newsletters and GitHub still doesn't support videos in Markdown preview.
17cupsofcoffee commentedon May 14, 2021
Some stats on what's taking up the space:
(the
.pack
is the index)ozkriff commentedon May 14, 2021
Another theoretical option is to use git LFS (don't know if it will help with GitHub Pages limitations) but GitHub LFS pricing doesn't fit for open source projects at all :(
AngelOnFira commentedon May 14, 2021
re: Videos instead of gifs, we did this recently with Veloren's blog on Zola, and it has made it significantly easier to show off more content with smaller filesize. I think this would be a good move even outside of this particular issue.
re: GitHub LFS, I would not recommend this path. We've look at it for Veloren, and when we asked if GitHub would sponsor us with more bandwidth, they recommended we just go back to checking files into the repo instead.
re: GitHub pages limit, it also seems like they have some soft limits that we could potentially break through. 100 GB bandwidth isn't that much. I don't think we track traffic, but that might come up in the future.
I think netlify is a pretty good path, and we can probably retroactivly go back and do a batch conversion of Gifs -> MP4 on the repo. Also, would it be possible to see if the Rust foundation has infrastructure that we could provision on? With Veloren, we have a dedicated server in Germany that is about 60 Euro a month, and it has full gigabit without data caps. I would imagine that the foundation has some servers that could just be used as a CDN pretty nicely.
17cupsofcoffee commentedon May 14, 2021
Sounds like a win-win then :)
I already use Netlify for all of my personal/Tetra stuff, and I much prefer it to GitHub Pages personally. They also have a 100GB a month bandwidth limit, but no deploy size limit as far as I've been able to tell. Think the only downside to this would be that we'd need a seperate login rather than being able to manage it all via GitHub (you can't add team members on a free account, annoyingly).
I think there's a few other services that offer similar 'instant deploy from Git' functionality, I'll look into those.
👍 though we may have to do some playing around with the Git history if we want to free up the associated space from the index: https://docs.github.com/en/github/managing-large-files/removing-files-from-a-repositorys-history
I don't think this would be a huge problem, as I don't think people keep long-lived checkouts of the repo, but it would require a force push to
source
.17cupsofcoffee commentedon May 14, 2021
Ah, one other thing we'd have to figure out to switch to videos would be how we embed them - there's not a standard syntax for embedding videos into Markdown, so we'd potentially have to use a Zola shortcode? Not sure how Veloren has solved this, will have to take a look.
I think if we do go that route we'd need to really clearly flag it up in the co-ordination issue.
AngelOnFira commentedon May 14, 2021
We've done this before when we moved Veloren's assets to LFS, and there are some pretty good tools out there to rewrite history like this 👍
Ya, we use Zola shortcodes for videos. It looks like this:
and can be used like:
I think the youtube one is a different shortcode.
17cupsofcoffee commentedon Sep 6, 2021
A bit of lunchtime hacking - running this in the root of the repo converts all GIFs to (hopefully) web-optimized MP4s:
Still need to figure out the best way of batch replacing image tags in Markdown with video tags or shortcodes - I feel like I'd need to pull in a Markdown parser to do that, since they're all formatted differently/some have alt text/some are inside links.
17cupsofcoffee commentedon Dec 7, 2021
Updated script with find-and-replace for the markdown files:
Script
I've tested this out locally with a few CSS tweaks, and it seems to work well (and vastly reduces the page download size, as the videos don't get preloaded):
I think the remaining questions before we could continue with this would be:
ozkriff commentedon Dec 7, 2021
yeah, losing GitHub previews still sounds like a really big downside to me :(
LPGhatguy commentedon Apr 4, 2023
Video embeds launched on GitHub in May, 2021, though you need to use the web editor, and they show up in the markdown file as just... the video's URL.
I would love support for videos! I made a 3 second gameplay video for this month's post. As an MP4, it's a megabyte at 720p30 and 500 KB at 360p30, but it doesn't seem like I can turn it into a intelligible GIF without making it gigantic. 😢
caspark commentedon May 30, 2024
FWIW I have been happy with (free) Cloudflare Pages for hosting my Zola site - there's a 20mb limit on individual files, but that's not a problem in this context. I tested that R2 is usable for storing & serving videos too (& has a free tier), but I haven't done the plumbing for my site to integrate it properly yet, so can't say I've used that portion in anger.
1 remaining item