On setting up Calibre for fanfic
Jan. 20th, 2019 03:16 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
jadislefeuUpdated Jan 2020 to add guide about custom columns for chapters and for multiple series, telling FFF to scrape more kinds of metadata, and adding them all to your title page or an updated-metadata log at the end.
I was asked on Discord for a tutorial about using Calibre for fanfic in general! Please feel welcome to share or link this wherever, because it was more work than I realized it would be when I started and I want it to be useful to people. Also feel free to ask any clarifying questions! I enabled anon comments and everything.
Firstly you want to download Calibre, if you haven't already. In calibre, click on the arrow next to the preferences button [brief edit July 2020: in a fresh install of new version of Calibre, 'Preferences' may be hidden off the top bar in the overflow menu, found by clicking the three dots on the far right side of the bar], and select 'Get plugins to enhance Calibre'. Search uninstalled plugins for FanFicFare (henceforth FFF because I'm lazy) and install it. You'll have to restart Calibre for it to take effect.
Once you have FFF installed, click on the arrow next to its button and select 'Configure FFF'. In the Customize FFF window, I'll walk you through the options I use and what they do (and any of the ones I don't use that I'm familiar with).
You want to select your default output format--mine is epub, because I read in the calibre ebook viewer. If you're going to send your downloaded fics to an ereader (an exercise I cannot help you with, because I've never done it, sorry), you'll want to select whatever is the appropriate format--mobi for kindles, for instance. Next one down is what the default action FFF will take if you ask it to check for updates on an existing story. I have mine set to 'Update EPUB if new chapters', because if it updates the file unconditionally to what's online, I'm vulnerable to losing fic if the author deletes chapters but leaves the story ostensibly on the site. (In cases of an author deleting chapters and reposting revised versions, I tend to delete the URL from the copy I have so that FFF won't register it as an extant duplicate, change the title to append {Archive} or (Old Version), and download the new version as its own separate file, but that's because I'm neurotic. If you're not paranoid about this, you can go for Update EPUB Always.)
I have default update calibre metadata checked, because I want to know about new tags if they're added. I have default update epub cover when updating epub checked, because I have covers generated with the tags and wordcount on them (more on that later), and I want them up to date. I have Default background metadata checked because I have all my usernames and passwords and isadult:true setup in personal.ini for every site I download from, but if you don't, you likely want to leave it unchecked, because sites that require registration or adult confirmation will silently fail with it checked if you don't have them set up in personal.ini (more on personal.ini in the next tab).
Next section of this tab! I do not have delete other existing formats checked, because if I've gone through the effort of generating another format I presumably want it for something, but if you think you're going to be converting into other formats for transfer to another device and want to make sure you only retain converted copies that are up to date, maybe you want to check it. I EMPHATICALLY recommend ticking 'keep existing tags when updating metadata', if you want ANY kind of imposed organization on your fic at all. I have a 'FanFiction' tag added automatically to all fic I download (more on that in the personal.ini tab, again), and I also go through and add some standardized ship tags so I can search for, say, HPDM and find fics with Harry/Draco from both FFN and AO3, which have drastically different ship tag formatting that I can't otherwise easily do in one search. (I suppose I could manually construct and then save a search for every variation of the ship tag instead.) Anyway, if you want to add any kind of categorization yourself, tags you want to search on that you think are missing (I also have an 'Undead Husband(s)' tag, for instance), CHECK THIS BOX OR YOU'LL LOSE WORK AND DATA.
I don't use the series anthology function, so I can't tell you what to do with the next box. I have check for changed story URL checked because I don't want that to be silent, and usually when I get prompts to update the URL it's for overwriting/merging AO3 and FFN copies of fic, which I don't want to do, I like keeping separate instances because what if something is different and, as previously mentioned, I'm neurotic. I don't have search inside ebooks for story URL checked and I'm not positive what the utility of it is--maybe if you have a bunch of fics downloaded with AO3's native downloading that you want to update metadata on, that'll find the URLs for them automatically? Probably useful if you have that kind of thing.
GUI Options section! I have take URLs from clipboard ticked because I have no particular reason not to, which just fills in any valid fic url in your clipboard into the box for entering URLs to download. (More on that later.) I have default to update when books selected, which means that if I select ebooks and hit the main FFF button, it'll pull up the dialog to check them for updates. Else, it'll open the window for URLs for new fic, which I get to with an extra click from the dropdown--up to you which you prefer. I literally just realized there was an option to keep the URLs window on top of other windows and immediately ticked it, because that'll be useful for dragging urls into it from open fic tabs. Show an estimated time left, why not.
Post processing options: I have none of these boxes ticked, and I have calculate word count set to 'yes, always'. I have no idea how to use the 'mark updated books' stuff, can't help you there. I don't trust 'smarten punctuation' to not break shit. If you want things in two formats, you can tick automatically convert new books, I have no real use for it.
I recommend keepting ticked the search by title/author box, because otherwise it will just silently duplicate all of your fics, even if they have the same source URL.
I don't use the reject list and I don't really know what the point of it is.
This is where you want to plug in your username and password for whatever sites to be able to download archive-locked fics. If you're paranoid, you could make an account specifically for this purpose, so you're not risking any data, because this is putting that information into a plaintext file on your computer that could be hacked or whatever I guess. I am lazy, so I mostly just use my regular accounts, though on AO3 and FFN I have accounts specifically for subscribing to everything I download so it updates by email, and I use those instead. You also want to set is_adult:true, or you will have you click 'yes I am an adult' in a confirmation dialog for every single explicit fic you download from any website (like AO3) that has a confirmation for such things and I promise you you do not want to do that, it is annoying as fuck. .
Anyway, your website username/password settings will look like this:
[archiveofourown.org]
username:XXXXXXXX
password:XXXXXXXX
With your actual username/password, obviously. I copied that out of 'view 'safe' personal.ini', which is designed for sharing your settings with other people without risking exposing your data.
The another thing I have in personal.ini is a categorization tag for all fic. Under the '## This section will override anything in the system defaults' section under the usernames section, I have 'extratags_label:FanFiction'. This adds the tag 'FanFiction' to all fic I download, which lets me do things with virtual libraries that I'll talk about later or possibly in another tutorial because this is long as hell already. Regardless, this allows you to separate out fic from anything else you may have in calibre (for instance, I have a bunch of issues of literary magazines that I don't need or want to be looking at when I'm looking for some nice HPLV or whatever.).
Update Jan 2020: I do much more with personal.ini now! I have custom update logs on the ends of fic (which a friend helped me with the setup of, many thanks to lethe), more metadata on the title pages, and most involved-ly, I have my new multiseries column.
Calibre only supports one series in the series field, so fics on AO3 with more than one series applied to them were having important information lost and I was having to open the website to figure out which fics were in continuity with each other, because I follow multiple authors that use organizational as well as continuity-based series, and the ones of less use to me were often the ones getting picked up. So I decided to set up a column that would scrape all series data in, even if it wouldn't work as series tags, so I wouldn't be losing metadata. (Note: technically, it's possible for me to still miss series metadata, because I only set it up for like... ten series? You have to set up each entry manually and I got tired of typing them and I really doubt I'm ever going to near that limit, but just for the record, it does exist.)
The column is added as noted in the Custom Columns subsection.
FFF supports what I'm aiming to do, but it takes adding a lot of things to personal.ini that aren't in the very basic settings it comes with. You have to set it up to recognize and scrape the additional metadata, first--here's the code I use to do that:
The 'extra_titlepage_entries' part are added to, you guessed it, the title page.
And here is my multiseries column, as mentioned above. It takes all of the series entries as defined above and feeds them into a single tag-style field. This includes the numbers, so you can't do
I also want to add these to my logpage, the code for which I did not previously include. Here's the setup for my log page--everything after 'warnings' is an extra-metadata addition.
Of course you can delete any of these that you don't want on your own log page. The 'default' version my friend helped me with ends at 'warnings'. For some reason series entries past 03 don't work, it's a mystery to me.
We're skipping the Reading List tab, because I don't use it and I can't help you there.
Okay, so the way I have this set up is useful for me, but it may not be what you want or what is useful for you. I'm going to run through how I have it set up, if you're interested in emulating it, but if you want a different kind of covers I'm not going to be of much use.
The first thing required here is to download and install the Generate Cover plugin, in the same way as we installed the FFF plugin at the start. The relevant settings in Calibre for this setup are Update Calibre Cover: Always and Generate Calibre Cover: Always. Select the radio button for Plugin Generate Cover rather than Calibre Generate Cover. I don't tick 'generate covers only for new books' because I want them to update. I do tick inject/update the cover inside the epub, because... why not really. In Generate Cover (Plugin) Settings, select your cover design preset for Default, and if you want a different preset for any other site, you can select it--I have one selected that uses a smaller font size for the tags for fics from AO3, because fics from AO3 can have way more tags and I was having issues where the tags would spill over the region for them on the cover and fill the entire cover on top of all the other text and it looked like garbage.
Backtracking, let's go into setting up Generate Cover options. There should be a button for it on your top toolbar, if not see my section on adding buttons to the toolbar. Hit that button and it'll open the dialog for generating a cover on your currently selected fic, which also allows you to customize your saved settings in general. Hit the green + next to Saved Settings to add a settings set, or edit the {Default} entry if you prefer. In the box below it, I just leave the image set to {Default Image}, which is just a blown up version of the calibre logo. It's just a space filler and I don't care what it is, so I've left it, but feel free to pick a different image if you prefer. You could set the image differently in presets for different sites if you wanted?
Anyway, once you have your new or extant settings... set... that you want to work with, go to the fonts tab. Here you can set the font and the size you want for each major element. You probably want the title and author (and series, if applicable) to be largest, and custom to be fairly small, if you're including tags like me, because there are a lot more words in tags, and also you have less need to read them at a glance. For my default settings, mine are set to, in order, 44, 34, 34, and 20. (For AO3 fics, they're 42, 24, 26, and 15.) I have use same font family for all text ticked, but you could make everything different fonts if you so desire. I do not tick auto-reduce font size to fit on one line, because that's going to absolutely ruin tag display. Bad enough on an FFN ebook, but on an AO3 one you might end up with about one pixel per word.
Dimensions tab, I leave the cover size at what I believe was the default of 590x750. I don't tick any of the boxes in size. I set the top and bottom margins to 10, the left and right margins to 0, the text padding to 20, and the image padding to 10. Border widths 0.
Now, the final tab, Contents. You can select items in the 'Field Order' box and use the green up and down arrows to rearange them as desired. You can also untick any that you don't want--if you don't want the placeholder image, for instance, you can untick the image box. I have them in the order of: Title, Author, Series, Image, Custom Text. The next box down, you can customize the custom text. I include word count and tags, which for me is formatted as:
{#wc} words
{tags}
(More on word counting, you guessed it, later.)
Other options, I don't tick the swap author box just because I don't think I've downloaded a single fanfic ever that has the author's name in LN,FN format, and I only use generated covers on fanfic. Series text I have as 'Book {series_index} of {series}', which I think is the default. Metadata you don't need to mess with. Hit the Save button in the bottom left to save your changes to the saved settings set, repeat steps to create any other schemas you desire for other sites, and hit cancel or the X to close out of this window without messing with the fic you have selected at the moment, or OK if you want to generate a cover on it right now. Now, back to FFF!
See, I told you we'd get to word counting. I'm only going to go over word counting setup, because I don't use any of the others. The plugin is called Count Pages, but I don't find pages a useful metric for me because my brain is calibrated to wordcount from all these years of fanfic. Presumably the setup is similar for page counting, I just haven't done it.
Okay, so, time to install another plugin! Go into the same search window, and find and install the Count Pages plugin. Before we go into setting it up, we have another step--we need to set up our custom column to save word counts in. Right click on a column header in the main pane and select the 'Add your own columns' option. Hit the green + to add a new one. In the edit custom column menu, give your column a lookup name--if you want ease of copying my settings, call it wc. Column heading Words (or Wordcount or whatever you desire, but I prefer it shorter so it isn't truncated in display in an ugly way). Column type should be 'Integers', description can be whatever but probably should be along the lines of 'Word Count'. I have nothing entered for 'Format for numbers' because frankly, I don't understand the syntax for how to set it, but if you know python you can probably put commas in or whatever. Hit OK, use the green arrows to move the column to whatever position in the list you prefer, hit apply.
Back to Count Pages, now that we're set up! Hit the arrow next to the button for it and select 'Customize plugin...'. We're in the Statistics tab. In the Word count options section, you should have the option to select your new custom column in the custom column dropdown, the entry for mine looks like '#wc (Words)'. I have 'use ICU algorithm' ticked, but I don't really know what difference it makes. In the Other tab, I have the button default set to 'Estimate word/page counts', because, as with FFF, I prefer to just select my fics and hit a button to update the count if I want to. I have 'Always overwrite an existing word/page count' ticked, because I want it to overwrite FFN and AO3's wordcounts so it's standardized, and I also don't want out of date counts. I have prompt to save counts ticked because it is by default and it says there's a small risk of losing other data if you don't. Hit OK.
Back to the FFF Configure menu and the Count Pages tab there. Tick Word Count. You can tick 'only if word count is missing in FFF metadata' if you don't want to overwrite FFN and AO3's native word counts. (One would equally tick Page Count there if you set that up instead or in addition.)
Standard Columns tab, I don't have any of the 'New Only' boxes ticked because I want it to update if it changes. I have 'Set calibre author URL' ticked, so clicking the author's name in the metadata pane opens their author page on whatever archive. I don't have the anthology boxes ticked, because, again, I don't use anthologies and I can't tell you anything about them.
The Custom Columns tab depends on what custom columns you have set up, so we're going to talk about that now!
There are a lot of types of automatic metadata that you can have filled in, like WIP/Completed status, word count, chapter count, language, and most of the already extant ones like tags and site and author. Much of this is saved in other locations, but you can put in in a column as well.
The custom columns I have are the already mentioned word count, the field for notes to myself on fic that I went over in my original mini tutorial here, and a field in which I mark fics as read or unread. (This is manual, I don't like automatic read/unread because sometimes I open a fic and decide not to read it and close it again, and sometimes I download fics I've already read online and want to mark them as read, or whatever. I can't help you with automatic marking as read.)
We already went over word count and notes, so I'm just going to quickly note that my Read/Unread status column is set up with lookup name readyn, column heading Read, column type Yes/No (this lets me select a green checkmark for read or a red X for unread, or leave it blank as it is set automatically), and description 'Have I read this fic?'.
In the Custom Columns tab, you can select from the dropdown what type of data you want to be filled into the column automatically, if any--for me, Read and Notes are blank, and Words autofills with Words, though this is mildly pointless as it's immediately thereafter overwritten by Count Pages. If you don't want to bother re-counting words on things that already have wordcount metadata, then definitely select Words for your wordcount column. As mentioned, this is also where you would select Chapters for your chapter count column, or Language in a language column, or whatever else you may so desire. I have allow custom_columns_settings from personal.ini to override ticked, though I don't think I have anything in there that affects it--if I add something later, it'll be enabled. I have all the options under 'Special column' blank.
Update Jan 2020: I have added additional custom columns now, and I'm just going to quickly run through my settings for them. One of them is quite simple--chapter count. I have this with lookup name chapters, column header Ch. so it takes up less real estate, and the type is set to Integers. In the Custom Column tab of FFF settings, I can select 'Chapters' from the dropdown menu next to it, and it'll be autofilled on new fics.
The second column I've added is "multiseries", added as a "comma-separated text, like tags" type column. It's a lot more involved. The point of it is to include metadata from multiple series applied to a fic, instead of the only one that calibre supports by default, to avoid losing important information. All of the setup for this column is in personal.ini, so the important thing here is to have the box for 'allow custom_columns_settings from personal.ini to override' ticked. Section on setting up multiseries added in the personal.ini section!
To fill both of these columns on fic I already have downloaded, I have to select it, click to update it with FFF, and select 'update calibre metadata from web site'. Because FFF throttles requests so you don't accidentally DDoS archives, this takes a very long time--I have thousands of fics downloaded, and I ended up doing it in batches of a few hundred at a time over the course of several days, because I didn't want to just not be able to use calibre for three days.
Okay, this is a convenience thing in FFF. If you set this tab up with an email account--it recommends you create a separate account for this for security, and I agree for that and so it's not having to hunt through all your non-subscription notification emails as well. (You can do something with an IMAP folder name and filter them into a folder if you don't have a dedicated email address for it, I think, but I have no experience with at and I don't really understand IMAP and I can't help you.)
I created an acount at gmx.com for this purpose, which means the IMAP server name is imap.gmx.com. I'm not positive this works with gmail, I haven't tried. I configured my main gmail email address to have a filter that collects and forwards all fic subscription notifications to the account I made for FFF, so I don't have to make new accounts at FFN/AO3/etc and re-subscribe to whatever all.
So plug in your username and password for your fic notification email into the appropriate boxes in this tab. I tick 'Mark Emails Read', because FFF only looks at URLs in unread emails, and this keeps it from looking at the same fics over and over again. I have Download from Email immediately ticked, because I feel no need to see a list of URLs and a confirmation dialog.
If you set this up, you can hit the arrow by the FFF button and click 'Get URLs from Email' and it'll check for updates on the url of every fanfic in an email in the specified inbox. (If you filter all AO3 emails, say, this will also pick up the URLs of any fic you leave a comment that is replied to on, and any fic you wrote that gets a comment or kudos notification, for the record.) I find this a much more convenient way to check for updates on WIPs than manually doing so, which I will talk about while I'm on the subject. (Particularly since I don't have anything to say about the Other tab.)
In brief: to download a fic with FFF, click the arrow, click download fics from URLs, paste the URLs you want to download into the box or drag them in. You can also do 'get URLs from web page' if you have a page with a list of them, like an author's works page--this has drawbacks on both FFN and AO3, unfortunately. On FFN, it will also scrape all of the author's favorites, which you'll have to find the boundary between the lists and delete those if you don't want them. On AO3, things only come in pages of twenty. I'll tend to use it to scrape the URLs, copy the list out into notepad, and then repeat the process for each page of an author's works/bookmarks/etc that I want to work with. (You can use FFF to download your bookmarks like this, though it will not register private bookmarks and you'll have to drag those in manually. It might see them if you have your login information entered but I can't remember.)
To download or update fics from the email setup above, just hit 'get URLs from email' and confirm on all dialogs.
To update fics you already have downloaded. select them in the main calibre pane and either hit the main FFF button (if you have the same options as me ticked for that) or the arrow next to it and then 'update existing fanfiction books'.
Enjoy your fic reading!
Please let me know if anything is missing or unclear :)
I was asked on Discord for a tutorial about using Calibre for fanfic in general! Please feel welcome to share or link this wherever, because it was more work than I realized it would be when I started and I want it to be useful to people. Also feel free to ask any clarifying questions! I enabled anon comments and everything.
Firstly you want to download Calibre, if you haven't already. In calibre, click on the arrow next to the preferences button [brief edit July 2020: in a fresh install of new version of Calibre, 'Preferences' may be hidden off the top bar in the overflow menu, found by clicking the three dots on the far right side of the bar], and select 'Get plugins to enhance Calibre'. Search uninstalled plugins for FanFicFare (henceforth FFF because I'm lazy) and install it. You'll have to restart Calibre for it to take effect.
Once you have FFF installed, click on the arrow next to its button and select 'Configure FFF'. In the Customize FFF window, I'll walk you through the options I use and what they do (and any of the ones I don't use that I'm familiar with).
Basic tab
You want to select your default output format--mine is epub, because I read in the calibre ebook viewer. If you're going to send your downloaded fics to an ereader (an exercise I cannot help you with, because I've never done it, sorry), you'll want to select whatever is the appropriate format--mobi for kindles, for instance. Next one down is what the default action FFF will take if you ask it to check for updates on an existing story. I have mine set to 'Update EPUB if new chapters', because if it updates the file unconditionally to what's online, I'm vulnerable to losing fic if the author deletes chapters but leaves the story ostensibly on the site. (In cases of an author deleting chapters and reposting revised versions, I tend to delete the URL from the copy I have so that FFF won't register it as an extant duplicate, change the title to append {Archive} or (Old Version), and download the new version as its own separate file, but that's because I'm neurotic. If you're not paranoid about this, you can go for Update EPUB Always.)
I have default update calibre metadata checked, because I want to know about new tags if they're added. I have default update epub cover when updating epub checked, because I have covers generated with the tags and wordcount on them (more on that later), and I want them up to date. I have Default background metadata checked because I have all my usernames and passwords and isadult:true setup in personal.ini for every site I download from, but if you don't, you likely want to leave it unchecked, because sites that require registration or adult confirmation will silently fail with it checked if you don't have them set up in personal.ini (more on personal.ini in the next tab).
Next section of this tab! I do not have delete other existing formats checked, because if I've gone through the effort of generating another format I presumably want it for something, but if you think you're going to be converting into other formats for transfer to another device and want to make sure you only retain converted copies that are up to date, maybe you want to check it. I EMPHATICALLY recommend ticking 'keep existing tags when updating metadata', if you want ANY kind of imposed organization on your fic at all. I have a 'FanFiction' tag added automatically to all fic I download (more on that in the personal.ini tab, again), and I also go through and add some standardized ship tags so I can search for, say, HPDM and find fics with Harry/Draco from both FFN and AO3, which have drastically different ship tag formatting that I can't otherwise easily do in one search. (I suppose I could manually construct and then save a search for every variation of the ship tag instead.) Anyway, if you want to add any kind of categorization yourself, tags you want to search on that you think are missing (I also have an 'Undead Husband(s)' tag, for instance), CHECK THIS BOX OR YOU'LL LOSE WORK AND DATA.
I don't use the series anthology function, so I can't tell you what to do with the next box. I have check for changed story URL checked because I don't want that to be silent, and usually when I get prompts to update the URL it's for overwriting/merging AO3 and FFN copies of fic, which I don't want to do, I like keeping separate instances because what if something is different and, as previously mentioned, I'm neurotic. I don't have search inside ebooks for story URL checked and I'm not positive what the utility of it is--maybe if you have a bunch of fics downloaded with AO3's native downloading that you want to update metadata on, that'll find the URLs for them automatically? Probably useful if you have that kind of thing.
GUI Options section! I have take URLs from clipboard ticked because I have no particular reason not to, which just fills in any valid fic url in your clipboard into the box for entering URLs to download. (More on that later.) I have default to update when books selected, which means that if I select ebooks and hit the main FFF button, it'll pull up the dialog to check them for updates. Else, it'll open the window for URLs for new fic, which I get to with an extra click from the dropdown--up to you which you prefer. I literally just realized there was an option to keep the URLs window on top of other windows and immediately ticked it, because that'll be useful for dragging urls into it from open fic tabs. Show an estimated time left, why not.
Post processing options: I have none of these boxes ticked, and I have calculate word count set to 'yes, always'. I have no idea how to use the 'mark updated books' stuff, can't help you there. I don't trust 'smarten punctuation' to not break shit. If you want things in two formats, you can tick automatically convert new books, I have no real use for it.
I recommend keepting ticked the search by title/author box, because otherwise it will just silently duplicate all of your fics, even if they have the same source URL.
I don't use the reject list and I don't really know what the point of it is.
personal.ini tab
This is where you want to plug in your username and password for whatever sites to be able to download archive-locked fics. If you're paranoid, you could make an account specifically for this purpose, so you're not risking any data, because this is putting that information into a plaintext file on your computer that could be hacked or whatever I guess. I am lazy, so I mostly just use my regular accounts, though on AO3 and FFN I have accounts specifically for subscribing to everything I download so it updates by email, and I use those instead. You also want to set is_adult:true, or you will have you click 'yes I am an adult' in a confirmation dialog for every single explicit fic you download from any website (like AO3) that has a confirmation for such things and I promise you you do not want to do that, it is annoying as fuck. .
Anyway, your website username/password settings will look like this:
[archiveofourown.org]
username:XXXXXXXX
password:XXXXXXXX
With your actual username/password, obviously. I copied that out of 'view 'safe' personal.ini', which is designed for sharing your settings with other people without risking exposing your data.
The another thing I have in personal.ini is a categorization tag for all fic. Under the '## This section will override anything in the system defaults' section under the usernames section, I have 'extratags_label:FanFiction'. This adds the tag 'FanFiction' to all fic I download, which lets me do things with virtual libraries that I'll talk about later or possibly in another tutorial because this is long as hell already. Regardless, this allows you to separate out fic from anything else you may have in calibre (for instance, I have a bunch of issues of literary magazines that I don't need or want to be looking at when I'm looking for some nice HPLV or whatever.).
Update Jan 2020: I do much more with personal.ini now! I have custom update logs on the ends of fic (which a friend helped me with the setup of, many thanks to lethe), more metadata on the title pages, and most involved-ly, I have my new multiseries column.
Calibre only supports one series in the series field, so fics on AO3 with more than one series applied to them were having important information lost and I was having to open the website to figure out which fics were in continuity with each other, because I follow multiple authors that use organizational as well as continuity-based series, and the ones of less use to me were often the ones getting picked up. So I decided to set up a column that would scrape all series data in, even if it wouldn't work as series tags, so I wouldn't be losing metadata. (Note: technically, it's possible for me to still miss series metadata, because I only set it up for like... ten series? You have to set up each entry manually and I got tired of typing them and I really doubt I'm ever going to near that limit, but just for the record, it does exist.)
The column is added as noted in the Custom Columns subsection.
FFF supports what I'm aiming to do, but it takes adding a lot of things to personal.ini that aren't in the very basic settings it comes with. You have to set it up to recognize and scrape the additional metadata, first--here's the code I use to do that:
The 'extra_titlepage_entries' part are added to, you guessed it, the title page.
And here is my multiseries column, as mentioned above. It takes all of the series entries as defined above and feeds them into a single tag-style field. This includes the numbers, so you can't do
#multiseries:"=Lightning [2]"-style searches by clicking on it like you can with regular series series:"=Lightning"-style searches, because the tag "Lightning [2]" is not the same as the tag "Lightning [1]", but other types of search work just fine, and it means ordering data is preserved, so I'm fine with that. This is the personal-ini code that sets up that column: I also want to add these to my logpage, the code for which I did not previously include. Here's the setup for my log page--everything after 'warnings' is an extra-metadata addition.
Of course you can delete any of these that you don't want on your own log page. The 'default' version my friend helped me with ends at 'warnings'. For some reason series entries past 03 don't work, it's a mystery to me.
We're skipping the Reading List tab, because I don't use it and I can't help you there.
Calibre Cover FFF tab and Generate Cover plugin
Okay, so the way I have this set up is useful for me, but it may not be what you want or what is useful for you. I'm going to run through how I have it set up, if you're interested in emulating it, but if you want a different kind of covers I'm not going to be of much use.
The first thing required here is to download and install the Generate Cover plugin, in the same way as we installed the FFF plugin at the start. The relevant settings in Calibre for this setup are Update Calibre Cover: Always and Generate Calibre Cover: Always. Select the radio button for Plugin Generate Cover rather than Calibre Generate Cover. I don't tick 'generate covers only for new books' because I want them to update. I do tick inject/update the cover inside the epub, because... why not really. In Generate Cover (Plugin) Settings, select your cover design preset for Default, and if you want a different preset for any other site, you can select it--I have one selected that uses a smaller font size for the tags for fics from AO3, because fics from AO3 can have way more tags and I was having issues where the tags would spill over the region for them on the cover and fill the entire cover on top of all the other text and it looked like garbage.
Backtracking, let's go into setting up Generate Cover options. There should be a button for it on your top toolbar, if not see my section on adding buttons to the toolbar. Hit that button and it'll open the dialog for generating a cover on your currently selected fic, which also allows you to customize your saved settings in general. Hit the green + next to Saved Settings to add a settings set, or edit the {Default} entry if you prefer. In the box below it, I just leave the image set to {Default Image}, which is just a blown up version of the calibre logo. It's just a space filler and I don't care what it is, so I've left it, but feel free to pick a different image if you prefer. You could set the image differently in presets for different sites if you wanted?
Anyway, once you have your new or extant settings... set... that you want to work with, go to the fonts tab. Here you can set the font and the size you want for each major element. You probably want the title and author (and series, if applicable) to be largest, and custom to be fairly small, if you're including tags like me, because there are a lot more words in tags, and also you have less need to read them at a glance. For my default settings, mine are set to, in order, 44, 34, 34, and 20. (For AO3 fics, they're 42, 24, 26, and 15.) I have use same font family for all text ticked, but you could make everything different fonts if you so desire. I do not tick auto-reduce font size to fit on one line, because that's going to absolutely ruin tag display. Bad enough on an FFN ebook, but on an AO3 one you might end up with about one pixel per word.
Dimensions tab, I leave the cover size at what I believe was the default of 590x750. I don't tick any of the boxes in size. I set the top and bottom margins to 10, the left and right margins to 0, the text padding to 20, and the image padding to 10. Border widths 0.
Now, the final tab, Contents. You can select items in the 'Field Order' box and use the green up and down arrows to rearange them as desired. You can also untick any that you don't want--if you don't want the placeholder image, for instance, you can untick the image box. I have them in the order of: Title, Author, Series, Image, Custom Text. The next box down, you can customize the custom text. I include word count and tags, which for me is formatted as:
{#wc} words
{tags}
(More on word counting, you guessed it, later.)
Other options, I don't tick the swap author box just because I don't think I've downloaded a single fanfic ever that has the author's name in LN,FN format, and I only use generated covers on fanfic. Series text I have as 'Book {series_index} of {series}', which I think is the default. Metadata you don't need to mess with. Hit the Save button in the bottom left to save your changes to the saved settings set, repeat steps to create any other schemas you desire for other sites, and hit cancel or the X to close out of this window without messing with the fic you have selected at the moment, or OK if you want to generate a cover on it right now. Now, back to FFF!
Count Pages tab and plugin
See, I told you we'd get to word counting. I'm only going to go over word counting setup, because I don't use any of the others. The plugin is called Count Pages, but I don't find pages a useful metric for me because my brain is calibrated to wordcount from all these years of fanfic. Presumably the setup is similar for page counting, I just haven't done it.
Okay, so, time to install another plugin! Go into the same search window, and find and install the Count Pages plugin. Before we go into setting it up, we have another step--we need to set up our custom column to save word counts in. Right click on a column header in the main pane and select the 'Add your own columns' option. Hit the green + to add a new one. In the edit custom column menu, give your column a lookup name--if you want ease of copying my settings, call it wc. Column heading Words (or Wordcount or whatever you desire, but I prefer it shorter so it isn't truncated in display in an ugly way). Column type should be 'Integers', description can be whatever but probably should be along the lines of 'Word Count'. I have nothing entered for 'Format for numbers' because frankly, I don't understand the syntax for how to set it, but if you know python you can probably put commas in or whatever. Hit OK, use the green arrows to move the column to whatever position in the list you prefer, hit apply.
Back to Count Pages, now that we're set up! Hit the arrow next to the button for it and select 'Customize plugin...'. We're in the Statistics tab. In the Word count options section, you should have the option to select your new custom column in the custom column dropdown, the entry for mine looks like '#wc (Words)'. I have 'use ICU algorithm' ticked, but I don't really know what difference it makes. In the Other tab, I have the button default set to 'Estimate word/page counts', because, as with FFF, I prefer to just select my fics and hit a button to update the count if I want to. I have 'Always overwrite an existing word/page count' ticked, because I want it to overwrite FFN and AO3's wordcounts so it's standardized, and I also don't want out of date counts. I have prompt to save counts ticked because it is by default and it says there's a small risk of losing other data if you don't. Hit OK.
Back to the FFF Configure menu and the Count Pages tab there. Tick Word Count. You can tick 'only if word count is missing in FFF metadata' if you don't want to overwrite FFN and AO3's native word counts. (One would equally tick Page Count there if you set that up instead or in addition.)
Standard and Custom Columns tabs, and custom Calibre columns in general
Standard Columns tab, I don't have any of the 'New Only' boxes ticked because I want it to update if it changes. I have 'Set calibre author URL' ticked, so clicking the author's name in the metadata pane opens their author page on whatever archive. I don't have the anthology boxes ticked, because, again, I don't use anthologies and I can't tell you anything about them.
The Custom Columns tab depends on what custom columns you have set up, so we're going to talk about that now!
There are a lot of types of automatic metadata that you can have filled in, like WIP/Completed status, word count, chapter count, language, and most of the already extant ones like tags and site and author. Much of this is saved in other locations, but you can put in in a column as well.
The custom columns I have are the already mentioned word count, the field for notes to myself on fic that I went over in my original mini tutorial here, and a field in which I mark fics as read or unread. (This is manual, I don't like automatic read/unread because sometimes I open a fic and decide not to read it and close it again, and sometimes I download fics I've already read online and want to mark them as read, or whatever. I can't help you with automatic marking as read.)
We already went over word count and notes, so I'm just going to quickly note that my Read/Unread status column is set up with lookup name readyn, column heading Read, column type Yes/No (this lets me select a green checkmark for read or a red X for unread, or leave it blank as it is set automatically), and description 'Have I read this fic?'.
In the Custom Columns tab, you can select from the dropdown what type of data you want to be filled into the column automatically, if any--for me, Read and Notes are blank, and Words autofills with Words, though this is mildly pointless as it's immediately thereafter overwritten by Count Pages. If you don't want to bother re-counting words on things that already have wordcount metadata, then definitely select Words for your wordcount column. As mentioned, this is also where you would select Chapters for your chapter count column, or Language in a language column, or whatever else you may so desire. I have allow custom_columns_settings from personal.ini to override ticked, though I don't think I have anything in there that affects it--if I add something later, it'll be enabled. I have all the options under 'Special column' blank.
Update Jan 2020: I have added additional custom columns now, and I'm just going to quickly run through my settings for them. One of them is quite simple--chapter count. I have this with lookup name chapters, column header Ch. so it takes up less real estate, and the type is set to Integers. In the Custom Column tab of FFF settings, I can select 'Chapters' from the dropdown menu next to it, and it'll be autofilled on new fics.
The second column I've added is "multiseries", added as a "comma-separated text, like tags" type column. It's a lot more involved. The point of it is to include metadata from multiple series applied to a fic, instead of the only one that calibre supports by default, to avoid losing important information. All of the setup for this column is in personal.ini, so the important thing here is to have the box for 'allow custom_columns_settings from personal.ini to override' ticked. Section on setting up multiseries added in the personal.ini section!
To fill both of these columns on fic I already have downloaded, I have to select it, click to update it with FFF, and select 'update calibre metadata from web site'. Because FFF throttles requests so you don't accidentally DDoS archives, this takes a very long time--I have thousands of fics downloaded, and I ended up doing it in batches of a few hundred at a time over the course of several days, because I didn't want to just not be able to use calibre for three days.
Email Settings tab
Okay, this is a convenience thing in FFF. If you set this tab up with an email account--it recommends you create a separate account for this for security, and I agree for that and so it's not having to hunt through all your non-subscription notification emails as well. (You can do something with an IMAP folder name and filter them into a folder if you don't have a dedicated email address for it, I think, but I have no experience with at and I don't really understand IMAP and I can't help you.)
I created an acount at gmx.com for this purpose, which means the IMAP server name is imap.gmx.com. I'm not positive this works with gmail, I haven't tried. I configured my main gmail email address to have a filter that collects and forwards all fic subscription notifications to the account I made for FFF, so I don't have to make new accounts at FFN/AO3/etc and re-subscribe to whatever all.
So plug in your username and password for your fic notification email into the appropriate boxes in this tab. I tick 'Mark Emails Read', because FFF only looks at URLs in unread emails, and this keeps it from looking at the same fics over and over again. I have Download from Email immediately ticked, because I feel no need to see a list of URLs and a confirmation dialog.
If you set this up, you can hit the arrow by the FFF button and click 'Get URLs from Email' and it'll check for updates on the url of every fanfic in an email in the specified inbox. (If you filter all AO3 emails, say, this will also pick up the URLs of any fic you leave a comment that is replied to on, and any fic you wrote that gets a comment or kudos notification, for the record.) I find this a much more convenient way to check for updates on WIPs than manually doing so, which I will talk about while I'm on the subject. (Particularly since I don't have anything to say about the Other tab.)
Downloading and Updating fic in Calibre
In brief: to download a fic with FFF, click the arrow, click download fics from URLs, paste the URLs you want to download into the box or drag them in. You can also do 'get URLs from web page' if you have a page with a list of them, like an author's works page--this has drawbacks on both FFN and AO3, unfortunately. On FFN, it will also scrape all of the author's favorites, which you'll have to find the boundary between the lists and delete those if you don't want them. On AO3, things only come in pages of twenty. I'll tend to use it to scrape the URLs, copy the list out into notepad, and then repeat the process for each page of an author's works/bookmarks/etc that I want to work with. (You can use FFF to download your bookmarks like this, though it will not register private bookmarks and you'll have to drag those in manually. It might see them if you have your login information entered but I can't remember.)
To download or update fics from the email setup above, just hit 'get URLs from email' and confirm on all dialogs.
To update fics you already have downloaded. select them in the main calibre pane and either hit the main FFF button (if you have the same options as me ticked for that) or the arrow next to it and then 'update existing fanfiction books'.
Enjoy your fic reading!
Please let me know if anything is missing or unclear :)
no subject
Date: 2019-02-08 05:23 am (UTC)no subject
Date: 2019-02-08 07:11 pm (UTC)no subject
Date: 2019-02-11 01:33 am (UTC)no subject
Date: 2019-02-11 12:12 pm (UTC)This looks very useful! Definitely bookmarking for later. Thank you so much for writing this up and sharing it. :D
no subject
Date: 2019-02-13 04:49 am (UTC)no subject
Date: 2019-02-11 04:21 pm (UTC)I don't read WIPs, so I don't worry about the update epub function, but I do use the series function. - there's a length below which there's not much point copying a story to my Kobo, I'll just read it on the Archive. But if there are several short stories, or a combination of shorter or longer ones, I'll make Anthologies to make a decent length book.
You can either Make Anthology Epub From URLS, which is very much the same as the download box - you enter a list of URLs and it turns them into one book.
Or you can Make Anthology Epub From Web Page, which is basically Get URLs from web page, and then it will make an anthology of them.
You can also Update Anthology Epub, which means you can add a new stories in a series to an existing book.
I leave the main settings as Check for existing Series Anthology books, and then it gives you the option to add a story to the existing Anthology.
+100
Date: 2019-02-12 09:06 pm (UTC)Here's a charming SGA series from
https://archiveofourown.org/series/2255
Paste that in and boom: one file with fifteen stories and a TOC
(no subject)
From:(no subject)
From:no subject
Date: 2019-02-11 06:15 pm (UTC)no subject
Date: 2019-02-12 01:45 am (UTC)Not sure if I can use it to update my existing library of 2k fics so that there's consistent tagging and such, but for the future it will be very helpful.
no subject
Date: 2019-02-12 04:41 am (UTC)I will additionally warn you that updating fics is rate limited so that you don't flood out the website(s), so if you want to update all 2k fics at once, you probably want to like, start it before you go to sleep and leave it overnight, because it will take hours. (First like half an hour to an hour to get all the URLs probably, and then once you confirm that list--I would guess at least six if not eight or more hours to update metadata on that many fics.) That's a large part of why I set up the email subscriptions, so I wouldn't have to wait for it to check a thousand fics and not be able to do anything in calibre while it was working.
(no subject)
From:(no subject)
From:(no subject)
From:(no subject)
From:no subject
Date: 2019-02-13 01:39 am (UTC)no subject
Date: 2019-02-13 04:46 am (UTC)no subject
Date: 2019-03-20 12:12 pm (UTC)no subject
Date: 2019-03-20 04:44 pm (UTC)no subject
Date: 2020-01-19 06:53 pm (UTC)One hurdle for me was updating older AO3 downloads, which lack metadata. When I tried "updating books" it kept failing, until I just marked it to update metadata only. I have hundreds of books and it is taking a while but SO WORTH IT.
I'm recommending this far and wide! Thanks!
no subject
Date: 2020-01-20 09:32 am (UTC)I'm glad this was useful!
(no subject)
From:(no subject)
From:(no subject)
From:(no subject)
From:Problems with step 2
Date: 2020-07-07 10:14 pm (UTC)In calibre, click on the arrow next to the preferences button,
I can't find the preferences button. I've searched all over my screen. I uninstalled Calibre to reinstall it, thinking it would ask about preferences, but it didn't; just zip and it was done installing. (I guess my system 'remembered' my choices from before. Phooey.)Wanted to be sure I wasn't bothering you for no reason. Finally noticed 3 tiny dots (almost invisible) at the right end of the Calibre menu bar. When I clicked on it, Preferences showed up, and I've now added FFF. Onward and upward, step by step.
Anyway, thank you so much for this resource. I'm trying to convert fanfic in Word to mobi for a severely non-techy friend of mine, so I can send a Kindle version direct to her. (For myself, I use Send to Kindle, which automatically converts Word for me, yay!)
But for others who are following your directions, you might want to add that newer versions of Calibre hide the 'Preference' under the right-hand dots.
.
Re: Problems with step 2
Date: 2020-07-09 05:00 pm (UTC)(FanFicFare is for downloading from the web, if you have the fic on your computer already, you can just drag and drop that straight into calibre and convert, with the caveat that Calibre does not natively support .doc/.docx files. If you save it as something else--html is probably the ideal choice--you can just import that file straight into calibre, and then use the 'convert books' function (in the top bar or the rightclick context menu) on that book now in your library, which will pull up a dialog where you can select a bewildering variety of options but you will likely only need to pick 'mobi' in the dropdown in the top right corner. I can uh explain that in better detail if you need, this is even more off the cuff and I'm in the middle of a 3d modeling project right now and happened to check my email.)
Re: Problems with step 2
From:Re: Problems with step 2
From:Re: Problems with step 2
From:Thank You
Date: 2020-12-27 12:42 am (UTC)no subject
Date: 2021-07-08 12:38 am (UTC)WIP/Complete Custom Columns
Date: 2021-08-02 05:33 pm (UTC)Re: WIP/Complete Custom Columns
Date: 2023-05-20 04:29 am (UTC)So if you make a column that checks the tags for that one in specific, you could filter it by it.
no subject
Date: 2021-12-01 11:53 pm (UTC)no subject
Date: 2023-01-30 05:47 pm (UTC)no subject
Date: 2023-05-13 02:06 am (UTC)no subject
Date: 2023-05-22 01:05 am (UTC)no subject
Date: 2023-05-20 04:27 am (UTC)I love using this database mainly due to the poor filter options AO3 currently has for bookmarked fics.
Up to now, I have been adding "Reading" and "Completed" to my Bookmark notes to see which fics I have finished or not. And with this, I can now search my fis by world count too.
But, the only downside is the creation of the columns.
Could you post a screenshot of your custom columns? I am trying to follow your instructions but I am getting a bit lost.
I am mostly trying to replicate the Ao3 tag division on the columns.
So:
Title/multi_series/Author/Fandom/characters/relationships/additional Tags/Published/Updated/World_count/chapters/bookmark_notes
I managed to add some of them, but I am having problems with the fandom column, characters, updated date, and bookmark notes.
no subject
Date: 2023-05-22 01:30 am (UTC)Okay, characters you should be able to do by adding a column for it and then going into configure FFF and the 'custom columns' tab and selecting 'characters' from the dropdown for what you want filled into it.
I think fandoms you should be able to do by ensuring you have this in your personal.ini:
extra_valid_entries:fandoms
and then
fandoms_label:Fandoms
and then
custom_columns_settings:
fandoms=>#fandoms,a
(or, yknow, whatever you named your column after the #)
Updated date I assume you want the last updated date on ao3 rather than the extant date that calibre shows for when you last checked it? That should also be under custom_columns_settings and you'll want
dateUpdated=>#updatedate,r
or like, whatever you call that column, etc.
Bookmark notes I believe should be bookmarksummary, also under custom_columns_settings, and you'll additionally need bookmarksummary in extra_valid_entries.
bookmarksummary=>#bookmarknotes,r
Also, in the custom columns tab, make sure you have 'allow custom_columns_settings from personal.ini to override' ticked.
If those don't work, let me know, and include the relevant bits of your personal.ini (from the 'view safe personal.ini' button) and I can try and troubleshoot better by like, actually testing instead of just going 'I'm pretty sure this is what would do it if I was going to do that'.
I have also been too lazy to remember how to do the little boxes I used in the main post to make my code bits all neat and tidy, sorry about that >_>
Thanks you
From:Re: Thanks you
From:Re: Thanks you
From:Re: Thanks you
From:no subject
Date: 2023-07-14 06:19 am (UTC)no subject
Date: 2023-07-20 03:52 am (UTC)Failing that, if you pop your (safe version! don't give me your passwords!) personal.ini in a comment in a <textarea style="width: 300px; height: 200px;"> </textarea> I can have a poke at troubleshooting when I've got a minute.
Need help
Date: 2023-09-16 05:52 pm (UTC)Sorry i don't know how to post an image
Re: Need help
Date: 2023-09-17 04:34 am (UTC)Hope this helps!
Re: Need help
From:Re: Need help
From:no subject
Date: 2024-10-25 08:14 pm (UTC)no subject
Date: 2024-10-27 02:13 am (UTC)Thanks!
Date: 2025-01-01 03:57 pm (UTC)Mistake on the logpage instruction
Date: 2025-07-07 02:45 pm (UTC)fandoms,<br />,freeformtags,<br />,freefromtags,
ao3categories,