What Is an MSF File? Thunderbird’s Index of One Mail Folder, and When It Is Rebuilt
An MSF file is the index Thunderbird keeps for one mail folder, so protect the mailbox next to it. Jump to your question below, or work through it in order.
What Is an MSF File? Thunderbird’s index of one mail folder, saved as .msf
Short answer
MSF files end in .msf, and they are Thunderbird’s index of one mail folder. Every one is a text file in the Mork format that lists the subject, sender and date of every message. The messages are in a separate mailbox file.
We read 8 public MSF files with four checks, and we damaged two of them in 56 ways, which gave 106 copies. Our script flagged 77 of 82 harmful copies when it also read its mailbox, and it flagged none of 24 harmless ones.
To see the parts, go to the inside part. To learn when Thunderbird rebuilds the MSF file, read the trust part. To fix a folder, follow the repair section.
An MSF file is a text file in the Mork format. It lists the subject, sender, date and size of every message and a few folder settings, and it has no message text. Thunderbird can write a new one from the mailbox whenever it needs to.
MSF stands for mail summary file. FileInfo lists three uses of the ending. The first is the mail summary file, which email programs use to index messages by their headers.
Mozilla SeaMonkey and Mozilla Thunderbird make it on Windows, Mac and Linux. The other two are a merit badge sheet from Blue Card Manager and a skin for Miranda IM.
The format inside is Mork. Wikipedia on Mork says .msf files contain mail folder summaries in Thunderbird and .mab files contain address book data. It says SeaMonkey used Mork for its POP and IMAP mail folder indexes as of 2012. Our MAB file guide covers the address books.
PRONOM lists Mork as one format with the endings .mab, .msf and .dat. Its signature is the text // <!-- <mdb:mork:z in the first 20 bytes, and its record dates from 27 February 2014.
Where Thunderbird stores it
Thunderbird stores folders as mailbox files with no ending, like Inbox, and writes MSF files next to them as Inbox.msf. Mozilla’s article on compacting folders says compacting creates a new index, for example Inbox.msf. Subfolders live in a directory that ends in .sbd, and the source code names them like folder1.sbd/.
Mozilla’s profile article gives %APPDATA%\ as the profile folder on Windows and ~/ on a Mac. On Linux it gives ~/.thunderbird/. Mail folders are inside the profile, and our MBOX file guide covers mailbox files.
Where our MSF files come from
We read 8 public MSF files. Seven come from test folders in the Thunderbird source code, and one comes from a test folder in the mork-parser package. They add up to 19,364 bytes with 5 messages, and two of them have a mailbox next to them.
We damaged two of them, bugmail-1.msf and migration.msf, in 56 ways. That gave 106 copies, because 6 edits fit one file only. We read every copy with four checks. The file command 5.45 names the type, and our own 105 line Python script checks the structure and the link to its mailbox.
mork-parser 0.9.0 and mork-converter 2.2 are two public readers of the format. We did not use Thunderbird, so every claim about what it does comes from its source code and from Mozilla’s help pages.
MSF file facts at a glance
| Property | Value |
|---|---|
| Meaning | A mail summary file. It indexes the messages of one folder. |
| Made by | Thunderbird. FileInfo adds SeaMonkey. |
| Format | Mork, a text database format with no formal standard, says Wikipedia. |
| Start |
The line // <!-- <mdb:mork:z v="1.4"/> -->. All 8 public MSF files begin with it.
|
| Standard | None. PRONOM lists the format as fmt/612, with the endings .mab, .msf and .dat. |
| Lives |
Next to the mailbox file of the same name, like Inbox and Inbox.msf.
|
| Opens with | Thunderbird opens it by itself. A text editor shows it, and two free parsers read it. |
| Size in our set | 1,835 to 3,822 bytes for the public MSF files, with a median of 2,126 bytes. |
| If it goes wrong | Thunderbird reads the mailbox again and writes a new MSF file. We read this in the source code. |
| Our tests | 8 public MSF files, 2 of them damaged in 56 ways, four checks and a 105 line script. |
Is the MSF file going away?
Not in the MSF file version we read. The source code of Thunderbird has a folder called panorama. Its design note calls Panorama the working name of a new global message database, where all folders live in one database. The note says it is under development, not live and without user facing parts.
Until that changes, every local folder has its own MSF file. The migration.msf test MSF file in our set comes from the Panorama tests, and it shows the old format as input for a migration.
Wrong page? Look at the MBOX page for the file that has the mail, the MAB page for address books or the DAT page for files of no clear type.
What Is Inside an MSF File? Five parts in a fixed order, from header to edit group
An MSF file has five parts in a fixed order. One header line comes first. A column dictionary, the message data, the folder information and an edit group follow. Numbers link the parts.
The figure cuts one sample file into its parts. It is bugmail-1.msf from test data in the Thunderbird source code. It has 2,258 bytes and lists one message. Other public MSF files repeat some parts.
The header
The first line is a comment that names the format and its version. All 8 public MSF files start with it, and PRONOM names its first 20 bytes as the signature of a Mork file. A reader that cannot find the line stops or warns.
// <!-- <mdb:mork:z v="1.4"/> -->
< <(a=c)> // (f=iso-8859-1)
(B8=account)(B9=keywords)(BA=sortType)(BB=sortOrder)(BC=viewFlags)
...
(B7=LastPurgeTime)>
<(82=0)(8A=4822253e)(80=1)>[1:m(^9C=0)(^8F=0)(^91^8A)(^92=1)]
...
{0:^80 {(k^98:c)(s=9)1:m } [0(^88=1)(^8A=0)(^8B=54)(^82^84)(^85^85)(^8D^85)
...
(^87^8D)(^8C=13)(^9B^8F)(^8E=0)]}
...
{1:^9F {(k^A0:c)(s=9)}
[1(^AC=1)(^AD=1)(^B7^81)(^A4=0)(^A1=1)(^A3^8D)(^A5^90)(^BE^91)]}
A second line starts the column dictionary. Its small group <(a=c)> marks the numbers as column numbers. The comment (f=iso-8859-1) names the character set, and all 8 public MSF files have it.
The column dictionary
Dictionaries in MSF files are lists of numbers and their meanings. In the column dictionary every number stands for a column name. Our file has 63 of them, like B8 for account and BA for sortType.
The column dictionary takes 1,189 of our 2,258 bytes, which is more than half. Every public MSF file has one, with between 58 and 79 names. The 8 MSF files use 95 different names, and 46 of them appear in all 8.
The message data
The next 636 bytes contain a value dictionary and tables. Values that repeat go into the value dictionary, so cells can point to values by number. Our file has one message, and its subject, sender and dates are in the first value dictionary.
Tables in MSF files are lists of rows, and one row stands for one message. The same row number can appear in several tables, so a scope after a colon says which kind of row it is. The next part follows one message.
The folder information
The folder information is a short row of settings for the entire folder. Ours has 152 bytes. It lists the version, message count, mailbox size, date and some more cells. The trust part shows why Thunderbird reads them first.
The edit group
MSF files change by adding text at the end. An edit group starts with @$${2{@ and ends with @$$}2}@. A change counts only when its end marker is there. Our group has 247 bytes. It adds four view settings to the folder information, which are sort type, sort order, view flags and view type.
18 edit groups appear in 7 of the 8 public MSF files. The eighth, withjson.msf, has none. A group that never ends signals a cut file, and our script reports it.
Escapes and line endings
Two hex digits after a dollar sign stand for one byte, so $C3 is the byte C3. A backslash before a bracket or dollar sign makes it text. A backslash at line end joins two lines.
7 of the 8 public MSF files have such escapes, 12 in all. Thunderbird writes a letter outside ASCII as UTF-8 bytes in this form.
Line endings differ. 5 of the 8 public MSF files use a carriage return and a line feed, 2 use a line feed and 1 uses a carriage return alone. Our sample uses the carriage return alone. Reading it with a tool that splits on line feeds shows the entire file as one line.
We read this layout from 8 public MSF files and the source of two readers. We did not use Thunderbird, so it can read some of these parts in other ways.
How Does an MSF File Describe One Message? A row of cells with a pointer into the mailbox
Every message in an MSF file is a row of cells. A row stores sender, recipients, subject, date, size and a pointer into its mailbox. It does not store the body. Folder numbers say how big the mailbox was when Thunderbird wrote the MSF file.
Our sample row has 19 cells. Every cell names a column and a value, and the dictionaries give names and text for the numbers. The table shows the cells that matter most.
| Column | What the cell says |
|---|---|
| sender, recipients, ccList | Addresses as the message header wrote them. Our file uses test addresses that end in .invalid and example.com. |
| subject | The subject text, in the value dictionary. Ours is [Bug 397009] A filter will let me tag, but not untag. |
| message-id | The Message-ID header. It stores the time 200805072155, which matches the date cell. |
| date | The cell says 4822253e. That is 1,210,197,310 seconds after 1 January 1970. The time is 21 hours, 55 minutes and 10 seconds UTC on 7 May 2008. |
| size, numLines | The cells say a77 and 13. The message has 2,679 bytes and 19 lines in the mailbox. |
| keywords, flags |
The tag $label4 and the read state. The mailbox repeats both in its X-Mozilla headers.
|
| statusOfset |
The cell says 54, which is byte 84. The X-Mozilla-Status header starts there in the message.
|
Numbers in the MSF file are hex. 13 is 19 lines, 54 is byte 84 and a77 is 2,679 bytes. Thunderbird writes dates as seconds since 1970 in the same form.
How the MSF file points into the mailbox
Three numbers tie it to its mailbox. The folder size is the mailbox file size in bytes. For our sample, folderSize says a77, and the mailbox has 2,679 bytes. The status offset says where the X-Mozilla-Status header starts in the message, and we checked that byte 84 of that message starts it.
Newer files add a storeToken cell to the message rows. In migration.msf it is the byte offset of a message in its mailbox.
That file lists two messages with the tokens 0 and 1143, and both point at a line that starts with From. The Thunderbird source compares the folder size and date of the mailbox with the cells.
The pair is not perfect. The mailbox next to migration.msf has three From lines, at bytes 0, 587 and 1143. The MSF file lists the first and third message and skips the second. The sizes still match, so the size check passes.
What the mailbox repeats
The mailbox keeps its own copy of the read state and tags. In our sample the first lines of that message are X-Mozilla-Status: 0001 and X-Mozilla-Keys: $label4.
The MSF cells flags 1 and keywords $label4 say the same thing. An MSF file is a fast copy of what its mailbox already knows, plus folder settings that only the MSF file has.
When Does Thunderbird Rebuild an MSF File? Four checks decide, and we read them in the code
Thunderbird checks four things before it uses an MSF file. The version must fit, the reparse flag must be off, the size must match its mailbox and the modified time must be close. If one check fails, it deletes the MSF file and reads its mailbox again. We read this in the source code and did not use Thunderbird.
The figure shows the decision. The database code opens the MSF file first. An MSF file that cannot be opened counts as out of date, and the code removes it. The mailbox store code adds the size and time checks for local folders.
The four checks
| Check | What the code asks |
|---|---|
| Version | The version cell must equal 1, the number in the source. All 8 public MSF files say 1. |
| Reparse flag | The forceReparse cell must not be 1. The code sets it when a folder has used up its message numbers, so the next open reads its mailbox again. |
| Size | The folderSize cell must equal the mailbox size in bytes. If both are zero, the MSF file passes at once. |
| Modified time | The folderDate cell must be within a few thousand seconds of the mailbox modified time. The preference mail.db_timestamp_leeway sets the gap, and mailnews.js gives 4000. |
The size check also wants a count of unread messages that is not negative. Our script does not test that count or the modified time.
The checks change with the folder type
The size and time checks belong to local folders in the mbox format. An IMAP folder has no mailbox on disk, so the code checks the version and reparse flag only. Our withjson.msf file is such a folder, and its cells onlineName and UIDValidity name the folder on the server.
A folder in the Maildir format has no single mailbox file. The code reads the maildirValid cell instead, and 3 of our 8 public MSF files have it. Mozilla’s Maildir article says Maildir is off by default and that it does not store message flags with the emails. Search folders have no storage, so they pass.
What happens when a check fails
Thunderbird sets the message counts to zero and copies the folder settings from the old file, if it can read the MSF file. It saves a backup of the old index when it can, and it removes the old file. Then it makes a new MSF file and parses the mailbox again, message by message.
The parser reads the flags from X-Mozilla-Status headers in the mailbox, so read state and starred state come back. Any setting that lived only in the old MSF file is gone when the old file cannot be read. That costs little, and big mailboxes need time to parse.
Why people disagree about what is lost
A 2014 thread on the Mozilla support group shows both views. One reply says the files contain nothing important and Thunderbird makes them again.
Another says they store the column layout and the sort order, which takes time to set again for many accounts. The cells we read agree with the second reply on settings.
Our sample has sortType, sortOrder and viewFlags, and withjson.msf adds a columnStates cell.
What our script checks on top
Our script also tests two cells that the four checks do not name. It flags numMsgs cells that do not match the message rows and highWaterKey cells that are lower than a message number. Both are clear contradictions inside the MSF file. We do not know what Thunderbird does with them.
Read this as the source code, not a test. We read the code of one Thunderbird source tree, and other versions can differ. We did not start Thunderbird.
What Do 8 Public MSF Files Contain? Small test MSF files, four checks and one reader that stopped
We read 8 public MSF files with four checks. File, mork-converter and our script loaded all 8. mork-parser loaded 7 and stopped on an IMAP folder. Our script flagged none. The files are small test MSF files, so they show how Thunderbird wrote its indexes and not every MSF file you can find.
The figure shows 8 MSF files, their sizes and what four checks did. Seven come from test folders in the Thunderbird source code. The eighth is Trash.msf from a test folder in the mork-parser package.
Where the MSF files come from
The sizes range from 1,835 to 3,822 bytes, with a median of 2,126 and a total of 19,364 bytes. Four files list no message. bugmail-1.msf, love.msf and withjson.msf list one message, and migration.msf lists two.
Two files have a mailbox next to them, bugmail-1.msf and migration.msf. withjson.msf is an IMAP folder, so its messages are on a server. Every file is a test MSF file, so none belongs to an actual person. They show the format well, but they do not show how a mail account that is ten years old looks.
What the MSF files show
All 8 MSF files start with the version 1.4 header and a character set comment. All 8 say version 1 in their folder cells. 5 use a carriage return and a line feed, 2 use a line feed and 1 uses a carriage return alone.
The files list between 58 and 79 column names, and 46 appear in all 8. Their folder information has between 8 and 31 cells. Edit groups show up in 7 MSF files, 18 in all. Escapes show up in 7 MSF files, 12 in all.
What the four checks did
The file command 5.45 named all 8 MSF files Mork databases. mork-converter 2.2 loaded all 8. Our script flagged none, so it raised no false alarm, and its mailbox check passed on both pairs.
mork-parser 0.9.0 stopped on withjson.msf. That file stores the column layout as JSON text in a columnStates cell, and curly brackets in the JSON confuse the parser. It printed Cannot parse and the start of the cell.
mork-converter is an old Python 2 program. We converted its source to Python 3 with the 2to3 tool and left its grammar as it was.
Two readers print the same file in two ways
Rows in Mork can share a number.
In migration.msf the number 1 names a message, a thread and folder settings. mork-parser joins rows with the same number, so its first record has the first message, its thread and folder settings.
It printed 4 records for 2 messages. mork-converter keeps the rows apart and names them by number and scope.
{
"subject": "Intuitive full-range analyzer",
"sender": "=?UTF-8?Q?\"Zo=C3=AB_Yundt\"?= <zoe.yundt@invalid>",
"sender_name": "0|\"Zoë Yundt\" <zoe.yundt@invalid>",
"date": "632017b0",
"size": "177",
"storeToken": "1143",
"flags": "0",
"message-id": "345692@invalid"
}
{
"subject": "Intuitive full-range analyzer",
"sender": "=?UTF-8?Q?\"Zo=C3=AB_Yundt\"?= <zoe.yundt@invalid>",
"sender_name": "0|\"Zoë Yundt\" <zoe.yundt@invalid>",
"date": "632017b0",
"size": "177",
"storeToken": "1143",
"flags": "0",
"message-id": "345692@invalid"
}
Look at the sender_name cell. The sender is Zoë Yundt, and the MSF file stores ë as the bytes $C3$AB. mork-parser printed Zoë. mork-converter printed Zoë, because it reads every escape as one Latin-1 character.
What the numbers do not show
We read the MSF files with four checks, and we did not use Thunderbird. The files come from test folders, and only two have a mailbox. We cannot say how Thunderbird reads them, only how our checks do.
What Happens When an MSF File Is Damaged? 106 copies, and 35 harmful ones that three public tools missed
We damaged two MSF files in 56 ways and read every copy with four checks. Our script flagged 77 of 82 harmful copies when it read its mailbox too, and 60 without it. The three public tools flagged 47, so 35 harmful copies passed all three. 24 copies are harmless.
We changed one thing at a time and sorted the edits into seven groups. They cover the header, cuts, structure, subject text, layout, folder numbers and mailbox. Edit numbers match the figure rows. Most edits went into both files, and 6 fit one file only, so the 56 edits made 106 copies.
A copy is harmful when it no longer matches what a reader or Thunderbird expects. Edit 45 moves the folder date by one second and is harmless, because Thunderbird accepts a gap of a few thousand seconds.
The file command reads only the start
The file command named 70 of 82 harmful copies a Mork database. It flagged 12, all from the first group.
They are a 20 byte cut (edit 1), a missing first line (edit 4) and a byte order mark (edit 5).
The others are an empty file (edit 6), 600 zero bytes (edit 7) and mailbox text under the name .msf (edit 8).
It named copies with version 1.3 and 9.9 (edits 2 and 3) Mork databases. It did the same for all 8 copies cut at 50, 75 or 90 percent or short by 10 bytes (edits 10 to 13). The start still looks right.
Two public readers, two sets of rules
mork-parser stopped on 22 harmful copies and warned on 2. mork-converter stopped on 30 and warned on 17. Both flagged 24 copies. mork-converter flagged 23 more, and mork-parser flagged none that mork-converter missed.
The header shows the gap. mork-parser skips comment lines and never checks the version. Copies with version 1.3 and 9.9 and a copy without the first line loaded with the same text as the intact MSF file (edits 2, 3 and 4). mork-converter warned on all three. Both objected to the byte order mark (edit 5).
mork-converter printed other text and no message on 20 harmful copies, and mork-parser did so on 35, including all 20 of the first. Both printed the edited numbers for 11 folder number copies, and that is the right output. The rest are misreads, 24 for mork-parser and 9 for mork-converter.
Damage that fools the readers
Edit 17 renamed the subject column of an MSF file to subjekt. Both readers printed the new name, our script stayed quiet and so did every other check. Both names are valid, and only a program that looks for subject finds a row without one.
Edit 18 defined the alias 81 twice. Both readers printed other text with no message, and our script flagged it. Edits 28 and 29 put bad escapes and NUL bytes into the subject. The readers printed other text again, and our script named both faults.
Edit 23 points the table at a scope that does not exist. mork-parser printed the same text as for the intact MSF file and no message, while mork-converter stopped. Our script found that no message rows were left and flagged it. Edit 24 removed the last group end. mork-parser warned, and mork-converter stopped.
Damage only the mailbox can show
26 harmful copies cannot be seen from the MSF file alone. 11 change a folder number, and 15 change the mailbox. A reader prints the number it finds, and none of the three opens a mailbox.
Folder numbers are the first case. Edit 41 adds one byte to folderSize, and edit 42 sets it to zero. Our script flagged both only when it read its mailbox.
Edits 43 and 44 change numMsgs and the version, edits 46 and 47 set the reparse flag and a low highWaterKey, and our script flagged all four without a mailbox.
Edit 48 moves the storeToken of the last message in migration.msf to byte 1000, which is not the start of a message. Without a mailbox the MSF file looks fine. With it, the script says the token does not point at a From line.
Mailbox edits are the second case. We added a byte (edit 49), removed the first 100 bytes (edit 51) and cut the mailbox in half (edit 52).
We also changed its line endings (edit 53), emptied it (edit 54) and added a message (edit 56). The size no longer matches, and our script flagged all six. The same six pass every tool that reads only the MSF file.
Damage that no reader cares about
24 copies changed nothing a reader needs. Our script flagged none of them, and neither did any other check.
Every reader loaded the copies with words after the header comment (edit 9), with all group markers removed (edit 26) and without a final line break (edit 37).
They also loaded copies with a blank line after every line (edit 38) and with 200 spaces at the end (edit 40).
A subject with the letter é written three ways (edits 30 to 32) and a subject of 5,000 letters (edit 33) loaded too. Changing the line endings (edits 34 to 36) changed nothing, so a file that was moved between systems is not damaged because of that.
Our script sees structure and the link to the mailbox, not meaning
Our script flagged 77 of 82 harmful copies and none of 24 harmless ones. Without a mailbox it flagged 60, so a mailbox adds 17 flags. It missed edit 17 on both files, edit 50 on both files and edit 55 on bugmail-1.msf.
Edit 50 changes one byte of the mailbox in place. The size stays the same, so the size check passes. Thunderbird would also look at the modified time, which our copies do not have.
Edit 55 changes the first From line. bugmail-1.msf has no storeToken that points at it, so the script has nothing to compare. On migration.msf the script flagged the same edit.
What the groups show
| Group | What we found |
|---|---|
| A. Header and file (1 to 9) | The file command flagged six edits. mork-parser passed three of the header edits with the intact text, and mork-converter flagged every harmful copy. Edit 9 is harmless. |
| B. Cuts (10 to 13) | The file command named all 8 copies Mork databases. Both readers stopped on 7 of them, and the cut of migration.msf at half its size fooled both. |
| C. Structure (14 to 26) | mork-converter flagged 20 of the 24 harmful copies and mork-parser flagged 7. The renamed column (edit 17) fooled all four checks. Edit 26 is harmless. |
| D. Subject text (27 to 33) | Unescaped brackets, bad escapes and NUL bytes were harmful. Four edits to the letter é and to length were harmless. |
| E. Layout and tail (34 to 40) | Six layout edits changed nothing. The 100 junk letters after the last group (edit 39) stopped mork-parser and made mork-converter warn. |
| F. Folder numbers (41 to 48) | No public tool flagged any of the 11 harmful copies. Our script flagged all 11 with its mailbox and 6 without it. |
| M. Mailbox changes (49 to 56) | No public tool flagged any of the 15 copies. Our script flagged 12 of them. |
Read the grid with care. We did not use Thunderbird, and we compared text and not every byte. Every test changed one thing in one small file, and our copies have no original modified time.
What Does Our Expert Say About MSF Files? A note from Neeraj Singh on protecting the mailbox
Protect the mailbox and treat the MSF file as a map that Thunderbird can redraw. In our test 35 of 82 harmful copies passed three public tools, because the damage was in the numbers that tie it to its mailbox.
Expert insight
An MSF file is a map of the mailbox, and a map that is a little wrong can still look fine. In our test 35 of 82 harmful copies passed three public tools, because the damage was in the numbers that tie the map to the mailbox. Protect the mailbox, copy the folder first and let Thunderbird write a new map.
Why an index needs care
People treat an index as harmless, because it has no mail. That is true for the mail. An MSF file still decides which messages a folder shows and in which order, and a wrong index can show a folder that looks fine and is not.
The readers add a second risk. Three public tools passed 35 harmful copies. Read the MSF file with two tools, add a script that also reads its mailbox, and trust Thunderbird’s own rebuild for a final answer.
Copy the folder first. Close Thunderbird, copy the mailbox file and its MSF file to a safe place and write down a SHA256 hash of the mailbox. The hash shows later whether the mailbox changed.
How Do You Open an MSF File? Let Thunderbird do it, or read it with a parser or a script
Thunderbird opens an MSF file by itself, so you never open it by hand. To read one, use a text editor, free parsers or our script. To read the mail, open the mailbox file, which has the name of the MSF file without .msf. Copy both files before you open anything.
We read MSF files on Linux with free tools. We did not use Thunderbird, the SysTools Thunderbird Import Wizard or the Freeviewer MBOX Viewer.
In Thunderbird
Thunderbird reads the MSF file when you open the folder. It makes the four checks of the rebuild section and writes a new file when one fails. You choose nothing.
A comment in the source code shows where folders live. Inside the profile folder, Mail/ is for a POP account and ImapMail/ is for an IMAP account.
Mozilla’s help article names the profile folder. Close Thunderbird before you copy anything from it, because the program writes to the files while it is open.
In a text editor
An MSF file is text, so Notepad, TextEdit and any code editor open it. Work on copies and do not save. You see the header, dictionaries and rows of cells. Subjects and addresses are readable, but the field names are numbers, so look every number up in the column dictionary.
Some editors show an MSF file with carriage return line endings as one long line. Our sample bugmail-1.msf is such an MSF file. Switch the line ending view or use an editor that handles it.
With a free parser
Two public programs read MSF files. mork-parser is a Node package with the MIT license, and its mork2json command writes JSON. mork-converter is a Python 2 program with the GPL 2 license that writes XML or CSV. We called both from small scripts, so the commands below are the ones their documentation gives.
sha256sum Inbox Inbox.msf
head -c 33 Inbox.msf
file Inbox.msf
mork2json Inbox.msf Inbox.json
python3 msf_peek.py Inbox.msf Inbox
mork --csv --outname=csvout Inbox.msf
The two parsers print the same message in different shapes, and the study part shows both. Neither reads a mailbox, so neither can say whether the MSF file fits it.
With a viewer
Two vendors list tools for the mailbox side of this problem on Windows. We tested neither, and both pages come from the vendors.
SysTools Thunderbird Import Wizard
Preview an MBOX file and import it into Thunderbird
The SysTools page for MSF files does not offer an MSF viewer. It says MSF files are index files with headers and summaries, and that you need the MBOX file with no ending to get the mail.
Its tool previews MBOX files and imports them into Thunderbird. The page lists no Thunderbird or Windows versions and no free or paid limits.
The Freeviewer page says its viewer opens MBOX files and shows mail, contacts and attachments. Reading is free, Pro costs $69 and Pro+ costs $129 on the page. Its lists of Windows versions disagree, so check the system requirements.
Freeviewer’s blog on the summary file error says to rename the MSF file to .mbox before you open it. An MSF file has no message text, so a viewer could show headers at best, and we did not test the step.
On a Mac or on Linux
A text editor works on any system. mork-parser needs Node, and the original mork-converter needs Python 2, which many systems no longer install. The two vendor tools are Windows programs. Our own script needs Python 3 and nothing else.
How the checks did on our 82 harmful copies
| Check | Result |
|---|---|
| file 5.45 | Named 70 copies Mork databases and flagged 12. It looks at the start of the MSF file only. |
| mork-parser 0.9.0 | Stopped on 22 copies and warned on 2. It printed other text and no message on 35, and it raised no false alarm on the 24 harmless copies. |
| mork-converter 2.2 | Stopped on 30 copies and warned on 17. It printed other text and no message on 20, and it raised no false alarm. |
| Our script | Flagged 77 copies with its mailbox and 60 without it. It flagged none of 24 harmless ones. |
How Do You Check an MSF File? Six steps and a short script
Check an MSF file in six steps. Copy the folder, read the first line, name the type and use our script, which also reads its mailbox. Then read the messages with a second tool. No check proves that Thunderbird will accept the MSF file.
The six steps
- Close Thunderbird. Copy the mailbox and its MSF file, write down the SHA256 hash of both and work on the copies.
-
Read the first line with
head -c 33 Inbox.msf. A Mork file starts with// <!-- <mdb:mork:z v="1.4"/> -->. -
Name the type with
file Inbox.msf. You want a line that starts with Mozilla Mork database. - Use the script below. Give it the MSF file and its mailbox, so it can compare sizes.
- Read the messages with a second tool, like mork-parser. Compare subjects and senders.
- Stop if the script names a fault or the tools disagree. Follow the repair steps and keep your copies.
The 105 line script that reads MSF files
The Python script below does steps 2 to 4. It reads the header, dictionaries, tables and edit groups. It changes no file and starts nothing. It read all 8 public MSF files in under 2 milliseconds, one at a time.
It checks brackets, aliases, scopes and every cell, and it reports a group that is not closed or was aborted. It reads the folder cells too. It flags a version other than 1, forceReparse cells set to 1, numMsgs cells that do not match the rows and highWaterKey cells below a message number.
With a mailbox it adds two more checks. It compares folderSize with the size of the mailbox in bytes, and it tests that every numeric storeToken points at a line that starts with From. It does not read the modified time.
#!/usr/bin/env python3
"""msf_peek.py: check a Thunderbird MSF file (Mork 1.4) and, if you give it, the mailbox that it indexes.
It reads the files and changes nothing. Use: python3 msf_peek.py Inbox.msf [Inbox]"""
import re, sys
MAGIC = b'// <!-- <mdb:mork:z v="1.4"/> -->'
TOK = re.compile(r'''(?P<com>//[^\r\n]*)|(?P<gs>@\$\$\{(?P<gsi>[0-9A-Fa-f]+)\{@)|(?P<ge>@\$\$\}(?P<ab>~abort~)?(?P<gei>[0-9A-Fa-f]*)\}@)
|(?P<cell>\((?:[^)\\]|\\.)*\))|(?P<open>[<{\[])|(?P<close>[>}\]])|(?P<ws>\s+)|(?P<id>[-+!]?[0-9A-Fa-f]+(?::\^?[0-9A-Fa-f]+|:[a-z])?)|(?P<bad>.)''', re.X | re.S)
CELL = re.compile(r'\(\s*\^?([0-9A-Fa-f]+)\s*(?:=(.*)|\^([0-9A-Fa-f]+)\s*)\)\Z', re.S)
INFO, MSGS = 'ns:msg:db:row:scope:dbfolderinfo:all', 'ns:msg:db:row:scope:msgs:all'
def value(v):
v = re.sub(r'\\\r?\n|\\\r', '', v)
b = re.sub(r'\$([0-9A-Fa-f]{2})', lambda m: chr(int(m.group(1), 16)), v)
b = re.sub(r'\\(.)', r'\1', b, flags=re.S)
try: return b.encode('latin-1').decode('utf-8')
except (UnicodeError, ValueError): return b
def peek(path, mbox=None):
data = open(path, 'rb').read()
errs, info = [], dict(rows=0, tables=0, cells=0, groups=0)
if not data.startswith(MAGIC):
return dict(errors=['the file does not start with the Mork 1.4 header'], info=info, folder={}, msgs=[])
if b'\0' in data: errs.append('NUL byte at offset %d' % data.index(b'\0'))
text, pos, stack, groups = data.decode('latin-1'), 0, [], []
cols, atoms, dscope, meta = {}, {}, 'a', False
rows, tscope, rowkey = {}, None, None # rows: (scope, id) -> {column: value}
while pos < len(text):
m = TOK.match(text, pos); pos = m.end(); k = m.lastgroup
k = 'gs' if m.group('gs') else 'ge' if m.group('ge') else k
top = stack[-1] if stack else ''
if k == 'bad': errs.append('unexpected %r at offset %d' % (m.group(), m.start()))
elif k == 'open':
c = m.group()
if c == '<' and top == '<': meta = True
if c == '<' and not stack: dscope = 'a'
if c == '{' and '{' not in stack: info['tables'] += 1; tscope = None
if c == '[': info['rows'] += 1; rowkey = None
stack.append(c)
elif k == 'close':
want = {'>': '<', '}': '{', ']': '['}[m.group()]
if top != want: errs.append('%s at offset %d does not match %s' % (m.group(), m.start(), top or 'nothing')); continue
if stack.pop() == '<': meta = False
elif k == 'id' and top in '{[':
sign = m.group()[0] if m.group()[0] in '-+!' else ''
nm, _, sc = m.group().lstrip('-+!').partition(':')
sc = cols.get(sc[1:].upper(), sc) if sc.startswith('^') else sc
if top == '{' and tscope is None: tscope = sc
if top == '[' and rowkey is None:
rowkey = (sc or tscope, sign + nm.upper()); rows.setdefault(rowkey, {})
elif k == 'cell':
v = m.group(); info['cells'] += 1
if top == '<':
a = re.match(r'\(\s*([0-9A-Fa-f]+)\s*=(.*)\)\Z', v, re.S)
if meta: dscope = 'c' if v == '(a=c)' else dscope
elif a:
d = cols if dscope == 'c' else atoms
if a.group(1).upper() in d: errs.append('alias %s is defined twice' % a.group(1))
d[a.group(1).upper()] = value(a.group(2))
else: errs.append('bad alias %s at offset %d' % (v[:12], m.start()))
elif top == '[':
c = CELL.match(v)
if not c: errs.append('bad cell %s at offset %d' % (v[:12], m.start())); continue
col = cols.get(c.group(1).upper())
if col is None: errs.append('column %s is not defined' % c.group(1))
if c.group(3) and c.group(3).upper() not in atoms and c.group(3).upper() not in cols: errs.append('value %s is not defined' % c.group(3))
val = value(c.group(2)) if c.group(2) is not None else atoms.get(c.group(3).upper(), cols.get(c.group(3).upper(), ''))
if rowkey: rows[rowkey][col or c.group(1)] = val
if re.search(r'(?<!\\)\$(?![0-9A-Fa-f]{2})', v): errs.append('bad $ escape in the cell at offset %d' % m.start())
elif k == 'gs': groups.append(m.group('gsi').upper()); info['groups'] += 1
elif k == 'ge':
gi = m.group('gei').upper()
if not groups: errs.append('group end %s has no start' % gi)
else:
g = groups.pop()
if m.group('ab'): errs.append('group %s was aborted' % g)
elif g != gi: errs.append('group %s ends as %s' % (g, gi))
errs += ['%s is never closed' % c for c in stack] + ['group %s never ends' % g for g in groups]
folder = next((r for (s, i), r in rows.items() if s == INFO and not i.startswith('-')), None)
msgs = [r for (s, i), r in rows.items() if s == MSGS and 'message-id' in r]
if folder is None: errs.append('there is no folder info row')
else:
hx = lambda n: int(folder[n], 16) if re.fullmatch(r'[0-9a-fA-F]+', folder.get(n, '')) else None
if folder.get('version') != '1': errs.append('version is %s and Thunderbird wants 1' % folder.get('version'))
if hx('numMsgs') is not None and hx('numMsgs') != len(msgs): errs.append('numMsgs says %d and the file has %d messages' % (hx('numMsgs'), len(msgs)))
if folder.get('forceReparse') == '1': errs.append('forceReparse is 1 and Thunderbird will read the mailbox again')
keys = [int(i, 16) for (sc, i) in rows if sc == MSGS and re.fullmatch(r'[0-9A-F]+', i)]
if hx('highWaterKey') is not None and keys and hx('highWaterKey') < max(keys): errs.append('highWaterKey %d is lower than message key %d' % (hx('highWaterKey'), max(keys)))
if mbox is not None:
size = len(open(mbox, 'rb').read())
if hx('folderSize') != size: errs.append('folderSize is %s and the mailbox has %d bytes' % (hx('folderSize'), size))
if mbox is not None:
box = open(mbox, 'rb').read()
for r in msgs:
t = r.get('storeToken', '')
if t.isdigit() and box[int(t):int(t) + 5] != b'From ': errs.append('storeToken %s does not point at a From line' % t)
return dict(errors=errs, info=info, folder=folder or {}, msgs=msgs)
if __name__ == '__main__':
r = peek(sys.argv[1], sys.argv[2] if len(sys.argv) > 2 else None); i = r['info']
print('%s: %d tables, %d rows, %d cells, %d groups, %d messages' % (sys.argv[1], i['tables'], i['rows'], i['cells'], i['groups'], len(r['msgs'])))
print(' ' + '\n '.join(r['errors'] or ['no problem found']))
On bugmail-1.msf and its mailbox the script finds 4 tables, 5 rows, 141 cells and 1 group. It reports no problem. On migration.msf it finds 5 tables, 8 rows, 195 cells and 3 groups. It also reports no problem, even though the mailbox has a message that the MSF file does not list.
python3 msf_peek.py bugmail-1.msf bugmail-1
bugmail-1.msf: 4 tables, 5 rows, 141 cells, 1 groups, 1 messages
no problem found
python3 msf_peek.py migration.msf migration
migration.msf: 5 tables, 8 rows, 195 cells, 3 groups, 2 messages
no problem found
The size check works in both directions. We added one byte to folderSize (edit 41) and one byte to a mailbox (edit 49), and the script named the gap every time.
python3 msf_peek.py size-plus1.msf bugmail-1
size-plus1.msf: 4 tables, 5 rows, 141 cells, 1 groups, 1 messages
folderSize is 2680 and the mailbox has 2679 bytes
python3 msf_peek.py bugmail-1.msf bugmail-1-plus1
bugmail-1.msf: 4 tables, 5 rows, 141 cells, 1 groups, 1 messages
folderSize is 2679 and the mailbox has 2680 bytes
Three more edits show other checks. The storeToken of the last message pointed at byte 1000 (edit 48). The reparse flag was set (edit 46). The numMsgs cell said 5 for two messages (edit 43).
python3 msf_peek.py token-moved.msf migration
token-moved.msf: 5 tables, 8 rows, 195 cells, 3 groups, 2 messages
storeToken 1000 does not point at a From line
python3 msf_peek.py reparse.msf
reparse.msf: 5 tables, 8 rows, 195 cells, 3 groups, 2 messages
forceReparse is 1 and Thunderbird will read the mailbox again
python3 msf_peek.py nummsgs.msf
nummsgs.msf: 5 tables, 8 rows, 195 cells, 3 groups, 2 messages
numMsgs says 5 and the file has 2 messages
Structure faults give different lines. A group without its end marker (edit 24) gives one line. A cut at 90 percent (edit 12) leaves a bracket and a group open. A bad escape in a subject (edit 28) gives the offset of the cell. A mailbox renamed to .msf (edit 8) fails at the first line.
python3 msf_peek.py no-group-end.msf
no-group-end.msf: 4 tables, 5 rows, 141 cells, 1 groups, 1 messages
group 2 never ends
python3 msf_peek.py cut90.msf
cut90.msf: 4 tables, 4 rows, 123 cells, 1 groups, 1 messages
< is never closed
group 2 never ends
python3 msf_peek.py badescape.msf
badescape.msf: 5 tables, 8 rows, 195 cells, 3 groups, 2 messages
bad $ escape in the cell at offset 1706
python3 msf_peek.py mbox-as-msf
mbox-as-msf: 0 tables, 0 rows, 0 cells, 0 groups, 0 messages
the file does not start with the Mork 1.4 header
What the script cannot see
The script reads MSF file structure and the link to a mailbox, and it does not read meaning. It printed no problem for edit 17, where the subject column is named subjekt. It also stayed quiet on edit 50, where one byte of a mailbox changed and the size stayed the same.
python3 msf_peek.py subjekt.msf
subjekt.msf: 5 tables, 8 rows, 195 cells, 3 groups, 2 messages
no problem found
python3 msf_peek.py bugmail-1.msf bugmail-1-flip
bugmail-1.msf: 4 tables, 5 rows, 141 cells, 1 groups, 1 messages
no problem found
It does not read the mailbox modified time, and Thunderbird does. We wrote it from our 106 copies and 8 public MSF files, so it knows the faults we tested and no others. A file that passes can still fail in Thunderbird, which we did not use.
Want the causes of a problem? Go to the repair part.
Is an MSF File Safe? Safe to read, and private data when you share it
An MSF file is safe to read and not safe to share. It is a data file, and it starts nothing by itself. It lists the subjects, senders and recipients of your mail in text, so treat every MSF file as private data.
Reading MSF files with text editors, scripts or parsers starts nothing. The risk is the data. We scanned no file for malware, and the 8 public MSF files are test MSF files with made up addresses.
What is inside an MSF file
The index can differ from the mailbox
An index can leave out a message that is in the mailbox. migration.msf lists two subjects, and its mailbox has three messages. If you need to know what a folder contains, read the mailbox. If you need to know what Thunderbird shows, read the index.
Before you share an MSF file
- Ask whether the other side needs the MSF file. It has no mail, and Thunderbird writes a new one.
- If the MSF file must go, open your copy in a text editor and look at the subjects and addresses.
- Send the MSF file over a channel you trust, and send its hash by another route.
- Delete your extra copies when the other side confirms the MSF file.
Share what is needed. Send one folder and not the profile. The profile folder has your accounts, your settings and every mailbox.
Should You Delete or Repair an MSF File? Eight causes and a seven step order of repair
You can delete an MSF file, and Thunderbird writes a new one the next time it opens the folder. Close Thunderbird first and leave the mailbox alone. If a folder shows wrong mail, try Repair Folder before you delete anything.
The advice comes from Mozilla volunteers and from two vendor blogs. A 2014 thread on the Mozilla support group says Thunderbird makes files again on startup or when you open the folder.
SysTools’ blog on rebuilding the MSF file lists Repair Folder and deletion. Freeviewer’s blog lists Repair Folder only. We did not test either route in Thunderbird.
The table lists what we saw in our 106 copies and what the readers said. Read every fix as a first step and not the final answer.
| Cause | Details and fix |
|---|---|
| You have a different kind of file |
A mailbox named .msf (edit 8), an empty file (edit 6) and 600 zero bytes (edit 7) were flagged by the file command and by our script. Fix: Read the first line. A Mork file starts with // <!-- <mdb:mork:z. The mailbox of an MSF file has the same name with no ending.
|
| The MSF file is cut short | A cut to 20 bytes (edit 1) and cuts at 50, 75 and 90 percent (edits 10 to 12) were flagged by our script. mork-parser stopped on 7 of the 8 copies in the cut group. Fix: Compare the size with the source. The smallest public MSF file has 1,835 bytes. Then let Thunderbird write a new file. |
| The header is damaged | A byte order mark before the header (edit 5) stopped mork-parser. Removed first lines (edit 4) and changed versions (edits 2 and 3) still loaded in mork-parser, and mork-converter warned. Fix: Copy the MSF file again from a backup, or let Thunderbird write a new one. |
| A dictionary, the table or a group is broken | An alias without a bracket (edit 14) stopped mork-converter, and mork-parser printed other text. A table that is not closed (edit 19) stopped both. A missing group end (edit 24) made mork-parser warn. A renamed column (edit 17) fooled all four checks. Fix: Use our script and read the fault it names. Then use Repair Folder. |
| A subject has a bad character | Unescaped brackets (edit 27), bad dollar escapes (edit 28) and NUL bytes (edit 29) gave other text or a stop in the readers. The letter é in three forms and a subject of 5,000 letters were harmless. Fix: Find the offset that the script names and compare it with a good copy. |
| The MSF file no longer fits its mailbox | Folder sizes that are off by one byte (edit 41) and mailboxes that grew by one byte (edit 49) pass every public tool. Our script flagged both, and Thunderbird’s size check would fail on both. Fix: Do not copy an MSF file to another machine without its mailbox. Let Thunderbird write a new file. |
| A flag asks for a new index | A forceReparse cell set to 1 (edit 46) and a version other than 1 (edit 44) are valid Mork and fail the checks in Thunderbird’s code. Our script names both. Fix: Let Thunderbird read its mailbox again, and wait until it finishes. |
| The tool or the folder type is the problem | mork-parser stopped on withjson.msf, which is a sound IMAP file, and mork-converter printed Zoë for Zoë. IMAP and Maildir folders use other checks than mbox folders. Fix: Try a second reader before you call the MSF file bad. |
Fix a folder in seven steps
- Close Thunderbird. Copy the mailbox and its MSF file to another folder, and work on the copies.
- Find the folder in the profile. The mailbox has no ending, and its MSF file has the same name with .msf.
- Read the first line and use the check script on the MSF file and its mailbox. Note every fault it names.
- Start Thunderbird, right click the folder, choose Properties and click Repair Folder on the General Information page. Mozilla volunteers and both vendor blogs describe these steps.
- If the folder still shows wrong mail, close Thunderbird, move the MSF file out of the folder or delete it, and start Thunderbird again. It writes a new file.
- Wait until Thunderbird finishes. Big mailboxes need time to parse.
- If mail is still wrong, the mailbox is the problem. An MSF file cannot repair a damaged mailbox, so look for a backup of it.
Deleting an MSF file is safe for the mail, because the mailbox has the messages and Thunderbird reads read state back from their headers. It can cost the folder view settings when Thunderbird cannot read the old file, as the four checks explains.
Not every missing mail problem is an index problem. In one Mozilla support thread repair and deleted MSF files did not help. The cause was a Quick Filter option that kept a filter active between folders.
What to send when you ask for help
If a forum or a support team asks about your folder, send these lines. They show what the files are and where the checks stopped, and they save rounds of questions. Remove names and email addresses first.
Subject: Thunderbird folder shows wrong mail
Files: names and sizes in bytes of the mailbox and the MSF file, with their SHA256 hashes
First line: the output of head -c 33 Inbox.msf
Type: the output of file Inbox.msf
Script: the output of msf_peek.py with the mailbox
Program: the Thunderbird version and the operating system
Account: POP, IMAP or Local Folders, and mbox or Maildir
History: what changed before the problem and what you tried
Not a problem, but a question about similar files? Go to the versus part.
How Do You Get the Mail Out of an MSF File? The mail is in the mailbox, and the index converts to JSON
You cannot convert an MSF file to a mailbox, because it has no message text. The mail is in the mailbox file next to it. A free parser can write the index as JSON, XML or CSV, and Thunderbird writes a new MSF file whenever it needs one.
This part reports what we tested on Linux. We did not use Thunderbird, so every claim about its behavior comes from its source code and from Mozilla’s help pages.
To JSON, XML or CSV
mork-parser writes JSON with its mork2json command. mork-converter writes XML or CSV, and its documentation lists the commands. We read 7 of the 8 public MSF files with mork-parser and all 8 with mork-converter from scripts. Both printed the subjects and senders, and mork-converter misread the letter ë.
To get the mail itself
The mail is in the mailbox. Thunderbird’s default store keeps one mbox file per folder, and our MBOX file guide covers it. A Maildir folder keeps one EML file per message, and our EML file guide covers that. Mozilla’s Maildir article says Maildir is off by default.
The SysTools page for MSF files points to a separate Thunderbird to PST converter for moving mail to Outlook. We did not test it. The MSF file plays no part in that step, because the converter reads the mailbox.
What you cannot convert
Our sample MSF file shows why. bugmail-1.msf has 2,258 bytes and its message has 2,679 bytes in the mailbox. An MSF file has a subject and a sender, and it has no body and no attachment. Renaming it to .mbox does not add them.
| Task | What we know |
|---|---|
| MSF to JSON | Tested with mork-parser on 7 of 8 public MSF files. It stopped on the IMAP file with JSON in a cell. |
| MSF to CSV or XML | Not tested with the command line. mork-converter’s documentation lists the commands, and we read all 8 MSF files with its parser. |
| MSF to MBOX | Not possible. The MSF file has no message text, so there is nothing to convert. |
| Mailbox to EML, CSV or PST | Not tested. Our MBOX and EML guides cover both formats, and the SysTools page names a converter. |
| MSF rebuilt by Thunderbird | Read in the source code and not tested. The mailbox is the source, and the MSF file is the result. |
How Does an MSF File Differ From MBOX and MAB Files? An index, the mail itself and an address book
An MSF file is an index of one mail folder. An MBOX file or a Maildir folder has the mail itself. An MAB file is a Mork address book, and a DAT file can be Mork too. They look alike, and every type has its own job.
People mix these types up because they all belong to Thunderbird and some share the Mork format. The table sorts them by what they store.
| Type | What it is |
|---|---|
| MSF | The index of one folder in the Mork text format. Subjects, senders, dates, sizes and folder settings, and no message text. |
| MBOX | The messages of one folder in one file, with no ending in a Thunderbird profile. Our guide to MBOX files covers it. |
| Maildir and EML | One EML file per message. Mozilla says Maildir is off by default and stores no flags with the emails. Our guide to EML files covers the format. |
| MAB | A Thunderbird address book in the Mork format. Thunderbird 78.0 stopped using it. Our guide to MAB files covers it. |
| DAT | PRONOM lists .dat as a Mork ending, but most DAT files are something else. Our guide to DAT files covers them. |
| SQLite | A binary database. Thunderbird uses SQLite for address books, and its new Panorama message database is also built on it. Our guide to SQLite databases covers the format. |
Of the types in the table, only the MSF file is one that Thunderbird can rebuild from its mailbox. That makes it a cache with settings, and the others are data.
What Is the MSF File Format Made Of? The facts we could check, and where they came from
An MSF file is a Mork text file with a header, a column dictionary, message data, folder information and edit groups. Numbers link the parts, and brackets mark them. The facts below come from Mozilla, Wikipedia, PRONOM, the Thunderbird source and our checks of 8 public MSF files and 106 damaged copies.
| Part | What we checked |
|---|---|
| Name | Mork, a textual database format by David McCusker, with no formal standard. (Wikipedia) |
| Endings | .msf for mail folder summaries, .mab for address books and .dat for other data. (Wikipedia, PRONOM) |
| Origin | McCusker’s primer is a Netscape newsgroup post of 16 March 1999. (Mozilla archive) |
| Signature |
The text // <!-- <mdb:mork:z in the first 20 bytes. PRONOM added it on 27 February 2014 as fmt/612. (PRONOM)
|
| Parts | Header, column dictionary, message data, folder information and edit group. (Our sample, 8 public MSF files) |
| Cells |
Round brackets with a column and a value, written as text or as hex ids like (^8D^85). (Mozilla primer, our sample)
|
| Rows and tables | Square brackets for rows and curly brackets for tables. A row starts with its id. (Mozilla primer, our sample) |
| Dictionaries |
Angle brackets with aliases like (80=1). Numbers 0 to 7F stand for the ASCII values of one byte strings. (Mozilla primer)
|
| Escapes | Closing brackets and backslashes need a backslash. Dollar signs start bytes written as two hex digits. (Mozilla primer, edits 27 and 28) |
| Edit groups |
A group starts with @$${2{@ and counts when @$$}2}@ closes it. (Our sample, mork-converter source)
|
| Character set |
The comment (f=iso-8859-1) is in all 8 public MSF files. Thunderbird writes other letters as UTF-8 bytes in dollar form. (Our files)
|
| Line endings | 5 of the 8 public MSF files use CRLF, 2 use a line feed and 1 uses a carriage return. (Our files) |
| Numbers | Counts, sizes and dates are hex text, so a77 is 2,679 and 4822253e is a date in seconds since 1970. (Our sample) |
| Folder cells | version, numMsgs, folderSize, folderDate, forceReparse and highWaterKey, plus sort and view settings. (Our files) |
| Message cells | sender, recipients, subject, message-id, date, size, numLines, flags, keywords, statusOfset and, in newer files, storeToken. (Our files) |
| Version check | The version cell must equal 1. (Thunderbird source, nsMsgDatabase.h) |
| Size check | folderSize must equal the mailbox size in bytes. (Thunderbird source, nsMsgBrkMBoxStore.cpp) |
| Time check | folderDate must be within mail.db_timestamp_leeway seconds of the mailbox, and mailnews.js gives 4000. (Thunderbird source) |
| Panorama | A new global message database, under development and not live. (Thunderbird source, panorama design note) |
| Our counts | 8 public MSF files, 106 copies from 56 edits, four checks and a 105 line script. |
About the values we list
Rules about the format come from McCusker’s Mork primer, from the source of two readers and from our tests. The mork-converter author calls the MDN page on Mork structure questionable, so we did not use it.
Rules about Thunderbird’s behavior come from the database code, the mailbox store code and Mozilla help pages. Counts come from our own files.
The Thunderbird source we read is one version of its code. A release can differ, and we did not start Thunderbird to compare.
What we left out
We damaged two small MSF files that list one and two messages. Files with thousands of messages, many threads or long edit groups can react in other ways. We did not damage an MSF file from an IMAP or Maildir folder.
We did not use Thunderbird and tested no Windows program. We compared text and not every byte. We used three readers on Linux, so Thunderbird and the vendor tools can behave in other ways.
The public MSF files come from test folders and not from actual mail accounts. We left out the Mork features that mail indexes do not use, like meta tables and row moves.
Sources checked
For the format, see Wikipedia’s Mork article, the PRONOM record and FileInfo’s page on the MSF ending. For the folder files, see the Mozilla articles on compacting folders, the profile folder and Maildir.
For the checks, see Thunderbird code in nsMsgDatabase.cpp and nsMsgBrkMBoxStore.cpp. For the future database, see the Panorama design note. Every count came from files and scripts in our research folder.
Frequently Asked Questions About MSF Files
What is an MSF file?
An MSF file is Thunderbird’s index of one mail folder, written in the Mork text format. It lists the subject, sender, date and size of every message and a few folder settings. The ending stands for mail summary file.
How do I open an MSF file?
You do not need to, because Thunderbird reads it by itself. To look inside, open a copy in a text editor. To list the messages as a table, use mork-parser or mork-converter. The mail itself is in the mailbox file next to it.
Can I delete an MSF file?
Yes. Close Thunderbird, delete the MSF file and start Thunderbird again. It reads its mailbox and writes a new MSF file. The mail stays safe in its mailbox. The folder can lose its view settings, and a big mailbox needs time to parse.
Is an MSF file a virus?
No. An MSF file is a data file, and it starts nothing by itself. It does list the subjects and senders of your mail, so share it with care. We scanned no file for malware.
Why does Thunderbird say it cannot open the summary file?
The summary file is the MSF file. The message means that Thunderbird could not use the MSF file for that folder. Try Repair Folder in the folder properties, and then delete the MSF file if the problem stays. We read the code and did not trigger the error.
How do I check an MSF file?
Copy the folder, read the first line and name the type with the file command. Then use our 105 line script with its mailbox and read the messages with a second tool. Our script flagged 77 of 82 harmful copies and none of 24 harmless ones.
Does an MSF file contain my emails?
No. It has the subject, sender, recipients, date and size of every message, and a pointer into its mailbox. Our 2,258 byte sample lists a message of 2,679 bytes. Text and attachments are in the mailbox.
What is the difference between an MSF file and an MBOX file?
An MBOX file has the messages of a folder. An MSF file indexes them, so Thunderbird can show a list without reading every message. Lose the MSF file and Thunderbird makes a new one. Lose an MBOX file and the mail is gone.
How do I rebuild an MSF file?
Use Repair Folder in the folder properties, or close Thunderbird and delete the MSF file. Thunderbird writes a new one when it opens the folder. Mozilla volunteers and two vendor blogs describe both routes, and we did not test them.
Can I copy an MSF file to another computer?
You can, but it rarely helps. Thunderbird compares the MSF file with the size and modified time of its local mailbox. A copy made for another mailbox fails that check, and Thunderbird writes a new file.
Can I convert an MSF file to PST or CSV?
Not to PST, because the MSF file has no message text. A free parser can write an MSF file as JSON, XML or CSV. To move mail to Outlook, convert the mailbox and not the MSF file.
Does Thunderbird still use MSF files?
Yes, in the source code we read. A new global message database called Panorama is under development. Its design note says it is not live, so every local folder still has its own MSF file.
Still Stuck With an MSF File? Three routes out, and where to look next
Close Thunderbird and follow the seven steps. The cause table lists what to try in order.
Read the message part, and then start our script.
Is it the mail itself? An MBOX file has the messages. Our page on MBOX files covers it, and the versus part shows how the two differ.
Is it an address book? An MAB file uses the same format for contacts. Our page on MAB files covers it.
Is it a SQLite database? Newer Thunderbird address books use one. Our page on SQLite files covers it.
Is the data unknown? Our page on DAT files covers files with no clear type, and the file extension resource library lists many more.
Is your type missing? Send it on the submit a file type page.