Using That Other OS Is A Liability

Yes, that other OS does crash and sometimes at the worst possible time.“The IRS says it has lost a trove of emails to and from a central figure in the agency’s tea party controversy. The IRS told congressional investigators Friday it cannot locate many of Lois Lerner’s emails prior to 2011 because her computer crashed that year.” That’s why I switched to GNU/Linux. I haven’t seen a crash since, on hundreds of PCs. Yet, the government of the USA continues to use that other OS and has huge problems as a result. When will they learn? Good government requires good IT and GNU/Linux is far superior to that other OS. I recommend Debian GNU/Linux. Give it a try. You know you should.

See IRS lost emails by official in tea party probe.

About Robert Pogson

I am a retired teacher in Canada. I taught in the subject areas where I have worked for almost forty years: maths, physics, chemistry and computers. I love hunting, fishing, picking berries and mushrooms, too.
This entry was posted in technology and tagged , , , , , , . Bookmark the permalink.

89 Responses to Using That Other OS Is A Liability

  1. There’s an article on ZDnet that sheds more light on the bad IT involved here:
    “In the current IRS case, there’s another piece of odd evidence. reports that the IRS contracted wsith an email archiving company named Sonasoft up until just about the time that the Lois Lerner issue became a hot potato. Sonasoft specializes in Exchange archiving. Right around the time the IRS issue became hot, Sonasoft’s contract ran out — and was not renewed.”

    So, it’s probably not just M$’s fault but their hopelessly complex system requiring experts to get right and the experts were no longer on the clock…

    The principal mode of backup that was used was printing on paper important e-mails… Sigh.

  2. dougman says:

    Irrelevant? If you have a business selling firearms and use M$, you could be setting yourself for future failure, which IS a major liability.

  3. oiaohm wrote, “If Microsoft fixed exchange and took the performance hit 8 and 16 tb mail achieves would be possible. This would help places with huge numbers of staff a lot.”

    It’s not just about huge numbers of staff although that’s a great way to generate lots of e-mail. It seems to be true that e-mail is in it for the longterm. Enough staff X FOREVER is a lot of e-mail and any organization thinking to grow or to be around for a long time should consider the storage and indexing of it. Some organizations archive or delete e-mails related to staff turnover but some e-mail should really be retained “alive” because it is part of the culture of the organization. It would really be valuable for future historians to have a perpetually growing database of e-mail. Without great indexing and search engines mankind will lose a lot of its history for no good reason.

  4. oiaohm says:

    On open source email servers its number of messages not size in most cases. 2 to power of 32 squared or 1.844E+19 with mysql MyISAM. Exchange ESE is 2 to power of 32 or ~4.295E+09 if you are optimistic. mysql MyISAM is using dynamic sized pages so a page equals a row. ESE uses fixed size pages difference a row in ESE may take up many pages where a row in mysql always takes up just 1. So mysql always gets it row count or out of disc-space.

    3.285E+10 messages per year if 90000 staff sending 1000 emails a day. Or in other words ISAM back-end of exchange completely failed if you asked it to store that in 1 database. MyISAM still has tones of room. The reality here not every ISAM backend can store 170TB of email. Minor implementation differences make huge differences. MyISAM is larger than C-ISAM ever was.

    When you understand the differences you under stand what is critical the 4 letter word about ISAM that is critical to scaling is the two magic 4 letter words. Rows and Page. Not the bull crap one your brought in. And the critical word to being able to fit a huge ISAM is compression. ESE lacks compression.

  5. oiaohm says:

    DrLoser it might take 5000 lines of code but is that 5000 lines of validated code to handle the size. There are reasons why you don’t code stuff yourself when you don’t have to. To test a back end to 256TB is not cheap.

    databases behind exchange has a 2 tb limit each that malfunctions when it hits limit not due to database engine but due to exchange.

    Sorry DrLoser you are a simpliton. Being able to change the database behind the server process allows you to change the limits. Excacty what stops a exchange update from changing how it talkes to a database. Nothing. Sorry 5000 lines of code is like a magically fix all number to quote. The reality go look at postgresql pretending to be orcale its way more than 5000 lines of code to be able to pretend to be a different database. And transforming from one relationship database to another is not always possible.

    What is behind exchange is called the “Extensible Storage Engine”. This is not a relation database as the incompetent DrLoser just claimed.

    Yes its a ISAM that does not scale. But its not just the back-end with the I don’t scale problem. Also most interesting right for mailboxs exchange explodes at 2TB even that the back-end support 8 to 16TB. Like your exchange calendars can be 16TB without issues. Exchange is buggy. Open source email servers you can use the full limits of ISAM backend in mysql and have compression. Some of it is optimisations in Exchange code when its doing read only actions bypassing the normal interfaces. So yes the ISAM behind exchange is very fast but its very unstable. Mind you the same backend database behind active directory on Windows behaves very well. No optimisations for speed bypasses in the active directory code.

    If Microsoft fixed exchange and took the performance hit 8 and 16 tb mail achieves would be possible. This would help places with huge numbers of staff a lot.

  6. DrLoser says:

    Nice eight-bar utterly irrelevant break, Dougie, love of my life.

    Any considered thoughts on oiaohm being an ignorant, offensive, moron? One whom everybody else on this site considers worthless?

    I defer to your expert opinion on this question.

  7. DrLoser says:

    Perhaps I’m not making myself plain enough, oiaohm.
    1) You are an ignorant, offensive, moron.
    2) Nobody on this site, or any other site, believes a word you say.
    3) Go away.
    4) Robert has more important things to defend than you.

    A small part of me wants to see evidence for those three propositions you made earlier, though. I will never see a single one, because, sadly, you are a bombastic ignorant moron who doesn’t even recognise that he is a bombastic ignorant moron.

    Life goes on.

  8. DrLoser says:

    Oh, and technically, almost anybody can “put Berkeley[DB] behind Exchange.” Or any other OLAP or C-ISAM based system, for that matter.

    Let me repeat this. The underlying storage system of Microsoft Exchange is a relational (I believe) database. It takes about 5000 lines of C to transform the relevant bits of a relational database, any relational database, into a BerkeleyDB/C-ISAM set of indexed files.

    I’m sure you can quote me the Norse Gods that forbid this.

    Not. Keep digging that hole.

  9. DrLoser says:

    DrLoser did I fail to mention Mysql implements the 4 letter word you were hiding.

    Yes, you did, oiaohm. Indeed you did.

    And do you have any idea why I hid it in plain sight (despite a very obvious clue)?

    The reason for that is that you are an obvious moron, oiaohm. An obvious moron who depends upon prior posts to build up a spurious argument because you just don’t understand the very basic technical information involved.

    And an obvious moron who will, consequently, sputter over those posts and try to find links on the Internet, based on information that you didn’t have in the first place, to prove a point that isn’t necessarily important and wouldn’t concern you either way.

    Why do you keep doing this to yourself?

    It’s not as though the rest of us care.

  10. oiaohm says:

    DrLoser did I fail to mention Mysql implements the 4 letter word you were hiding.

    OLAP is not part of database or ISAM. Tell the the 4 letter that allows ISAM to span multi storage. Without it ISAM cannot store very large data.

    Of course I was absolutely sure I was dealing with a person who just google who has never directly used ISAM.

    DrLoser you cannot put Berkeley behind exchange either.

  11. DrLoser says:

    I gave you a full day on XXXXXX, oiaohm. I even gave you an (inadvertent) clue: two vowels and two consonants, Vanna.

    No, not SMNP. Not only does that lack vowels, not only is it irrelevant to mail systems, but it doesn’t actually exist.

    XXXXXX is OLAP, oiaohm. Blatantly obviously, to anybody with the remotest knowledge in the field.

    But you’ve just proved your total ignorance once again, haven’t you? When will you stop embarrassing yourself? This is pathetic.

    I’ve given up on the “don’t push me, Big Boy, I can produce the links” crap, btw. I’ve asked you four times and you’ve come up empty on the single time you even attempted an answer. So be it. You are an empty echo chamber, oiaohm.

    Which demonstrates that you are not to believed on any subject whatsoever.

    But … what do we have here? A technical position paper, perhaps?

    DrLoser by the way how do you hookup ISAM up to the exchange backend. You say they were using ISAM. The answer is you cannot hook it up to exchange. If they were using a ISAM backend they are using open source email servers.


    Pick an ISAM of choice. Berkeley DB would do nicely. They’re all the same, as far as APIs and performance and so on go.

    What you do, you miserable little ignoramus, is you takes your relational database (or even hierarchical database, I don’t care) and you mirrors/archives it offline — as in OLAP, I believe I mentioned OLAP, this appears to be the first time you have ever heard of OLAP, have fun Googling irrelevant links — and you writes yourself a tiny little program.

    A tiny little program, oiaohm. I didn’t do it myself at Visa — a couple of very intelligent Chinese guys did it for me — but it turns out that you can do this inside, from memory, five thousand lines of C code.

    Which included the presentation layer to a rudimentary (3270) GUI.

    You are stupid beyond comprehension, oiaohm. If nothing else, I forbid you from insulting other people until you get at least one single thing correct. And by “correct,” I mean “accepted as correct by anybody else at all on this web site,” because no matter what you think, everybody else but me also considers you to be a worthless cretin.

    I’m done with this topic. I’ve said all I have to say. I’m demolished what I need to demolish.

    And I’m still waiting for anybody, anybody at all, to put up a decent defense for the accused here, to whit, Lois Lerner.

    Think of Lois as Superman’s Girlfriend, and perhaps you lot might get your shrivelled little gonads in gear on her behalf.


  12. oiaohm says:

    DrLoser by the way how do you hookup ISAM up to the exchange backend. You say they were using ISAM. The answer is you cannot hook it up to exchange. If they were using a ISAM backend they are using open source email servers.

    Please note using mysql MyISAM you still get compression. So the size email in does not equal the size email you store on disc.

    Yes lot of open source email servers can hook up to mysql MyISAM. C-ISAM it self does not include site to site replication or compression as MyISAM does. C-ISAM is no longer maintained as a independent code base the only maintained C-ISAM code base is inside mysql and its relations. So if IRS has ISAM data they should have migrated it to mysql MyISAM by now. Or are they incompetent here as well.

    A lot of people about databases have wanted to keep on laughing at Mysql there happens to be a few things it insanely good at due to the old code bases it built from.

  13. oiaohm says:

    Link1 DrLoser What is MyISAM. A front end for C-ISAM that makes it way more pleasant. Each table in Mysql is a single ISAM in a MyISAM setup. Guess what Mysql supports the 4 letter word you want to make me work out. Why would I deal with raw ISAM when until I get to 256TB I can place something that provides a stack of extra functionality so avoiding have to recode that stuff myself.

    By the way 256TB is the tested limit of Mysql ISAM backend called MyISAM. Theory max that you can set a config file option to unlock 65,536TB untested and pray.

    Of course MyISAM backend in fact limits out some functionality. All the issues you raised are pre-solved in MyISAM in Mysql.

    Its not the 1970 DrLoser. Why use the old interface when there are newer better interfaces with tested operational limits. Open Source email server have a habit of support mysql with MyISAM backend. Huge storage easy. Orcale has the resources to setup and test 256TB size databases. The limit is set to what is tested not to what is possible.

    Link2 DrLoser exchange can be set to generate PST and OST files on the server harddrive. PST and OST are not only client files. Microsoft never removed this functionality from the server. Why some crappy archiving software use made by third parties uses it. Sorry that was not client side. Its an option a idiot can setup when setting exchange archiving today. The warning in the documentation is that you can setup exchange server to auto generate PST or OST files. Personally I think this functionality should have been removed due to the fact it is a problem. Replaced with something like mbox that thunderbird uses that can in fact if require span multi files. So limitless in size.

    Database failures under exchange are surprising common that they are not even classed as odd events.

    Under exchange you cannot go dump all the emails you have to open format files without using third party tools. mbox format could in fact be used to dump a 170TB of emails. Lots of files when dealing with file-systems with 4G limit on file size.

    Its not simple to search 256TB of data. There are very few programs built to do it. Mysql is one of those programs.

  14. DrLoser says:

    I suppose I should mention XXXXXX (I will leave oiaohm to fill in his hilarious interpretation of those four letters. Clue, Vanna: I’ll take two vowels and two consonants. I assume you can tell the difference, 50% of the time).

    Now, XXXXXX is how you actually deal with Big Data. It’s how we’ve dealt with Big Data since at least the 1970s. But it isn’t really a database in any meaningful sense, and it certainly isn’t “a database table.” In fact, it’s nothing more than an indexed file system, as per C-ISAM.

    Now, it’s certainly true that you can store 170 terabytes of data, and in fact rather more, in a C-ISAM file system to support XXXXXX.

    In theory, this is what the IRS should have done. In practise, I’m pretty darned sure that that’s exactly what they did do.

    But I see no reason to go any further until oiaohm takes his best guess at what XXXXXX might be.

  15. DrLoser says:

    In other words, Robert, I suspect that your definition of “utter nonsense” differs in several important and quantifiable respects from my definition of “utter nonsense.”

  16. DrLoser says:

    Per Ted Codd and the relational model, Robert, a database table is nothing but a set of tabulated rows with constraints. These constraints deal with, amongst other things, how to sort the rows. I’ll deal with that first, and move on to a more important point.

    Imagine a table with a million columns, each of which is a hundred bytes wide. For these purposes, we can ignore practicalities such as extents and RAM and even cache-lines and so on. Which we should not. However, each row of our imaginary database table requires 100MB storage.

    Absurd, isn’t it? But let’s presume there is a practical requirement. Okey dokey.

    Seventeen terabytes divided by one hundred megabytes is …170,000 rows. With a single identity key to those rows (for simplicity, assume a 64-bit integer), that boils down to a nice simple O(log n) of 15 for the look-up. And, ignoring the balancing process, O(log n) for insertion/deletion, assuming a B-Tree.

    Which is where we can no longer ignore the practicalities of the underlying hardware. Because no normal server farm has 17 terabytes of RAM. And even if they did, it would still have to be mapped via virtual pages to a rather large disk. And this mapping is going to be, conservatively, tens of millions of times more expensive.

    But let’s assume this ludicrous table is almost entirely static, so we don’t pay those costs.

    Okey dokey.

    Now let’s consider a teeny weeny little second table. I’m thinking maybe a thousand rows at 10MB each, which is unlikely given the behemoth main table, but what the heck.

    Let’s do an inner join between the tables. Splendid! In the worst case, we get a thousand rows, each at 100MB.

    Let’s buy us more RAM! Thrashing be gone!

    That should do the trick for inner joins. Now for outer joins.

    Oops, we’re back to a worst case of loading the entire 17 terabytes into RAM.

    I can’t begin to tell you how much fun I used to have with these numbers when I worked with Bing.

    I suggest you save yourself the pain.

    Oh, and then there’s the cognitive pain of having a totally denormalised database residing in a single table with (example only, as above) 100MB in each row.

    No schema on Earth is going to save you from the Hamster’s insane proposition here.

  17. DrLoser wrote, “I think I can confidently assert that anybody, anywhere, whose database has “a single table” spanning 170 terabytes is clinically insane.”

    That’s utter nonsense. The structure of a database should reflect the nature of the data and how users use the data. It’s easy to imagine a huge amount of data of very simple form. Just ask the NSA… On the other hand, I’ve seen very complex databases created for holding just a few thousand records. That’s insane when the whole thing can easily fit in RAM and the structure of the model becomes almost irrelevant.

  18. DrLoser says:

    I did appreciate Dougman’s info, though. He won’t like me for appreciating it, but there you go. It’s fair to say that Dougman actually has experience behind him when he talks — not just northern NSW agrarian fantasies.

  19. DrLoser says:

    Just for starters, oiaohm, I think I can confidently assert that anybody, anywhere, whose database has “a single table” spanning 170 terabytes is clinically insane. Thanks for confirming my suspicions on that one.

    Now: to your links. As so often, you refuse to attach context to any of them. So, I’m forced to guess.

    Number 1 is a reference to Exchange 2010 – A Practical Approach. Inter alia, it points out that PST files are insecure and liable to lossage.

    Heck, who’d a thunk? Completely irrelevant. Was there any other point to that cite?

    Number two explains how to (cross your fingers on this one) repair a PST or an OST file. Also completely irrelevant. These files reside on the client computer.

    Are we beginning to see a pattern of incompetent quoting here? I promise you I’m doing this in real time. I’m not backtracking over my comments above. Let’s see:

    Number 3 describes a procedure to repair a corrupted (Exchange) database. At least this one features server software, for once.

    But what does it mean? Who knows? How did the database get corrupted? Is this a frequent occurrence? How one repairs a database — of any kind, email or not, FLOSS or not — is not an interesting question. Why you need to do so is the interesting question.

    Once again, therefore, we are left with a lack of evidence for any of your three assertions, oiaohm. Let me remind you of all three, since you have the attention span of a gnat:

    1) There’s a “12+ year old Outlook issue. To quote:
    <blockquote Sorry DrLoser keep on pushing I will end up bring in the Australian government report on the email failure here. The design issues found in exchange and outlook 12+ years ago are exactly the same today.
    I am still pushing, oiaohm. I am still pushing. On the end of a wet noodle, as far as I can tell. First: a cite, please, so we may judge. Second: explain these design issues, and why the problem is “exactly” the same today. I’m going to ignore the obvious fact that you have no authority whatsoever inside the Australian tax department and therefore your personal distress is of no consequence to anybody but your psychiatrist in this instance.
    2) Incompetent users will always result in lost data. Not on a properly audit-driven server, they won’t. And completely irrelevant in the present case.
    3) Email clients can be used as a back-up for the email server. Not once in the history of mankind, oiaohm. And certainly not relevant in the present case.

  20. oiaohm says:

    Robert Pogson every Australian department has a archives section. So even a multi location server set-up this is no a problem. You compact all the emails at each source to no duplicates then send this to the archives section who proceeds to merge what is sent in with what is already stored.

    Single network vs multi network is not a difference. Unified archiving process vs non-unified. Non unified can cost multi millions of dollars to search. Unified can be a pain in but with multi locations but is way more workable.

    Even if only the servers had working archives this takes you from like 90000+ workstations back to like 200+ servers. Its hard enough syncing 200+ machines in a hurry.

    Sogo running openchange provides calendar in outlook directly. Using MAPI plugins for other open source solutions you could make outlook use open source calendars..

    Inside exchange without de-duplication and compression you end up with unworkable insanity.

    Lois Lerner is in trouble because the IT of the IRS suxs.

  21. dougman says:

    I had to recover a drive for a client last year, here are the numbers that was quoted to me for a single drive.

    Data recovery estimate for the hard drives:

    – Standard Service (1-2 business days): $900-3900
    No charge if the data is unrecoverable

    – Economy Service (5-7 business days): $700-2700
    No charge if the data is unrecoverable

    – Priority Service (24/7): $1800-8900
    $800 attempt fee if the data is unrecoverable

  22. DrLoser wrote, “Maybe Lois Lerner was the star pupil at Mavis Beacon? Because otherwise I don’t see the storage issue here.”

    I suspect a lot of these e-mails are long and iterative and they kept copying the full text of the previous e-mail. They grow exponentially in size and have redundancy. Worst would be the case of a large group playing post office with this stuff. I can see a single e-mail growing to hundreds of MB in a few days. Many years ago, before e-mail became commonly used, I worked in a large hospital. One day a fax was sent to every departmental office in the place with instructions to distribute it to the staff who were required to read it and the thing was to be archived in the “policy manual”, a huge binder. It was a full page of text that basically was so-and-so announcing that from now on he was in charge of such-and-such. This went to 3000 employees. A few days later a similar message was distributed that said another so-and-so was actually in charge and the first message could be disregarded. We had similarly valuable messages almost daily all on dead-tree pages. Today, bureaucrats have the utter joy of being able to push “Send” and hit thousands of employees/members just like that, one person wasting the time of thousands a few minutes or so and possibly having unintended bad side-effects. The power of a bureaucrat seems to be measured either/both by salary or how much time they can waste with a single click. IRS is a layered bureaucracy with regional/functional offices/departments, a perfect labyrinth of time-wasting. On top of that we add Congress “providing oversight” and untold $millions of taxes are just being wasted with no concern at all what the intended purpose of government is to provide services to the country as efficiently as possible, and avoiding duplication… That’s just like monopoly in IT, layers of irrelevant stuff wasting resources.

  23. oiaohm wrote, “This is the problem doing 7 years storage properly should not be a problem at all even for 90000 staff.”

    That’s probably true if they are on a single network. I get the impression each office had its own setup. That’s incredibly inefficient as the cost of the search demonstrates. They seem not to have stuff from the servers but are looking at individual PCs… Also, I get the impression these folks used e-mail for everything and copied many individuals. So, what could be a simple search by user id and dates becomes an N-dimensional search over a huge sparse matrix. Essentially, in their search they are having to collect everything at one place and sift through it. What should have been a simple iterative/indexed process becomes an enormous task involving thousands of workers and a crew at head office. All the so-called desirable layers of abstraction that Exchange/Outlook is supposed to provide just got in the way. Here’s where a straight FLOSS solution would have made the world a better place. Why must they use Exchange? Oh, the calendaring… The FLOSS approach of keeping things simple and doing them very well is so superior.

  24. dougman wrote, “a typical data recovery is about $1,500”.

    Data-recovery sometimes involves as little as unsoldering/resoldering the controller but things get really hairy when the platters have damage. In the old days that was ~$1500 but it must be much more for today’s huge drives. If the platters are scratched it is sometimes possible to step the head across the surface and skip the damage but if the heads are damaged it could require custom remanufacturing of the drive, a huge expense. In a proper IT-system it’s probably far easier to scan everyone’s e-mails to find the ones sent to/from some individual but it can still happen that both sender and receiver’s e-mails were on the same failed drive. I think in this particular case, it was a single PC that crashed so it should have been possible to find copies but on ~100K PCs not properly networked, that’s a ghastly expense.

  25. oiaohm says:

    dougman I smell rat of incompetence in the IRS it for another reason. Proper archive databases are mirror raid. I have seen harddrive recovery cost in the 10,000 dollars this is normally desperation because you never had proper backups in the first place.

    Closed source SAN and Raid sound o so good until you have a failed drive.

    SAN the data can be written to disc fragmented even that to the OS it appear not fragmented so making recovery down right hard. Raids can end up doing stripping between drives again making file recovery down right hard.

    Cluster file-systems recovery is simpler why whole files normally dropped at each storage location. Cluster filesystem and general home drivers are the simplest to recovery data from.

  26. oiaohm says:

    USA IRS is 90000 staff this is public information 170TB works out to 1.8g per user. Or at least 4-5 years of storage if it used correctly at DrLosers usage rates without compression. With compression 1.8G should do 7 years at least. Yes a 2TB hard-drive per staff member x 3 for archiving reasons does the job. Or roughly 1 thousand dollars a staff member. Around 90 million spend todo job properly every 5 years. This one investigation has cost 10 million+. Have this happen a few more times and you spend the same money anyhow. Yes the reason the ATO has huge data centres it simpler.

  27. dougman says:

    I smell BS.

    Recovery of “bad sectors” does not cost $10K per drive, seeing as how the average price for a typical data recovery is about $1,500. Prices vary based on how quickly a client needs their data to be recovered, the operating system that created and managed the files, and the capacity of the affected system—if it’s a complex RAID 5/6/10 or Xsan setup.

    Before even engaging a external company for recovery, if the drive is still working, I have used SpinRite to repair drives that Windows said was bad. Once that was done, then you copy off all the data bit-by-bit.

    In my opinion, Win-dohs really wrecks drives for some strange reason, perhaps it the fragmentation issue it has. Seems Win-dohs really is messy in file orgnanization at the sectore level.


  28. oiaohm says:

    I decide to give you some current linkish links. Note in link1 the recommendation not to back exchange upto PST files. Why they break of course. No hardware failure required. Then the outlook OST files break as well. All documented in link2. Yes this current Outlook. Ok before you pull the thunderbird repair that is way more detailed and tells you to backup everything before you attempt recover(yes something Microsoft instructions always fail todo).
    Each folder in Thunderbird is a individual archive. Each folder in Thunderbird can be max size the largest file the file system can support on fat is 4G. A PST or OST file max size is 2G and it many folders. Multi file Thunderbird does is data loss reduction. A error effects 1 folder not everything. Higher resistance to failure.

    The exchange archive solution is send to another database. Then Microsoft books direct you to use exchanges internal archiving that does not compress or deduplicate. Of course Microsoft warns you the database exchange supports has a 2TB limit. 170TB=So a Min of about 86 databases. Look at postgresql backed email. Maximum Table Size 32 TB ok that is not scary so 6 tables in a postgresql store everything. Mysql is scary MyISAM backend 256TB in 1 table of 1 database. This is not even the biggest backend. MariaDB and mysql general backend size is 64 TB per table.

    This should explain why I see 170TB as nothing. 170TB is less than 1 database table or just tables inside a single database. Yes some open source email servers allow a table per user.

    170TB to a single tape I don’t think could work.

    Of course you can go HBase what is hadoop behind some of the open source email servers that is 10PB in 1 database.

    But I will stick to ones that can do exchange emulation behind sogo. Yes postgresql, mysql or oracle. All three large storage per database than exchange. All three support compression in database where exchanges database backend does not.

    This is my problem 170TB sounds raw. 90000 staff 170TB is what I kinda expect from maildir option or exchange both insane.

    Basically anyone who has really worked with open source and closed source email server that are not Microsoft and have also worked with exchange can clearly see that the numbers quoted are Exchange. Worst an Exchange system without an archiving solution connected.

    Exchange makes your email problem look way worse than it is on anything else. Exchange 2TB limits is not a polite limit either same with Exchange running out of disc space. Both exchange backing database don’t handle well. Postgresql and mysql both handle out of disc space better remaining in a non half written state.

    It does not change the fact that email archiving properly has nothing todo with if pop3 or imap or mapi being used. As your first post claimed DrLoser. Do you accept bring pop3 into the argument was a mistake and should never been raised.

    The reality here exchange is trouble. Exchange has kept market share by being the only server for outlook for a long time.

  29. oiaohm says:
    Hadoop is one of the archiving options. Does a really good job of it. But it was not implemented in this case.

    DrLoser there is a reason why archive software has compression. Text compresses extremely well. So between 1/4 to 1/10 of the value per year you quote if you are using proper archiving solutions for email that compress and even smaller with duplication removal. This is the problem doing 7 years storage properly should not be a problem at all even for 90000 staff.

    170TB figure sounds like all emails are stored in exchange databases not compressed. So never archived out of the server. Archived emails are taken off-line. In all the description of there system they don’t in fact mention a email archiving solution. They mention a server backup solution but that is not email archiving.

    DrLoser you typed a name wrong I type SNMP instead of SMTP. Its the same thing get off your high horse ever you make that kind of mistake. Spelling checkers don’t help you with these mistakes. Spell checkers don’t change Dave into Mike. So most humans have some form of dyslexia I am just worst than average. Carp instead of Camp that is spell checker.

    So exacty what is wrong with the USA IRS
    This is in my first post on this topic. From the start I want explanation of the buractact incompetence. I have been holding nothing against Lois Lerner. Best defence is offence. Best way to defender her it have the ones who were not doing the arching properly take the fall. The privacy bit I will not see the same way. This is mostly due to Australia. I guess USA if she is found not gulity she will not be due millions of dollars as here??

  30. DrLoser says:

    (Dave Camp, even. Darn this spell-checker.)

  31. DrLoser says:

    (I assume you missed the part about Mike Carp and the possibility of a political witch-hunt. No matter. These things are trivial in the grand scheme and all.)

    There is a definite reference to “bad sectors” on the hard drive and experts failing to recover the data.

    For personal reasons that I have no wish to go into, Robert, I have some knowledge of what it takes to recover data from “bad sectors.” It isn’t cheap, I’ll agree. There are specialist firms out there that will do it for you, at a cost of about $10K per hard drive — last time I looked. I imagine that this is well within budget for a federal level investigation.

    And it’s also completely irrelevant, because as I have pointed out (and oiaohm has obligingly followed my lead, in between making stuff up), at a bare minimum there should be logged details of these supposed emails.

    Consider your standard SMTP (or as oiaohm would have it, SMNP) protocol header. There’s a To, a From, a Subject, all in plain text and totalling about 1K on average. Throw in a source and destination IP address and let’s be generous some sort of trace, and bingo! You’ve hit a big ole limit of about 10K max.

    Now let’s put a hundred monkeys to work, duplicating Shakespeare for a year. A hundred emails a day. That would be 100*100*365*10K or, ahem, 365GB archive space.

    Maybe Lois Lerner was the star pupil at Mavis Beacon? Because otherwise I don’t see the storage issue here.

    Also Ms Lerner seems to have spent her days fulltime sending/receiving e-mail, many thousands per annum.

    The witch is guilty!

    The IRS spent more than $10million just sorting through them for relevant e-mail.

    In which case they should hire somebody who can deal with Big Data, perhaps using FLOSS like Hadoop. God knows, it’s not like they don’t have a plethora of Big Data. Where’s the excuse here?

    They had to search individual hard drives because there was no way to do it all from the network.

    A completely unfounded proposition. How do you know this? Where are the numbers? Why do you take their stupid excuses at face value?

    That sounds like M$’s OS to me…

    Every single god-damned cockup in IT sounds like “M$’s OS” to you, Robert.

    If you’ll forgive me, that is a problem that all moral crusaders will inevitably face.

    Sometimes, it is actually somebody else’s fault.

    And if that somebody else is the Federal Government (for bureaucratic failures) or the Republican Party (for a political witch-hunt, although it could equally well be the Democratic Party) or the Judicial Arm (for failing to protect the First Amendment) …

    Then I’m afraid that, in this particular case, you are possibly doing serious harm to Freedom by pursuing an entirely tangential Moral Crusade.

    Please pick a more suitable target.

  32. DrLoser says:

    Just to repeat the three claims that you owe us all links to, oiaohm, because I have a brother who genuinely suffers from dyslexia (it doesn’t stop him being successful, or even poetic) and I want to give you these three chances:

    Now, about that 12+ year old Outlook issue. And about that “incompetent users will always result in lost data.”

    Bollocks, twice over. Prove even one of them.

    And after that, we can return to your earlier asinine observation that, somehow, an email client can be used as a back-up for the email server.

    Can you actually dig yourself a bigger hole? You certainly can’t prove competence in any IT field that I can think of.

    The hole is so deep right now that I would even accept you giving up and admitting that you had a “dyslexic moment” when you read the original cite.

    Man up, little rodent.

  33. DrLoser says:

    DrLoser wrong about me having no empathy. The current victim only exists due to techincial incompetence that is most likely caused by bureaucratic incompetence.

    Considering that I was the first person on this thread to bring up the three possible issues of bureaucratic incompetence, invasion of privacy, and a political witch-hunt, oiaohm … you can bleat all you like. I don’t believe you for a second.

    All you’ve done is to recycle information that I have fed you, line by line. All you’ve done is to insult me whilst parroting what I’ve just told you. All you’ve done is to defend … actually, I have no idea what you were defending, and I doubt you do, either.

    What I do know is that, at no point below have you ever ONCE defended Lois Lerner against (a) bureaucratic incompetence or (b) invasion of personal privacy. Or indeed (c) anything else at all.

    So don’t come the raw prawn to me, mate. You’ve done sod all so far. You’ve never bothered yourself about these things, in five years of continual walls of gibberish.

    I’ll accept that it’s a bit too late to back-track on this one, and I will accept your promise to pay more attention to (a) bureaucratic incompetence (OS-agnostic) (b) personal privacy and (c) political witch-hunts in future.

    Until you are prepared to do that, shut your pointless little gob.

    Oh, apart from those three sets of links to outrageously stupid assertions that you promised, I have asked for three times, and you have yet to provide.

  34. oiaohm says:

    DrLoser wrong about me having no empathy. The current victim only exists due to techincial incompetence that is most likely caused by bureaucratic incompetence.

    Maybe its because I am Australian on the person privacy bit. Australian Privacy law gives no personal Privacy to anything done by a company email or in a company name.

    I hate political witch hunts as well. Mind you since governement departments here properly archive every year at least 10 bureaucrats go to jail and at least a few politician are forced to stand down in their 3 year term for doing stuff wrong. This was not happening before the Australian system got fix. This is not political witch hunts as such. A percentage of so called political witch hunts will turn out to be true. This is why its important the system work. So the non-gulity get to avoid their name pulled in the mud and the gulity get hung. The solid email archives here means a false witch hunt is deformation if there is no truth to it and the party to start it is out of office even the political office.

    DrLoser only way to reduce victoms of these problems is have absolutely no tolerance for the technical problems. Felling sorry for the current victims as an execuse to disregard future victims I class a sick as well DrLoser.

  35. DrLoser says:

    Still waiting for all three of those impressive “links” you said you had, Hamster.
    And I’m tired of you parroting my own arguments back at me.
    And I’m sickened by your inability to see that this is not a technical issue. It’s an issue of bureaucratic incompetence, personal privacy, and political witch-hunts.

    Perm 1 to 3 of 3.

    I say “sickened,” and I am indeed sickened.

    But I’m not surprised, Hamster. You just have no empathy for your fellow man, do you?

  36. oiaohm says:

    Sound like exchange storage to me as well.
    There is no mention of deduplication of emails stored. in bbs days introduced this to the open source world. There are many paid and open source email archiving solutions that perform de-duplication on stored emails.

    Really how small are the IRS servers. 170TB giving them trouble??? Sorry to say 170TB is a nothing size.

    Australian Tax Office audits where they image businesses computer harddrives can be 170 TB for one business.

    The ATO here has 3 known under ground building over 8 stories into the ground full block in size. You are talking multi cluster filesystems in Petabyte sizes with plans to be exabyte. Exactly what does the IRS storage centre look like. The biggest data system in Australia is the Tax Office. It is too large to be transfer to the Australian national archives.

    Also the other thing they are using tape. Are they nuts. ATO in Australia does not use tape. Harddrives are slow with the size they need to back up. High speed fiber links and harddrives.

    IRS sound to be using 10 year old methods.

  37. NetworkWorld has a link to documents describing in greater detail what has been happening to e-mail at IRS of USA. Apparently, their servers don’t talk to each other and their total capacity for e-mail is a problem so they do delete a lot of stuff. They did make backup tapes but they “recycled” them after six months…
    see a PDF
    see IRS computer crash eats email evidence: Conspiracy or ‘worst IT department ever’?

    There is a definite reference to “bad sectors” on the hard drive and experts failing to recover the data. Also Ms Lerner seems to have spent her days fulltime sending/receiving e-mail, many thousands per annum. The IRS spent more than $10million just sorting through them for relevant e-mail. They had to search individual hard drives because there was no way to do it all from the network. That sounds like M$’s OS to me…

  38. oiaohm says:

    DrLoser where some people get confused in in fact you should be doing archiving twice.

    Once in the delivery server and once in the receive server. So someone says they sent you something you can go yep it was in the receive server it never made it to delivery. Spam-filter or something in the middle ate it.

  39. oiaohm says:

    DrLoser all three email types Pop3, IMAP and MAPI can record when user deletes or reads emails into logs if the server is configured todo so. You said it had to be POP3. It does not matter the transport protocol its how the delivery server is configured. Configured properly all the data should be in existence either to prove Lois Lerner guilt or set her free. Exchange makes it hard to record all the details and particular paths with Exhchange can result in data disappearing. If they are not using exchange they really don’t have any form of excuse. Over all email delivery servers are prity good at doing archiving except for the one exception Exchnage.

    Remember mail delivery servers can be run spamfilters these days. So you record something in a SMTP proxy and the end user never saw it why the spamfilter run after the proxy. You archiving has to be delivery server side other wise you can end up declaring person guilty who is not.

    Dr Loser there are very strict requirements todo good quality email archiving with the intention of using it in court.

  40. DrLoser says:

    Still waiting for all three of those impressive “links” you said you had, Hamster.

  41. oiaohm says:

    DrLoser if correct archiving was done even SMTP proxy archiving there should be proof that the emails did or did not exist. Proper legal archiving should have when the the end client machine received it. This provides very useful time frame to check for use actions triggered by email so be able to find more proof the person knew of it.

    That there are questions about if the email existed or not the IRS IT Officers should be pulled in. IRS policies need to be audited. Like all business emails has to be on business accounts should be in effect.

  42. DrLoser says:


    I apologise for putting that in block caps, oiaohm, but you seem to be impervious to normal English conversation.

    Which, given your thorough-going ineptitude in the skill, is regrettably unsurprising.

  43. DrLoser says:

    And as a small point of interest, oiaohm, you might want to consider how the IMAP protocol works. (I’m not saying the protocol in question is IMAP. But let’s just make that imputation.)

    See, what happens when an IMAP server gets a notification that one of the possibly many clients has received the email is … go on, guess. You are Thetan! You are a Titan!

    … it sorta obviously has a confirmation that the client has received that email.

    Now, clearly, there is no optical proof that actual human eyeballs have scanned that email. Nor is there proof that a human brain behind those eyeballs, unencumbered by distractions such as worrying where that $1 million bull your incompetent fencing just let out might be, has read that email.

    I guess that would be up to the courts, in a specific case.

    But funny thing use you speaking may I mate?

    If the client of an IMAP email system sends a delete command, I think it’s a fair assumption that the client in question was aware of the presence of that email.

    And there is no reason, no reason on earth, why an IMAP delivery system should not be able at least to log the request to delete an email.

  44. oiaohm says:

    This is one of my rare case of doing 3.

    If you only install SMTP proxy your staff member gets to court and says I never saw that email. How do you prove otherwise. Now if your archive was on pop3 or imap or mapi server side. You get to record when the message was marked read/recieved by the client. So then it comes down to what spam filter was fitted in their machine.

    Legal requirements to be solid against staff incorrect actions SMTP proxy is not good enough. SMTP proxy is good to have a record of what was sent out not good on receive.

  45. DrLoser says:

    As to your “point” that there is no proof, even given an SMTP proxy — that’s a Simple Mail Transfer Proxy, not a Simple Network Management Protocol proxy, although I fail to see the point of one of those even given my years of Telco background experience — with persistent storage, that the end client has read the email, oiaohm?

    So What?

    The point at issue here is not whether Ms Lerner read the “untold” (ie zero upwards) emails. The point at issue is that there is no proof that those untold emails existed.

    Thus, it looks like a political smear campaign, to put it politely.

    Don’t try to fight dirty politics with your deep technological ignorance, oiaohm. These people are operators. They will beat you each way from Sunday and back again, and they will treat you like the miserable no-hope runt of the litter.

    And quite honestly, that’s how you deserve to be treated.

  46. oiaohm says:

    DrLoser and it has absolutely not changed the fact that SMTP server cannot record when was retrieved.

  47. oiaohm says:

    DrLoser I have told you in the past I have dyslexia. SNMP instead of SMTP was dyxlexia.

    DrLoser and oldman the memory cross happened for a reason. Microsoft SNMP protocol equal WMI can in fact extract details when email was retreaved by particular users by exchange. So its one of the ways to expand recorded data without having to rework email server.

  48. DrLoser says:

    In fairness to lil’ Hammie, I should point out that both Oldman and I posted at practically the same time, with no knowledge of each others’ posts.

    Anybody and everybody is of course free to don their tin-foil hats and accuse us of collusion, of course. As if it really mattered, even if it were true.

  49. DrLoser says:


    oiaohm June 17th, 2014

    DrLoser your first post here you need to say sorry for. Yes SMNP archiving is using ancient Unix Protocol. Problem here SMNP Proxy does not record when the user picked up the email or possible read the email.

    SMTP, oiaohm. As in “Simple Mail Transfer Protocol.” Not to be confused with SNMP, “Simple Network Management Protocol.”

    You can’t really fob that one off on a simple mis-spelling, can you?

    Be that as it may and call it what you will. If a given organisation, we shall call them the Australian Federal Tax Authorities for the sake of argument, and because that’s the only one you appear to have any knowledge of so it’s only fair, wishes to have a persistent record of email transactions, what they use — and this may surprise you, despite your astonishing yet totally unproven track record in IT — is the equivalent of a proxy.

    In this case, an SMTP proxy. Not, sadly, an SNMP proxy.

    Let me explain the way that a proxy server works to you, oiaohm. You will ignore me, of course, but then you have spent the best part of forty years ignoring people who try to educate you.

    A proxy server sits between a network port (we will call it the SMTP port for simplicity) and the actual server. It collects incoming messages on the SMTP port. It forwards those messages to the SMTP server, or relay.

    By the miracles of modern technology, in between these two points, the proxy can do whatever it likes with the copied message. It can log it, it can zip it up, it can store it, it can even choose to send one of those annoying “You May Just Have Won A Prize Bull!” messages that I understand plague your life.

    Anything. But what that proxy will normally do, oiaohm, in an organisation which has legal obligations to track emails, is that it will persist and archive details of those emails.

    SMNP, pah. You can’t even get that right, can you?

    Now, about that 12+ year old Outlook issue. And about that “incompetent users will always result in lost data.”

    Bollocks, twice over. Prove even one of them.

    And after that, we can return to your earlier asinine observation that, somehow, an email client can be used as a back-up for the email server.

    Can you actually dig yourself a bigger hole? You certainly can’t prove competence in any IT field that I can think of.

  50. oldman says:

    “Sorry DrLoser only incompetent in fact say SNMP proxy when it comes to meeting legal requirements.”


    Lil Hammie has once again proved he is a liar and a fraud. He goes on and on apparently not realizing that he a confused SMTP (simple mail transport protocol) with SNMP (Simple Network Management Protocol)

    I’m sorry to but in on your discussion my dear doctor, but I just couldn’t resist.

    O BTW Lil; Hammie, dont bother responding to me unless you are going to prove that you also actually know something about the IBM SAN volume Controller. Describing an generic setup will do.

  51. oiaohm says:

    DrLoser your first post here you need to say sorry for. Yes SMNP archiving is using ancient Unix Protocol. Problem here SMNP Proxy does not record when the user picked up the email or possible read the email. Really DrLoser the mail server to Archiving with good detail to answer courts questions is not straight forwards. Archiving is in fact not part of SNMP protocol I will give you this is where people using exchange thinks it should be because that is where the exchange archiving plugin goes. Due to not recording when the email was possible read it is not a complete record.

    Yes SNMP proxy server is possible todo secure persistent and achieved but incomplete record for information a investigator will ask. Investigator will always want to know when it was read or at least possible read.

    I agree its an administration issue DrLoser. The problem is you keep on saying transfer protocols this in fact has nothing todo with the problem. Good archiving should be performed from inside the mailserver so all details are recorded including pickup.

    Sorry DrLoser only incompetent in fact say SNMP proxy when it comes to meeting legal requirements. You are googling not reading documentations on meeting legal requirements. Last email server to send to client is where you want archive data from to have a true and complete record as you can without the client.

  52. DrLoser says:

    And once you have done that, oiaohm, and not before if you please, you can provide a link to justify the following asinine nonsense:

    DrLoser there are some very detailed reviews into email storage for legal requirements. Over and over again they say the same thing Transfer proto is a moot point. Incompent end users will always result in lost data.

    Heck moving data between folders can result in server side being marked as deleted.

    Not in the secure, persistent, archived repository it can’t. This isn’t a protocol issue. It’s an administrative issue. If you have a SMTP proxy sitting on ports 25, 465, 587 and whatever others are open and necessary, and that SMTP proxy forwards mail to the mail server/relay, guess what?

    That SMTP proxy is quite capable of dealing with the issue of secure, persistent, archived mail, whatever email protocol is in use.

    I was under the misapprehension that we had agreed on that obvious and simple fact. Apparently I overestimated your intelligence.

  53. DrLoser says:

    I might note that you are using precisely the same technique as the late and unloved Senator Joe McCarthy here, oiaohm.

    “I have the links, but I’m not going to show them to you.”

    Wretched, just wretched.

  54. DrLoser says:

    All right, oiaohm. Enough of the wall-o-gibberish.

    Keep pushing? I’m pushing. This is actually irrelevant to my line of argument, which you have finally caught up with despite calling me names along the way, so I will assume we can agree that
    a) Robert’s cite has nothing to do with M$.
    b) Robert’s cite has nothing to do with hard disk failures.
    c) Robert’s cite points to some interesting, if dark, propositions concerning IRS bureaucratic incompetence/concealment/ lies, the possible invasion of personal privacy, and a political witch hunt.

    None of which, as we all know here, is remotely as important as watching you quote endless irrelevant googled links in a doomed attempt to prove a point that was not relevant in the first place.

    So, this point that was not relevant in the first place. The Australian Government report on email failure in Outlook from “12+” years ago, please.

    And, following that, a link proving, or even suggesting, that the same problems are still here today.

    ‘Fess up the links or admit that you are a blustering incompetent, oiaohm.

  55. oiaohm says:

    DrLoser there are some very detailed reviews into email storage for legal requirements. Over and over again they say the same thing Transfer proto is a moot point. Incompent end users will always result in lost data.

    Heck moving data between folders can result in server side being marked as deleted. Why the user moved email from a server side imap folder to a local machine folder now if the local machine goes by by and the server is not autoarchiving that might be the last copy of the message in existence gone.

    IMAP or MAPI does not magically prevent the problem.

  56. oiaohm says:

    DrLoser Australian government departments switched there core email servers to Linux because having archiving software that works with exchange is expensive.

    Sorry there was a issue here in Australia 12 years ago with lost emails in side major government departments. So my hate of Exchange and Outlook comes from that review.

    DrLoser note what you just said about POP3 most modern day POP3 servers left on default are persistent just like IMAP is to a point.

    Sorry a badly configured IMAP server can in fact delete without user consent. There is a option on IMAP servers to delete email older than X number of days of course this would be Administrator error remember items like junk folders on imap servers have delete after x number of days set this setting in fact can be applied by error to the base folder causing a disaster.

    Outlook when archiving locally with IMAP will move email messages it archiving on the imap to trash(one of the default items that can be auto delete after so many days) then the IMAP auto clean up will delete it.

    Dr Loser it does not matter the protocol between client and server email data can be lost unless there is mandated archiving in place at server.

    DrLoser it does not change the fact notice what you said the IRS had to piece together the emails from what other employees had. This means they did not have server side arching and a policy that all business must be done by government email address.

    massive and malicious effort that is outlook. Australian governement when they had email issues over 12 years ago found this. End user thinks they are going to archive emails. Result of archiving off imap or mapi folders is the emails being locally stored get moved to trash server side then the emails are meant to disappear from client view when trash clears. Exchange makes it extreamly hard to hook the trash-can empty and send to archive instead of delete. Then the file format outlook has archived into itself PST files can due to design be rendered unreadable without harddrive failing in fact over written by outlook with invalid data due to outlook issues. Yes thunderbird and open source clients might end up triggering the server to delete the emails but the storage formats used even with damaged sectors will not result in 100 percent unreadable. Lose here would require complete harddrive loss of some form with Thunderbird and other open source email clients heck other closed source email client other than ones made by Microsoft normally also use standard resistant storage.

    Put Exchange and Outlook in a combination mix in a incompetent user you have a perfect mix to lose data no matter the transfer protocol you use between them. Lot of companies with exchange pay for third party email archiving software with exchange because exchanges own archiving format also can become unreadable just like PST files from outlook can.

    Sorry DrLoser keep on pushing I will end up bring in the Australian government report on the email failure here. The design issues found in exchange and outlook 12+ years ago are exactly the same today.

    Wait how do you repair windows issues. Reformat right or clone right. DrLoser from my point of view Client machines should always be treated as their hard-drives will become non existent. Since repair operations some people will perform will result in that state.

    DrLoser you say the protocol has to be POP3. You are not allowing that the User either moved it to trash themselves or configured something in client todo it. With outlook wizards it can be highly deceptive that archiving does not show the fact it deleting clearly to end user.

    No matter the protocol used between client and user the data can be placed in a location on server that by default it will delete unless the server is able to turn that server side delete into a move to archive.

  57. DrLoser says:

    (Dave Camp. Been watching the Red Sox recently.)

  58. DrLoser says:

    And the fact that this is obviously a politically-motivated campaign of innuendo “from the office of Dave Carp,” with spurious allegations that sound awfully like McCarthy. Go, on, tell me you can tell the difference between these two statements:

    But an untold number [of emails] are gone.

    I have here in my hand a list of two hundred and five people that were known to the Secretary of State as being members of the Communist Party and who nevertheless are still working and shaping the policy of the State Department.

    The only difference here is that Joe “Mr Nice Guy” McCarthy was stupid enough to put an exact number on his lies. Dave Carp’s “office” knows better. “Untold” sounds really impressive … unless you are numerate and realise that it means “anywhere from zero upwards.”

    Please stop trying to defend political scum.

    Please stop trying to defend bureaucratic incompetence.

    Feel free to attack M$ any other day, for any other reason.

    But don’t blind yourself to the obvious fact that this is possibly an important matter of bureaucratic incompetence and/or an invasion of personal privacy and/or a politically-motivated Witch Hunt.

    I really appreciate you bringing the OP up in the first place, because there are darned few tech blogs out there that would do so.

    However, I am disappointed in your conclusions.

  59. DrLoser wrote, “I’m still not ruling out a cover-up or just basic incompetence. I’m certainly ruling out Robert’s ill-considered assertion that “It’s All Microsoft’s Fault.””

    Well, something caused her PC to crash. I would bet on the fragility of M$’s OS over any other cause. e.g. In my last school we had up to 80 PCs running. I was there more than a year. In that time no hard drive failed (~1% per annum). However, I was re-imaging XP machines weekly after doing a bunch at once. When I arrived half the machines weren’t working and only one or two had died from any hardware-related cause. So, M$ is the cause, with ~99% confidence, I would say. OK, the IT-people should have compensated for that fragility with a better backup and/or using GNU/Linux but they didn’t. I still think M$ is the root of the problem. The fact that M$ has enlisted millions of IT-people in their fraud on the world is besides the point.

  60. DrLoser says:

    And to repeat the background:

    But an untold number are gone. Camp’s office said the missing emails are mainly ones to and from people outside the IRS, “such as the White House, Treasury, Department of Justice, FEC, or Democrat offices.”

    Camp as in Dave Camp, a Sainted Member of the Moderate Republican House. A man with a life-long love of “White House, Treasury, Department of Justice, FEC, or Democrat offices,” who is understandably distraught that his heroes have let him down.

    Can you spell “Republican Witch-Hunt?”

    (Don’t worry, oiaohm, I did that for you. Feel free to cut and paste.)

  61. DrLoser says:

    For anybody still interested: the latest on this story.

    I’m even more inclined to blame government incompetence, but then again that would be boring, wouldn’t it?

    (Apart from the interesting intersection between “government incompetence” and “invading personal privacy,” which appears to be the case here.)

  62. DrLoser says:

    You know what depresses me beyond measure about you, oiaohm?

    It’s not that you’re pig-ignorant. That’s a given. It’s not that you come up with preposterous tales of major systemic failure in globally-used products like Outlook, typically preceded by “Fun thing …” or some similar banality, but absolutely never once supported by those Google links you are so fond of.

    It’s not that you throw insults around like confetti. It’s not even that you never, never once, bother your Thetan-level intellect with the small task of following somebody else’s argument.

    No. After all these years, those things are a given.

    What really depresses me beyond measure is not even that you are incapable of original thought … what depresses me is that you don’t even recognise this obvious fact. So, I’ll recap on the story so far, and then I will present an original thought as an example.

    1) This story is tied in to a specious attack on an innocent OS. Actually, we don’t know what that OS is — it might even be the Linux Desktop.

    2) There’s some nonsense about unrecoverable hard-drives. I think we can dismiss that one as irrelevant.

    I interrupt this recap for a Fact Break! (Which is to say that I read the cited article in detail, and apparently nobody else has.)

    The IRS was able to generate 24,000 Lerner emails from the 2009 to 2011 because Lerner had copied in other IRS employees. The agency said it pieced together the emails from the computers of 82 other IRS employees.

    But an untold number are gone. Camp’s office said the missing emails are mainly ones to and from people outside the IRS, “such as the White House, Treasury, Department of Justice, FEC, or Democrat offices.”

    That Fact Break will come in handy when I explain my original thought. But, to continue:

    3) I have made it plain that I don’t believe this bull-shit from the IRS. The only way that this makes any sense is for the mail account in question to be a POP3 account. Even then, the POP3 server would have to be massively badly configured, because these days it is not uncommon for a POP3 server to treat a DELE request as a “marked as read” account. It is not mandatory to delete that message from the server.

    4) As an alternative, the server might have used IMAP (or even MAPI, which has similar characteristics). Ditto for the delete thing. However, in this particular case, and it’s why I brought it up: even a badly-configured IMAP server will not delete the OP. That would imply a massive and malicious effort on the part of the email client, viz Ms Lerner.

    5) Even given all that, it’s not easily conceivable that the IRS (of all people) has a policy in place that does not mandate the archiving of each and every email going through their system.

    Which is where my original thought comes in, oiaohm. Are you listening? This might be instructive. Instead of feverishly reading Google articles and trying for an illusory and temporary one-upmanship (what a shallow little person you must be), you could actually try to figure out what is going on in this particular case.

    Now, if you tried that, you might come up with an even more interesting possibility than the one I am going to suggest. But here it is, anyway.

    The IRS were not trying to retrieve emails sent via official channels.

    Barring incompetence or a cover-up (both possible, I agree), that’s the only way that this makes any sense.

    In 2014, you don’t need to confiscate a PC and retrieve local copies of emails unless … that’s the only place you can find them. Which implies that the server for these emails is some shoddy little thing that doesn’t routinely archive them (Oh, how the mighty NSA is fallen!). And by implication suggests that the email trail in question is via one or more ISPs of a questionable nature. And by implication suggests that Ms Lerner has been playing fast and loose with confidential US government information. Which would make sense in a way, because otherwise the First Amendment (amongst other resources) would preclude taking a personal computer away for forensics.

    I’m still not ruling out a cover-up or just basic incompetence. I’m certainly ruling out Robert’s ill-considered assertion that “It’s All Microsoft’s Fault.”

    But, even given that this cite is from a gutter-level organisation like “US News” — as always, nobody here actually cares about the quality of the source — the explanation behind it requires thought.

    Thought, oiaohm. Not fantasy. Nor unsubstantiated insults and a lack of evidence, although I fully recognise that these are all you have left in your pathetic pretence to understand anything at all about Information Technology.

    And the Law.

    And penning in Prize Bulls …

    … The list goes on and on and on, doesn’t it, oiaohm?

  63. oiaohm says:

    DrLoser incompetent idiot. For legal reason archives must remain intact even of the end user requests deletion.

    Outlook can be configured to auto archive locally. The result is delete instructions sent to servers. Welcome to imap persistent storage failure. Imap does not replicate on emails that 1 client has in fact deleted. Imap protocol persistent storage is only to allow many clients to download the same messages not to protect or store the messages for long term legal archiving requirements.

    Imap also gets more fun persistent storage failure number 2. All clients have downloaded a email and 1 client tells the server to delete a message. The persistents of the imap replicates that instruction to all clients. So deleting every copy.

    Imap persistant storage is just has as much a hazard to archive requirements as pop3 is. Same with MAPI. The transfer protocol when it comes to archive requirements is your enemy it is never the your friend with email.

    Sorry Loser only a person who has not run email servers with archive requirements believes that the imap persistent storage is more persistent than it is.

    DrLoser the government has legal requirements. Its their IT department that is required to configure their servers correctly to mean the storage requirements. Just that a lot of email servers are not configured correctly.

    Emails on servers when legal requirements exist that they may require to be presented in court like IRS or Australia Tax Office should never be obeying the email client deletions. Email delete commands on Australian Tax Office server means send to long term archive. This is a server configuration.

  64. DrLoser says:

    Incidentally, oiaohm, the reason that IMAP and relatives enforce persistent storage, rather than leaving it as an implementation detail, is because they are relatively modern protocols that recognise the possibility of multiple clients for a single server (or indeed relay).

    Thus, it would be a very bad thing for an IMAP server (or relay) to delete a message. If it deletes the message, it cannot forward it to the multiple other clients.

    Now, there is (I believe) an end-to-end message in IMAP, whereby the ultimate client can request “deletion.”

    But, you know what? If I was in charge of a large bureaucratic organization, I would treat this message as a “don’t bother me again” request. I would take the email in question off the list of “inbox/whatever” messages.

    And yet I would still archive the bloody thing. There’s an enormous amount of legal requirements out there, from Sarbanes-Oxley on, that would plonk me right in the klartz if I didn’t.

    And archive storage is cheap, cheap, cheap.

  65. DrLoser says:

    DrLoser as normal you are have incompetence. Using pop3 or imap or mapi does not mean data retention problems. Are you not aware every Unix/Linux mail server can be configured to do data retention. Yes for ever store every email received and sent to archives.

    As usual, the Tin-Foil Hat comes out with a Straw Man Argument.

    Of course I am aware of the fact that a POP3 server can be configured to persist each email, oiaohm. Unlike you, I actually have a job in IT.
    However, the point at issue is that, apparently, an email trail has been lost. Let’s lose the OS argument for a moment, since it isn’t obviously relevant.

    The only two ways that an email can fail persistence are:
    1) The persistent storage in question is wiped, for some reason. I’m guessing bureaucratic cover-up, but it’s just a guess.
    2) There was no persistent storage in the first place.

    Point (2) requires us to believe that
    (a) the protocol in question does not necessarily enforce persistent storage. This is easy to believe with POP3. Any number of mail servers can be configured to deliver a message via POP3 and persist the original message. A number of mail servers that deliver a message via POP3 do not do so.

    This is such an obvious detail, oiaohm, that I cannot believe you have missed it. What, Google links failing you again?

    The server is meant to be the backup to the client and the client is meant to be a backup to the server in a email setup.

    Really? Cite a single instance where an email client is the backup to an email server, oiaohm. You can’t. There is no such thing.


    To be correct one of the big problems with outlook has been telling pop3 email servers to delete everything even when outlook configurations set not to.

    Now, that’s more like it. This is interesting. If true, it’s a real M$ issue.

    And you know what? I don’t believe you. Show us the evidence, please. Note that you are claiming that Outlook acts as a relay in this case: there is a POP3 server on one side and a POP3 client on the other.

    This is an interesting scenario, because, in order to fulfil your fantasies, the relay (ie Outlook) would have to specifically issue a DELE command to the POP3 server. (And, incidentally, there is no requirement for the POP3 server to honour this command. At the very least, it is free to archive the message somewhere.)

    So, proof please, or I will assume that as usual you are making things up.

    Yes exchanges pop3 provide interface also triggers deletion of emails on server random-ally when the client has not asked for it to happen again another Microsoft item breaching protocol standards.

    And which end-to-end standard would we be talking about here, oiaohm?

    There isn’t one, is there?

  66. oiaohm says:

    DrLoser as normal you are have incompetence. Using pop3 or imap or mapi does not mean data retention problems. Are you not aware every Unix/Linux mail server can be configured to do data retention. Yes for ever store every email received and sent to archives.

    Using Linux Desktops would get away from being stuck on Exchange as much with its self corrupting database and lack of auto archiving options. The problem with emails is two fold. Heavy usage of windows clients has lead to the usage of outlook that has file storage issues that then depends on a email server will file storage issues and lack of archiving options.

    The server is meant to be the backup to the client and the client is meant to be a backup to the server in a email setup. When both are dud you have a big problem.

    Transfer protocol that you are trying to make the issue is not the issue DrLoser because the archiving is meant to exist at both ends not in the transfer protocol choose. To be correct one of the big problems with outlook has been telling pop3 email servers to delete everything even when outlook configurations set not to. Yes pop3 and imap might be old protocols but Microsoft email clients support for them is unstable. Intentionally so. Yes exchanges pop3 provide interface also triggers deletion of emails on server random-ally when the client has not asked for it to happen again another Microsoft item breaching protocol standards.

    Exchange is a broken email server. Outlook is a broken email client.

    DrLoser no matter what the lost data was a double failure. Australian tax department their Linux servers auto archive all emails for 7 years in cluster storage. No single hard-drive failure will cause a loss. Client machines being completely deleted will not cause loss either. So exactly what is wrong with the USA IRS. Maybe too much dependence on Microsoft and incompetent IT staff or the lady is lieing and never attempted to get backup copies.

    A government department in Australia using the excuse lost email will not fly. A 30 day delay to recover it from archives is about as far as they go. One of the biggest users of openchange in Australia is the Australian government to redirect outlook mapi protocol to Linux based email servers so that the archiving requirement can be meet.

  67. dougman says:

    Au contraire!

    My previous posts were indeed germaine, in that the blog title states “Using that other OS is a liability”. Unfortunately, you refuse to see Win-Doh’s problems, so they are irrelevant and insignificant to the topic at hand for you.

    You focus on the minutiae, but I stand-up and look at the complete picture. Instead of fiddling with bits of sand, you should see the forest for the trees and realize that Win-Doh’s is a dismal sorry state.

  68. DrLoser says:

    None of which assertions (tedious as your repetition of them is) have anything whatsoever to do with the matter at had, Dougie.
    Unrecoverable data from hard disk.
    Government cover-up.
    Now, we’re all well aware that you are so technologically ignorant that you cannot begin to understand those first two points.
    But the great thing about you, Florence (may I call you Florence?) is that you are as one with the community of tin-foil hat connoisseurs.
    Looks good on you, btw. Much better than that butch “wannabe hoody” backward baseball cap.
    So, let’s recap, shall we?
    This has nothing at all to do with Microsoft.
    This has nothing at all to do with Linux.
    This has nothing at all to do with Android.
    This has nothing at all to do with Gnu.
    In fact, it’s a complete waste of everybody’s time, not that your time, Douglet, is worth spit.
    Can we move on, please? Nothing to see here, madam, small car crash, on we go to the next preposterous assertion.

  69. dougman says:

    Justify my assertions? HAHA! your fingers broken?

    Just Google, or BING if you will (BEWARE: correct and/or relevant search results not guaranteed, when using BING) the term “Windows Top Problems” and click on the first, fourth and tenth links that come up.

    Surely, you cannot set there with a straight face and tell us that Win-Doh’s does not have problems.

  70. DrLoser says:

    Have you ever claimed that a random instance of thousands of emails going astray without any sort of record could have been avoided if only the US government department had chosen to use the Gnu/Linux desktop, Mats? I’m going to guess that the answer is “not.”
    POP3, possibly. Bureaucratic lies, almost certainly. There may be the odd really sharp cucumber involved if you want to cause a disastrous loss of data on the local hard drive.
    Otherwise, this entire conversation is a complete waste of everybody’s time.
    Cue oiaohm>/b>.

  71. Mats Hagglund says:

    I’ve never claimed that Linux desktop is perfect. There ain’t perfect desktop at all. What’s true is that Linux is much more user friendly, easier to install, use and learn than IT media bundits and most of folks are believing. My only mistake was that i’d have moved to Linux desktop much earlier.

  72. DrLoser says:

    Kurks, you’re ignoring the topic at hand … as usual!
    Besides, 8086 would have been trade-marked by Mr Peano, if nobody else.
    I mean, it’s the successor to the successor to the successor to the THIS TEDIOUS COMMENT REDACTED BY A COMBINATION OF BUREAUCRACY AND THE FRANKLY UNBELIEVABLE USE OF POP3

  73. kurkosdr says:


    Enough with the trademark nonsense. Even if Windows is not a trademark, Microsoft has the right to call their product as they want. Just because a word cannot be trademarked, it doesn’t mean you can’t use it to name your product.

    For example, “8086” cannot be trademarked, but there is a product with the name.

    Phrases like “Lose” and “TOOS” make you sound like a 14-year old who calls the teacher he doesn’t like “pumkin-head” and refuses to call him by his normal name. It’s juvenile and silly.

  74. DrLoser says:

    Douglet, my sweet:

    I understand the missionary zeal here, but you really need to focus on actual, as opposed to imaginary, problems.

    Naturally you prefer to completely ignore the posted topic of discussion. This is nothing new. Snake-oil salesmen do that as a matter of course.

    Well, fine. Whilst we’re outside the cage at the zoo, listening to you gibbering to yourself:

    This topic is clearly dead, since not even the originator is prepared to support the preposterous notion that Microsoft (completely un-named in the cite) is to blame. Not even any of those posting since has seriously tried to claim that the fault lies at the door of anything other than (a) a bureaucratic cover-up or (b) a bozo reliance on the ancient *nix protocol, POP3.

    We’re almost done here. All we need, my love, is for you to justify one or more of your dreary twenty four “assertions.”

    What a huge, unfeasibly huge, number. I mean, huge.

    Don’t you think that a couple of billion customers might have noticed one or two of those little issues and decided to go for the far superior Gnu/Linux desktop?

    Because I do. But then again, I don’t despise my fellow man the way you do, Dougie. Nor do I sell them crap and destroy their original media.

    Apparently we have a small difference of opinion here.

  75. dougman says:

    Microsoft sure hit the ball out of the park, when they named their software. As ‘Windows’ in the generic version of the word, is easily broken, synonymously the software version of the word ‘Windows’, breaks just as easily and routinely.

    I like the colloquialism, Win-Doh’s, as the IT vernacular fits Bill, pun intended. Win-Doh’s also represents a one Homer Simpson and his many Doh’s throughout his life.

    Malware – DOH!
    Restarts – DOH!
    Reboots – DOH!
    Trojans – DOH!
    Viruses – DOH!
    Broken updates – DOH!
    Proprietary Lock-in – DOH!
    Forced upgrades – DOH!
    Blue, Red and Black Screens – DOH!
    Unexplained crashes – DOH!
    Incompatible drivers – DOH!
    Memory Leaks – DOH!
    Slow Windows – DOH!
    Aero Shell not working – DOH!
    Older programs not Compatible – DOH!
    DVD/CD drives not found – DOH!
    Slow shutdowns – DOH!
    Slow startups – DOH!
    Bugs are features – DOH!
    Features are bugs – DOH!
    Upgrades hang the system – DOH!
    No Start button – DOH!
    Forcing a unwanted tablet UI – DOH!
    Hidden extensions – DOH!

    and the biggest DOH! of them all?? No one cares anymore about Micro$h1t, they rather use ChromeOS and their Android phone.


  76. pogson says:

    kurkosdr wrote, “Windows”.

    Windows is a generic term used on rectangular regions of screens to interact with processes. As such it may not legally be protected by trademark law.The reason for this is to prevent folks from locking out competitors. e.g. I cannot trademark “doors” and sue everyone who tries to sell doors. USPTO, as usual, dropped the ball. They now hold that M$, having been a monopolist for a long time deserves to be a monopolist forever. I disagree.

  77. kurkosdr says:


    It’s Windows. Not “That Other OS” (Desktop Linux distros are actually “those other OSes” if you count them individually, simce the fall below the 1% mark), not “Lose”.

    Enough with the newspeak. Would you like if we calledDebian “NeckbeardOS” and Ubuntu “Opoompoo”?

    Trademark or not, the name of the product is Windows.

  78. luvr says:

    Surely they could ask NSA for a copy of the e-mails? They store them anyways… 🙂

  79. DrLoser says:

    Incidentally, Dog-Brain:
    About that legal threat of yours.
    Thanks for giving me a legitimate opportunity to use that meme.

  80. DrLoser says:

    Thrilling though this little diversion into a Chromebook deployment in Woolworths, Australia …
    Don’t you think this is a little off-topic, Dougie and Bob?
    It’s perfectly possible to celebrate the insanely great 8,000 deployment of Google’s latest “it’s Linux, Jim, but not as we know it…” and yet still provide a penetrating and convincing analysis of how an un-named OS, presumed to be M$, managed to trash a hard drive, presumed next to impossible according to Robert’s rather dubious statistics, into a state which was completely unrecoverable, presumed to be a legally justifiable position although we are all perfectly aware that this is a cover-up, and then not be able to retrieve the relevant e-mails from the Cloudy thing, even though we all know that the only way to do this is to use POP3 directly and without 20 years’ worth of persistent email storage which one might have thought the US government has by now got on top of, considering that it was the US government that gave us the Internet via DARPA and all?

    I mean, this has to be the most monumentally stupid discussion I have yet seen on these pages. And oiaohm hasn’t yet brought his unique combination of illiteracy and scattergun quotes to bear on it yet …
    I’m looking forwards to that particular amusement a little later on.

  81. dougman says:


    There is no way in hell that the federal government uses POP3, you are a dolt.

    POP3 actually removes the email from the server and stores it on the local computer without leaving a copy remotely unless configured otherwise on the client (outlook, KMail, Thunderbird, etc.) side when accessed.

    IMAP functions similarly to web-mail (Yahoo, GMail, Hotmail, etc.), the messages remain stored on the server — sort of like you browsing a webpage.

    Use your brain, computers don’t magically stop working and even when they stop working that doesn’t magically make the DATA corrupted and unacceptable.

    If you think that the Government does not have backups, you are sorely mistaken. Federal regulation requires backups and maintenance of backups of all mail data. Durations may vary slightly between certain divisions, but in almost all cases this is required and not optional.

    That said, the issue is what regulations have they broken if in fact they are claiming correctly that a persons computer was configured and managed illegally? Followed immediately by “Who is going to lose their job in addition to Lois Lerner?” I have a feeling that if jail time is threatened for management and employees responsible for mismanagement, backups may magically appear.

    Then again, they could be telling the truth which should not prevent the termination of employees failing to follow regulation and law. Simple solutions to these types of problems have huge impact on future cases.

    I’d love to see what she would say to a taxpayer “losing” 2 years of receipts during an audit.

  82. dougman says:

    This is all great news.

    Chrome surpasses IE and Woolworths will begin rolling out 8000 Google Chrome OS devices to replace Microsoft Windows desktop computers — believed to be the largest such deployment in the world.

    MicroSh1t can’t even keep up if they tried…just spinning the wheels, not going anywhere.

  83. DrLoser says:

    That “back-up” thing was my entire point, Robert.
    I neglected to mention the obvious possibility that this is just a pack of lies cooked up to protect the bureaucrats. Even at face value, one might expect to be able to pay a specialist firm, say, $50,000 or so to retrieve 80% of the disk.
    But, at face value, we’re left with an unattributed assumption that this is an M$ PC (and, further, that the same would not have happened with any other OS). And we’re left with yet another personal anecdote from a teacher in Canada who is so convinced that an entire Corporation had it in for him that he can apparently believe that there is a special, secret, API that causes the PC to cut a groove in the hard-drive platter with … well, who knows? A Stanley knife? A cucumber?
    Nah, at best this is a POP3 story.
    Have fun trying to pin the tail on the donkey, but I’d advise you to find evidence that the donkey exists in the first place.

  84. dougman says:

    Oh no!, the dog ate my homework line….LOL.

    The IRS and Lois are a pack of liars; of course they have the emails, will they ever allow them to be seen? Noooo… They probably were deleted with any other incriminating evidence.

    Speaking on liability, isn’t funny that M$ forces you to agree to multiple liability clauses?

    “Sorry chump, we are not responsible for anything, even if we really are, you still cannot sue us or invoke a class action suit against us, if you so desire to use this software and agree to click [YES]. ”

    M$ is fortunate, had they not done this, they would be bankrupt as Cyber crime costs global economy $445 billion a year.

    With Linux, all that silliness is moot for myself and millions of others.

  85. DrLoser wrote, “No OS mentioned, and even if it was, there’s clearly a difference between a PC crashing and a hard disk eating itself”

    Hard drives don’t eat themselves but that other OS often tears up file-systems. I’ve seen a few hard drives fail and the worst case has been a groove cut in the platter. All kinds of files could be recovered from them if they were of value. I am certain the US government would have been able to recover data from a failed hard drive but that other OS running amok? Perhaps not. These days, the chief failure mechanism of that other OS is to allow malware to do whatever it wants including erasing files or encrypting them.

    Further, the probability of a hard drive failure is ~1% per annum. The failure of the particular hard drive in question would be extremely improbable. Also, isn’t the USAian government using e-mail servers? Don’t they make a backup? Again, losing many thousands of e-mails is a rare event. Even more rare is such a failure during an investigation. I think the simplest explanation is the most likely: M$ messed up again…

  86. Bob Parker wrote, “Woolworth’s the largest supermarket chain in Australia is firing Windows in favour of ChromeOS.”

    Good news. I love it when I hear folks decide to quit being M$’s slave.

  87. Bob Parker says:

    Woolworth’s the largest supermarket chain in Australia is firing Windows in favour of ChromeOS. This chain is no relation at all to FW Woolworth USA.
    Of course they could have upgraded to Win 7 or 8 but they are doing the sensible thing instead.

  88. DrLoser says:

    A lovely little anecdote underlining the obvious fact that nobody whose emails need data persistence should use a lousy ancient *nix protocol like POP3, Robert. Obviously Outlook, Gmail or even IMAP at a pinch would have been a better choice. (I’ll reserve judgement on Lotus Notes…)
    Not much to see beyond that, is there? No OS mentioned, and even if it was, there’s clearly a difference between a PC crashing and a hard disk eating itself, which seems to be what happened here.
    I understand the missionary zeal here, but you really need to focus on actual, as opposed to imaginary, problems.

Leave a Reply